Sep 09 19:53:10.811149 ip-10-0-132-226 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 09 19:53:10.811162 ip-10-0-132-226 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 09 19:53:10.811171 ip-10-0-132-226 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 09 19:53:10.811407 ip-10-0-132-226 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 09 19:53:21.025624 ip-10-0-132-226 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 09 19:53:21.025642 ip-10-0-132-226 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 3b1b8fc5318a4c9fa556bb4de896c523 --
Sep 09 19:54:53.958808 ip-10-0-132-226 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 19:54:54.353626 ip-10-0-132-226 kubenswrapper[2523]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 19:54:54.353626 ip-10-0-132-226 kubenswrapper[2523]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 19:54:54.353626 ip-10-0-132-226 kubenswrapper[2523]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 19:54:54.353626 ip-10-0-132-226 kubenswrapper[2523]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 19:54:54.353626 ip-10-0-132-226 kubenswrapper[2523]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 19:54:54.354533 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.354464    2523 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 19:54:54.361329 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361315    2523 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 19:54:54.361329 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361330    2523 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 19:54:54.361394 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361333    2523 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 19:54:54.361394 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361337    2523 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 19:54:54.361394 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361340    2523 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 19:54:54.361394 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361343    2523 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 19:54:54.361394 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361345    2523 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 19:54:54.361394 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361348    2523 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 19:54:54.361394 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361350    2523 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 19:54:54.361394 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361361    2523 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 19:54:54.361394 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361363    2523 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 19:54:54.361394 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361366    2523 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 19:54:54.361394 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361369    2523 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 19:54:54.361394 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361371    2523 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 19:54:54.361394 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361374    2523 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 19:54:54.361394 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361377    2523 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 19:54:54.361394 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361379    2523 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 19:54:54.361394 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361382    2523 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 19:54:54.361394 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361385    2523 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 19:54:54.361394 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361388    2523 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 19:54:54.361394 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361390    2523 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 19:54:54.361394 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361393    2523 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 19:54:54.361892 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361395    2523 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 19:54:54.361892 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361398    2523 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 19:54:54.361892 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361401    2523 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 19:54:54.361892 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361403    2523 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 19:54:54.361892 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361406    2523 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 19:54:54.361892 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361409    2523 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 19:54:54.361892 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361412    2523 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 19:54:54.361892 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361414    2523 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 19:54:54.361892 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361417    2523 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 19:54:54.361892 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361419    2523 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 19:54:54.361892 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361421    2523 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 19:54:54.361892 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361424    2523 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 19:54:54.361892 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361427    2523 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 19:54:54.361892 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361429    2523 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 19:54:54.361892 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361432    2523 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 19:54:54.361892 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361434    2523 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 19:54:54.361892 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361438    2523 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 19:54:54.361892 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361440    2523 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 19:54:54.361892 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361442    2523 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 19:54:54.362388 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361445    2523 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 19:54:54.362388 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361450    2523 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 19:54:54.362388 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361453    2523 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 19:54:54.362388 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361456    2523 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 19:54:54.362388 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361459    2523 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 19:54:54.362388 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361463    2523 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 19:54:54.362388 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361467    2523 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 19:54:54.362388 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361470    2523 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 19:54:54.362388 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361474    2523 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 19:54:54.362388 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361477    2523 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 19:54:54.362388 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361480    2523 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 19:54:54.362388 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361482    2523 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 19:54:54.362388 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361485    2523 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 19:54:54.362388 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361487    2523 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 19:54:54.362388 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361491    2523 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 19:54:54.362388 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361494    2523 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 19:54:54.362388 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361497    2523 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 19:54:54.362388 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361499    2523 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 19:54:54.362388 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361502    2523 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 19:54:54.362861 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361504    2523 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 19:54:54.362861 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361507    2523 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 19:54:54.362861 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361511    2523 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 19:54:54.362861 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361513    2523 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 19:54:54.362861 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361516    2523 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 19:54:54.362861 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361518    2523 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 19:54:54.362861 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361521    2523 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 19:54:54.362861 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361523    2523 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 19:54:54.362861 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361526    2523 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 19:54:54.362861 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361529    2523 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 19:54:54.362861 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361531    2523 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 19:54:54.362861 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361534    2523 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 19:54:54.362861 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361537    2523 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 19:54:54.362861 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361539    2523 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 19:54:54.362861 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361542    2523 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 19:54:54.362861 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361544    2523 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 19:54:54.362861 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361547    2523 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 19:54:54.362861 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361549    2523 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 19:54:54.362861 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361563    2523 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 19:54:54.362861 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361565    2523 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 19:54:54.363346 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361568    2523 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 19:54:54.363346 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361570    2523 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 19:54:54.363346 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361573    2523 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 19:54:54.363346 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361575    2523 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 19:54:54.363346 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361578    2523 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 19:54:54.363346 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361580    2523 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 19:54:54.363346 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361583    2523 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 19:54:54.363346 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361586    2523 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 19:54:54.363346 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361589    2523 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 19:54:54.363346 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361592    2523 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 19:54:54.363346 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361595    2523 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 19:54:54.363346 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361597    2523 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 19:54:54.363346 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361600    2523 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 19:54:54.363346 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361602    2523 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 19:54:54.363346 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361605    2523 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 19:54:54.363346 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361607    2523 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 19:54:54.363346 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361610    2523 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 19:54:54.363346 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361612    2523 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 19:54:54.363346 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.361615    2523 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 19:54:54.363346 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362288    2523 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 19:54:54.363848 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362295    2523 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 19:54:54.363848 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362298    2523 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 19:54:54.363848 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362301    2523 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 19:54:54.363848 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362304    2523 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 19:54:54.363848 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362308    2523 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 19:54:54.363848 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362311    2523 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 19:54:54.363848 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362313    2523 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 19:54:54.363848 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362316    2523 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 19:54:54.363848 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362318    2523 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 19:54:54.363848 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362321    2523 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 19:54:54.363848 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362323    2523 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 19:54:54.363848 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362326    2523 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 19:54:54.363848 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362329    2523 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 19:54:54.363848 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362332    2523 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 19:54:54.363848 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362334    2523 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 19:54:54.363848 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362336    2523 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 19:54:54.363848 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362339    2523 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 19:54:54.363848 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362341    2523 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 19:54:54.363848 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362344    2523 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 19:54:54.364314 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362348    2523 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 19:54:54.364314 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362352    2523 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 19:54:54.364314 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362355    2523 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 19:54:54.364314 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362359    2523 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 19:54:54.364314 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362362    2523 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 19:54:54.364314 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362365    2523 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 19:54:54.364314 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362367    2523 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 19:54:54.364314 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362370    2523 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 19:54:54.364314 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362372    2523 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 19:54:54.364314 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362375    2523 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 19:54:54.364314 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362377    2523 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 19:54:54.364314 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362380    2523 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 19:54:54.364314 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362382    2523 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 19:54:54.364314 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362384    2523 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 19:54:54.364314 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362387    2523 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 19:54:54.364314 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362390    2523 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 19:54:54.364314 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362393    2523 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 19:54:54.364314 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362396    2523 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 19:54:54.364314 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362399    2523 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 19:54:54.364314 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362401    2523 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 19:54:54.364857 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362404    2523 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 19:54:54.364857 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362406    2523 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 19:54:54.364857 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362409    2523 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 19:54:54.364857 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362412    2523 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 19:54:54.364857 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362414    2523 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 19:54:54.364857 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362417    2523 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 19:54:54.364857 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362419    2523 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 19:54:54.364857 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362422    2523 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 19:54:54.364857 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362424    2523 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 19:54:54.364857 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362427    2523 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 19:54:54.364857 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362429    2523 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 19:54:54.364857 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362432    2523 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 19:54:54.364857 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362435    2523 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 19:54:54.364857 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362437    2523 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 19:54:54.364857 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362440    2523 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 19:54:54.364857 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362443    2523 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 19:54:54.364857 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362445    2523 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 19:54:54.364857 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362448    2523 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 19:54:54.364857 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362450    2523 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 19:54:54.364857 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362453    2523 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 19:54:54.365342 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362455    2523 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 19:54:54.365342 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362459    2523 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 19:54:54.365342 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362462    2523 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 19:54:54.365342 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362465    2523 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 19:54:54.365342 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362468    2523 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 19:54:54.365342 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362471    2523 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 19:54:54.365342 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362473    2523 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 19:54:54.365342 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362475    2523 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 19:54:54.365342 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362478    2523 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 19:54:54.365342 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362480    2523 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 19:54:54.365342 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362482    2523 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 19:54:54.365342 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362485    2523 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 19:54:54.365342 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362487    2523 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 19:54:54.365342 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362490    2523 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 19:54:54.365342 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362496    2523 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 19:54:54.365342 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362498    2523 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 19:54:54.365342 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362501    2523 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 19:54:54.365342 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362504    2523 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 19:54:54.365342 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362508    2523 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 19:54:54.365811 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362510    2523 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 19:54:54.365811 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362513    2523 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 19:54:54.365811 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362515    2523 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 19:54:54.365811 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362518    2523 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 19:54:54.365811 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362520    2523 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 19:54:54.365811 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362523    2523 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 19:54:54.365811 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362525    2523 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 19:54:54.365811 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362528    2523 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 19:54:54.365811 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362530    2523 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 19:54:54.365811 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362533    2523 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 19:54:54.365811 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362535    2523 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 19:54:54.365811 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362537    2523 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 19:54:54.365811 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362540    2523 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 19:54:54.365811 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362542    2523 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 19:54:54.365811 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362545    2523 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 19:54:54.365811 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362547    2523 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 19:54:54.365811 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362550    2523 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 19:54:54.365811 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362568    2523 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 19:54:54.365811 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362570    2523 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 19:54:54.365811 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362573    2523 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 19:54:54.366279 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362915    2523 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 19:54:54.366279 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362919    2523 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 19:54:54.366279 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362922    2523 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 19:54:54.366279 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362925    2523 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 19:54:54.366279 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362928    2523 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 19:54:54.366279 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362930    2523 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 19:54:54.366279 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362933    2523 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 19:54:54.366279 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362938    2523 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 19:54:54.366279 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362941    2523 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 19:54:54.366279 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362943    2523 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 19:54:54.366279 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362946    2523 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 19:54:54.366279 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362948    2523 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 19:54:54.366279 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362951    2523 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 19:54:54.366279 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362954    2523 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 19:54:54.366279 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362956    2523 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 19:54:54.366279 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362958    2523 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 19:54:54.366279 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362962    2523 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 19:54:54.366279 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362964    2523 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 19:54:54.366279 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362967    2523 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 19:54:54.366279 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362969    2523 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 19:54:54.366771 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362972    2523 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 19:54:54.366771 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362974    2523 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 19:54:54.366771 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362977    2523 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 19:54:54.366771 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362979    2523 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 19:54:54.366771 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362982    2523 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 19:54:54.366771 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362984    2523 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 19:54:54.366771 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362987    2523 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 19:54:54.366771 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362989    2523 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 19:54:54.366771 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362992    2523 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 19:54:54.366771 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362994    2523 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 19:54:54.366771 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362997    2523 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 19:54:54.366771 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.362999    2523 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 19:54:54.366771 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363002    2523 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 19:54:54.366771 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363004    2523 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 19:54:54.366771 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363007    2523 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 19:54:54.366771 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363009    2523 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 19:54:54.366771 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363011    2523 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 19:54:54.366771 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363014    2523 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 19:54:54.366771 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363017    2523 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 19:54:54.366771 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363021    2523 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 19:54:54.367249 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363025    2523 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 19:54:54.367249 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363028    2523 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 19:54:54.367249 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363031    2523 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 19:54:54.367249 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363034    2523 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 19:54:54.367249 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363038    2523 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 19:54:54.367249 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363041    2523 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 19:54:54.367249 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363044    2523 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 19:54:54.367249 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363046    2523 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 19:54:54.367249 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363048    2523 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 19:54:54.367249 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363051    2523 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 19:54:54.367249 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363054    2523 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 19:54:54.367249 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363056    2523 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 19:54:54.367249 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363059    2523 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 19:54:54.367249 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363061    2523 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 19:54:54.367249 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363063    2523 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 19:54:54.367249 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363066    2523 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 19:54:54.367249 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363069    2523 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 19:54:54.367249 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363073    2523 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 19:54:54.367249 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363075    2523 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 19:54:54.367724 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363078    2523 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 19:54:54.367724 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363080    2523 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 19:54:54.367724 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363083    2523 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 19:54:54.367724 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363086    2523 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 19:54:54.367724 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363088    2523 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 19:54:54.367724 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363091    2523 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 19:54:54.367724 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363093    2523 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 19:54:54.367724 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363097    2523 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 19:54:54.367724 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363099    2523 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 19:54:54.367724 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363102    2523 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 19:54:54.367724 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363105    2523 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 19:54:54.367724 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363107    2523 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 19:54:54.367724 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363111    2523 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 19:54:54.367724 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363114    2523 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 19:54:54.367724 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363116    2523 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 19:54:54.367724 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363119    2523 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 19:54:54.367724 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363121    2523 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 19:54:54.367724 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363124    2523 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 19:54:54.367724 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363127    2523 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 19:54:54.367724 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363129    2523 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 19:54:54.368231 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363131    2523 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 19:54:54.368231 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363135    2523 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 19:54:54.368231 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363137    2523 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 19:54:54.368231 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363139    2523 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 19:54:54.368231 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363142    2523 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 19:54:54.368231 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363144    2523 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 19:54:54.368231 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363147    2523 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 19:54:54.368231 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363149    2523 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 19:54:54.368231 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363152    2523 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 19:54:54.368231 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363154    2523 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 19:54:54.368231 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363157    2523 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 19:54:54.368231 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363159    2523 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 19:54:54.368231 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363162    2523 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 19:54:54.368231 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363164    2523 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 19:54:54.368231 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363167    2523 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 19:54:54.368231 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363169    2523 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 19:54:54.368231 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363172    2523 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 19:54:54.368231 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363174    2523 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 19:54:54.368231 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363176    2523 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 19:54:54.368231 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363180    2523 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 19:54:54.368743 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363245    2523 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 19:54:54.368743 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363281    2523 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 19:54:54.368743 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363286    2523 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 19:54:54.368743 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363291    2523 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 19:54:54.368743 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363297    2523 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 19:54:54.368743 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363300    2523 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 19:54:54.368743 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363304    2523 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 19:54:54.368743 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363308    2523 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 19:54:54.368743 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363311    2523 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 19:54:54.368743 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363314    2523 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 19:54:54.368743 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363317    2523 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 19:54:54.368743 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363320    2523 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 19:54:54.368743 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363324    2523 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 19:54:54.368743 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363326    2523 flags.go:64] FLAG: --cgroup-root=""
Sep 09 19:54:54.368743 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363329    2523 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 19:54:54.368743 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363332    2523 flags.go:64] FLAG: --client-ca-file=""
Sep 09 19:54:54.368743 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363335    2523 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 19:54:54.368743 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363338    2523 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 19:54:54.368743 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363342    2523 flags.go:64] FLAG: --cluster-domain=""
Sep 09 19:54:54.368743 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363345    2523 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 19:54:54.368743 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363348    2523 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 19:54:54.368743 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363351    2523 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 19:54:54.368743 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363355    2523 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 19:54:54.368743 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363362    2523 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 19:54:54.369314 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363365    2523 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 19:54:54.369314 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363368    2523 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 19:54:54.369314 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363371    2523 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 19:54:54.369314 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363374    2523 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 19:54:54.369314 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363377    2523 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 19:54:54.369314 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363380    2523 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 19:54:54.369314 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363383    2523 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 19:54:54.369314 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363387    2523 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 19:54:54.369314 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363390    2523 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 19:54:54.369314 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363393    2523 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 19:54:54.369314 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363396    2523 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 19:54:54.369314 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363398    2523 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 19:54:54.369314 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363402    2523 flags.go:64] FLAG: --enable-server="true"
Sep 09 19:54:54.369314 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363406    2523 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 19:54:54.369314 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363410    2523 flags.go:64] FLAG: --event-burst="100"
Sep 09 19:54:54.369314 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363413    2523 flags.go:64] FLAG: --event-qps="50"
Sep 09 19:54:54.369314 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363415    2523 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 19:54:54.369314 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363418    2523 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 19:54:54.369314 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363421    2523 flags.go:64] FLAG: --eviction-hard=""
Sep 09 19:54:54.369314 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363424    2523 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 19:54:54.369314 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363427    2523 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 19:54:54.369314 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363430    2523 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 19:54:54.369314 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363433    2523 flags.go:64] FLAG: --eviction-soft=""
Sep 09 19:54:54.369314 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363435    2523 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 19:54:54.369314 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363438    2523 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 19:54:54.369972 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363441    2523 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 19:54:54.369972 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363444    2523 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 19:54:54.369972 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363446    2523 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 19:54:54.369972 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363449    2523 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 19:54:54.369972 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363452    2523 flags.go:64] FLAG: --feature-gates=""
Sep 09 19:54:54.369972 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363455    2523 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 19:54:54.369972 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363458    2523 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 19:54:54.369972 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363461    2523 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 19:54:54.369972 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363464    2523 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 19:54:54.369972 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363467    2523 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 19:54:54.369972 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363470    2523 flags.go:64] FLAG: --help="false"
Sep 09 19:54:54.369972 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363472    2523 flags.go:64] FLAG: --hostname-override="ip-10-0-132-226.ec2.internal"
Sep 09 19:54:54.369972 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363475    2523 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 19:54:54.369972 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363478    2523 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 19:54:54.369972 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363482    2523 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 19:54:54.369972 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363485    2523 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 19:54:54.369972 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363488    2523 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 19:54:54.369972 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363491    2523 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 19:54:54.369972 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363494    2523 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 19:54:54.369972 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363498    2523 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 19:54:54.369972 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363501    2523 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 19:54:54.369972 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363504    2523 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 19:54:54.369972 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363507    2523 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 19:54:54.369972 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363510    2523 flags.go:64] FLAG: --kube-reserved=""
Sep 09 19:54:54.370586 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363513    2523 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 19:54:54.370586 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363515    2523 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 19:54:54.370586 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363518    2523 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 19:54:54.370586 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363521    2523 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 19:54:54.370586 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363523    2523 flags.go:64] FLAG: --lock-file=""
Sep 09 19:54:54.370586 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363526    2523 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 19:54:54.370586 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363529    2523 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 19:54:54.370586 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363532    2523 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 19:54:54.370586 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363537    2523 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 19:54:54.370586 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363540    2523 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 19:54:54.370586 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363542    2523 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 19:54:54.370586 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363545    2523 flags.go:64] FLAG: --logging-format="text"
Sep 09 19:54:54.370586 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363548    2523 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 19:54:54.370586 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363569    2523 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 19:54:54.370586 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363573    2523 flags.go:64] FLAG: --manifest-url=""
Sep 09 19:54:54.370586 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363575    2523 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 19:54:54.370586 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363580    2523 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 19:54:54.370586 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363582    2523 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 19:54:54.370586 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363586    2523 flags.go:64] FLAG: --max-pods="110"
Sep 09 19:54:54.370586 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363589    2523 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 19:54:54.370586 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363592    2523 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 19:54:54.370586 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363595    2523 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 19:54:54.370586 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363599    2523 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 19:54:54.370586 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363602    2523 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 19:54:54.370586 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363605    2523 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 19:54:54.371204 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363607    2523 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 19:54:54.371204 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363614    2523 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 19:54:54.371204 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363618    2523 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 19:54:54.371204 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363621    2523 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 19:54:54.371204 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363624    2523 flags.go:64] FLAG: --pod-cidr=""
Sep 09 19:54:54.371204 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363627    2523 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 19:54:54.371204 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363632    2523 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 19:54:54.371204 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363635    2523 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 19:54:54.371204 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363638    2523 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 19:54:54.371204 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363641    2523 flags.go:64] FLAG: --port="10250"
Sep 09 19:54:54.371204 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363644    2523 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 19:54:54.371204 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363647    2523 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0dcb328539e4414b8"
Sep 09 19:54:54.371204 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363650    2523 flags.go:64] FLAG: --qos-reserved=""
Sep 09 19:54:54.371204 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363652    2523 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 19:54:54.371204 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363655    2523 flags.go:64] FLAG: --register-node="true"
Sep 09 19:54:54.371204 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363658    2523 flags.go:64] FLAG: --register-with-taints=""
Sep 09 19:54:54.371204 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363661    2523 flags.go:64] FLAG: --registry-burst="10"
Sep 09 19:54:54.371204 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363664    2523 flags.go:64] FLAG: --registry-qps="5"
Sep 09 19:54:54.371204 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363667    2523 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 19:54:54.371204 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363669    2523 flags.go:64] FLAG: --reserved-memory=""
Sep 09 19:54:54.371204 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363673    2523 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 19:54:54.371204 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363675    2523 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 19:54:54.371204 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363678    2523 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 19:54:54.371204 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363681    2523 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 19:54:54.371204 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363683    2523 flags.go:64] FLAG: --runonce="false"
Sep 09 19:54:54.371837 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363686    2523 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 19:54:54.371837 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363689    2523 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 19:54:54.371837 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363692    2523 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 19:54:54.371837 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363694    2523 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 19:54:54.371837 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363698    2523 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 19:54:54.371837 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363701    2523 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 19:54:54.371837 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363704    2523 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 19:54:54.371837 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363707    2523 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 19:54:54.371837 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363709    2523 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 19:54:54.371837 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363712    2523 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 19:54:54.371837 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363716    2523 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 19:54:54.371837 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363718    2523 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 19:54:54.371837 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363721    2523 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 19:54:54.371837 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363725    2523 flags.go:64] FLAG: --system-cgroups=""
Sep 09 19:54:54.371837 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363727    2523 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 19:54:54.371837 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363732    2523 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 19:54:54.371837 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363735    2523 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 19:54:54.371837 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363737    2523 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 19:54:54.371837 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363741    2523 flags.go:64] FLAG: --tls-min-version=""
Sep 09 19:54:54.371837 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363744    2523 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 19:54:54.371837 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363747    2523 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 19:54:54.371837 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363749    2523 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 19:54:54.371837 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363752    2523 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 19:54:54.371837 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363755    2523 flags.go:64] FLAG: --v="2"
Sep 09 19:54:54.371837 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363758    2523 flags.go:64] FLAG: --version="false"
Sep 09 19:54:54.372439 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363762    2523 flags.go:64] FLAG: --vmodule=""
Sep 09 19:54:54.372439 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363765    2523 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 19:54:54.372439 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.363768    2523 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 19:54:54.372439 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363875    2523 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 19:54:54.372439 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363879    2523 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 19:54:54.372439 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363882    2523 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 19:54:54.372439 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363885    2523 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 19:54:54.372439 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363889    2523 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 19:54:54.372439 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363893    2523 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 19:54:54.372439 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363896    2523 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 19:54:54.372439 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363899    2523 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 19:54:54.372439 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363903    2523 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 19:54:54.372439 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363905    2523 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 19:54:54.372439 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363908    2523 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 19:54:54.372439 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363910    2523 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 19:54:54.372439 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363913    2523 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 19:54:54.372439 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363916    2523 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 19:54:54.372439 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363919    2523 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 19:54:54.372439 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363922    2523 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 19:54:54.372439 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363924    2523 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 19:54:54.372943 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363926    2523 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 19:54:54.372943 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363929    2523 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 19:54:54.372943 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363931    2523 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 19:54:54.372943 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363934    2523 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 19:54:54.372943 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363937    2523 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 19:54:54.372943 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363939    2523 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 19:54:54.372943 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363941    2523 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 19:54:54.372943 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363944    2523 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 19:54:54.372943 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363946    2523 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 19:54:54.372943 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363949    2523 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 19:54:54.372943 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363951    2523 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 19:54:54.372943 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363953    2523 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 19:54:54.372943 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363956    2523 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 19:54:54.372943 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363958    2523 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 19:54:54.372943 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363960    2523 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 19:54:54.372943 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363963    2523 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 19:54:54.372943 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363965    2523 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 19:54:54.372943 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363968    2523 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 19:54:54.372943 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363970    2523 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 19:54:54.373436 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363973    2523 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 19:54:54.373436 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363975    2523 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 19:54:54.373436 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363977    2523 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 19:54:54.373436 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363980    2523 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 19:54:54.373436 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363984    2523 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 19:54:54.373436 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363987    2523 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 19:54:54.373436 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363990    2523 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 19:54:54.373436 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363993    2523 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 19:54:54.373436 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363995    2523 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 19:54:54.373436 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.363998    2523 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 19:54:54.373436 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.364001    2523 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 19:54:54.373436 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.364004    2523 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 19:54:54.373436 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.364008    2523 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 19:54:54.373436 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.364011    2523 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 19:54:54.373436 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.364014    2523 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 19:54:54.373436 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.364018    2523 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 19:54:54.373436 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.364020    2523 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 19:54:54.373436 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.364023    2523 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 19:54:54.373436 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.364025    2523 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 19:54:54.373908 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.364027    2523 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 19:54:54.373908 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.364030    2523 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 19:54:54.373908 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.364032    2523 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 19:54:54.373908 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.364035    2523 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 19:54:54.373908 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.364037    2523 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 19:54:54.373908 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.364039    2523 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 19:54:54.373908 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.364042    2523 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 19:54:54.373908 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.364044    2523 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 19:54:54.373908 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.364047    2523 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 19:54:54.373908 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.364049    2523 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 19:54:54.373908 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.364052    2523 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 19:54:54.373908 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.364054    2523 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 19:54:54.373908 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.364057    2523 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 19:54:54.373908 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.364059    2523 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 19:54:54.373908 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.364061    2523 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 19:54:54.373908 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.364064    2523 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 19:54:54.373908 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.364066    2523 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 19:54:54.373908 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.364069    2523 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 19:54:54.373908 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.364071    2523 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 19:54:54.373908 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.364073    2523 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 19:54:54.374392 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.364076    2523 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 19:54:54.374392 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.364078    2523 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 19:54:54.374392 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.364081    2523 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 19:54:54.374392 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.364083    2523 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 19:54:54.374392 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.364086    2523 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 19:54:54.374392 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.364089    2523 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 19:54:54.374392 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.364091    2523 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 19:54:54.374392 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.364093    2523 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 19:54:54.374392 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.364096    2523 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 19:54:54.374392 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.364098    2523 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 19:54:54.374392 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.364101    2523 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 19:54:54.374392 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.364103    2523 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 19:54:54.374392 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.364106    2523 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 19:54:54.374392 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.364108    2523 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 19:54:54.374392 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.364110    2523 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 19:54:54.374392 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.364112    2523 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 19:54:54.374392 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.364115    2523 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 19:54:54.374392 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.364117    2523 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 19:54:54.374392 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.364119    2523 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 19:54:54.374392 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.364122    2523 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 19:54:54.374901 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.364124    2523 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 19:54:54.374901 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.364126    2523 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 19:54:54.374901 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.364129    2523 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 19:54:54.374901 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.364131    2523 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 19:54:54.374901 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.364709    2523 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 19:54:54.374901 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.371463    2523 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 19:54:54.374901 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.371477    2523 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 19:54:54.374901 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.371490    2523 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 19:54:54.374901 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.371496    2523 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 19:54:54.374901 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371581    2523 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 19:54:54.374901 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371586    2523 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 19:54:54.374901 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371589    2523 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 19:54:54.374901 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371592    2523 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 19:54:54.374901 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371594    2523 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 19:54:54.375325 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371597    2523 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 19:54:54.375325 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371600    2523 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 19:54:54.375325 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371603    2523 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 19:54:54.375325 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371605    2523 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 19:54:54.375325 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371608    2523 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 19:54:54.375325 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371612    2523 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 19:54:54.375325 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371615    2523 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 19:54:54.375325 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371618    2523 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 19:54:54.375325 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371621    2523 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 19:54:54.375325 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371624    2523 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 19:54:54.375325 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371627    2523 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 19:54:54.375325 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371630    2523 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 19:54:54.375325 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371634    2523 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 19:54:54.375325 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371637    2523 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 19:54:54.375325 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371640    2523 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 19:54:54.375325 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371643    2523 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 19:54:54.375325 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371646    2523 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 19:54:54.375325 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371649    2523 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 19:54:54.375325 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371652    2523 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 19:54:54.375807 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371654    2523 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 19:54:54.375807 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371657    2523 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 19:54:54.375807 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371659    2523 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 19:54:54.375807 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371662    2523 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 19:54:54.375807 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371667    2523 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 19:54:54.375807 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371674    2523 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 19:54:54.375807 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371678    2523 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 19:54:54.375807 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371681    2523 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 19:54:54.375807 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371683    2523 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 19:54:54.375807 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371686    2523 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 19:54:54.375807 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371688    2523 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 19:54:54.375807 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371691    2523 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 19:54:54.375807 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371693    2523 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 19:54:54.375807 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371696    2523 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 19:54:54.375807 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371698    2523 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 19:54:54.375807 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371701    2523 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 19:54:54.375807 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371703    2523 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 19:54:54.375807 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371706    2523 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 19:54:54.375807 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371708    2523 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 19:54:54.376257 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371711    2523 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 19:54:54.376257 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371713    2523 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 19:54:54.376257 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371716    2523 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 19:54:54.376257 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371718    2523 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 19:54:54.376257 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371721    2523 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 19:54:54.376257 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371723    2523 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 19:54:54.376257 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371726    2523 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 19:54:54.376257 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371728    2523 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 19:54:54.376257 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371730    2523 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 19:54:54.376257 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371733    2523 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 19:54:54.376257 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371736    2523 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 19:54:54.376257 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371738    2523 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 19:54:54.376257 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371741    2523 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 19:54:54.376257 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371743    2523 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 19:54:54.376257 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371746    2523 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 19:54:54.376257 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371748    2523 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 19:54:54.376257 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371751    2523 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 19:54:54.376257 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371753    2523 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 19:54:54.376257 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371756    2523 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 19:54:54.376257 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371759    2523 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 19:54:54.376771 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371762    2523 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 19:54:54.376771 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371764    2523 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 19:54:54.376771 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371767    2523 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 19:54:54.376771 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371769    2523 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 19:54:54.376771 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371771    2523 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 19:54:54.376771 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371774    2523 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 19:54:54.376771 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371776    2523 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 19:54:54.376771 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371779    2523 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 19:54:54.376771 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371781    2523 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 19:54:54.376771 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371783    2523 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 19:54:54.376771 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371786    2523 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 19:54:54.376771 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371788    2523 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 19:54:54.376771 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371791    2523 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 19:54:54.376771 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371793    2523 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 19:54:54.376771 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371796    2523 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 19:54:54.376771 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371798    2523 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 19:54:54.376771 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371801    2523 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 19:54:54.376771 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371803    2523 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 19:54:54.376771 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371805    2523 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 19:54:54.376771 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371808    2523 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 19:54:54.377286 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371810    2523 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 19:54:54.377286 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371812    2523 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 19:54:54.377286 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371815    2523 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 19:54:54.377286 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371817    2523 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 19:54:54.377286 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371820    2523 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 19:54:54.377286 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371822    2523 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 19:54:54.377286 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371825    2523 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 19:54:54.377286 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371827    2523 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 19:54:54.377286 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371829    2523 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 19:54:54.377286 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371832    2523 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 19:54:54.377286 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371834    2523 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 19:54:54.377286 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371837    2523 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 19:54:54.377286 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371840    2523 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 19:54:54.377286 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371843    2523 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 19:54:54.377286 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371846    2523 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 19:54:54.377286 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371848    2523 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 19:54:54.377687 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.371853    2523 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 19:54:54.377687 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371956    2523 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 19:54:54.377687 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371961    2523 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 19:54:54.377687 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371964    2523 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 19:54:54.377687 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371966    2523 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 19:54:54.377687 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371969    2523 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 19:54:54.377687 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371971    2523 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 19:54:54.377687 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371974    2523 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 19:54:54.377687 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371976    2523 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 19:54:54.377687 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371979    2523 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 19:54:54.377687 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371981    2523 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 19:54:54.377687 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371983    2523 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 19:54:54.377687 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371986    2523 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 19:54:54.377687 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371989    2523 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 19:54:54.377687 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371991    2523 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 19:54:54.378043 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371993    2523 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 19:54:54.378043 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371996    2523 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 19:54:54.378043 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.371998    2523 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 19:54:54.378043 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372001    2523 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 19:54:54.378043 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372004    2523 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 19:54:54.378043 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372006    2523 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 19:54:54.378043 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372009    2523 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 19:54:54.378043 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372011    2523 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 19:54:54.378043 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372013    2523 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 19:54:54.378043 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372016    2523 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 19:54:54.378043 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372018    2523 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 19:54:54.378043 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372020    2523 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 19:54:54.378043 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372023    2523 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 19:54:54.378043 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372025    2523 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 19:54:54.378043 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372028    2523 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 19:54:54.378043 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372030    2523 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 19:54:54.378043 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372033    2523 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 19:54:54.378043 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372035    2523 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 19:54:54.378043 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372038    2523 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 19:54:54.378485 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372040    2523 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 19:54:54.378485 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372043    2523 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 19:54:54.378485 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372045    2523 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 19:54:54.378485 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372048    2523 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 19:54:54.378485 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372050    2523 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 19:54:54.378485 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372052    2523 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 19:54:54.378485 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372055    2523 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 19:54:54.378485 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372057    2523 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 19:54:54.378485 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372059    2523 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 19:54:54.378485 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372062    2523 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 19:54:54.378485 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372064    2523 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 19:54:54.378485 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372067    2523 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 19:54:54.378485 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372069    2523 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 19:54:54.378485 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372071    2523 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 19:54:54.378485 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372074    2523 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 19:54:54.378485 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372076    2523 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 19:54:54.378485 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372079    2523 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 19:54:54.378485 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372081    2523 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 19:54:54.378485 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372084    2523 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 19:54:54.378485 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372086    2523 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 19:54:54.378982 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372089    2523 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 19:54:54.378982 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372091    2523 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 19:54:54.378982 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372093    2523 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 19:54:54.378982 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372096    2523 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 19:54:54.378982 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372098    2523 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 19:54:54.378982 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372100    2523 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 19:54:54.378982 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372103    2523 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 19:54:54.378982 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372105    2523 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 19:54:54.378982 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372109    2523 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 19:54:54.378982 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372111    2523 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 19:54:54.378982 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372113    2523 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 19:54:54.378982 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372117    2523 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 19:54:54.378982 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372120    2523 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 19:54:54.378982 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372123    2523 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 19:54:54.378982 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372126    2523 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 19:54:54.378982 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372128    2523 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 19:54:54.378982 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372131    2523 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 19:54:54.378982 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372133    2523 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 19:54:54.378982 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372136    2523 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 19:54:54.379445 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372139    2523 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 19:54:54.379445 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372141    2523 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 19:54:54.379445 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372144    2523 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 19:54:54.379445 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372146    2523 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 19:54:54.379445 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372149    2523 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 19:54:54.379445 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372151    2523 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 19:54:54.379445 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372153    2523 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 19:54:54.379445 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372156    2523 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 19:54:54.379445 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372159    2523 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 19:54:54.379445 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372163    2523 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 19:54:54.379445 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372166    2523 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 19:54:54.379445 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372168    2523 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 19:54:54.379445 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372170    2523 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 19:54:54.379445 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372173    2523 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 19:54:54.379445 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372175    2523 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 19:54:54.379445 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372178    2523 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 19:54:54.379445 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372180    2523 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 19:54:54.379445 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372182    2523 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 19:54:54.379445 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372185    2523 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 19:54:54.380027 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372188    2523 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 19:54:54.380027 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372192    2523 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 19:54:54.380027 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372194    2523 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 19:54:54.380027 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372198    2523 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 19:54:54.380027 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372200    2523 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 19:54:54.380027 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372203    2523 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 19:54:54.380027 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372206    2523 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 19:54:54.380027 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:54.372208    2523 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 19:54:54.380027 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.372212    2523 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 19:54:54.380027 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.372301    2523 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 19:54:54.380027 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.374213    2523 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 19:54:54.380027 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.375210    2523 server.go:1019] "Starting client certificate rotation"
Sep 09 19:54:54.380027 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.375315    2523 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 19:54:54.380027 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.375353    2523 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 19:54:54.380378 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.377936    2523 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 19:54:54.380378 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.378951    2523 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 19:54:54.391790 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.391772    2523 log.go:25] "Validated CRI v1 runtime API"
Sep 09 19:54:54.394468 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.394454    2523 log.go:25] "Validated CRI v1 image API"
Sep 09 19:54:54.395349 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.395325    2523 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 19:54:54.399871 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.399847    2523 fs.go:135] Filesystem UUIDs: map[6eb58e83-7418-4f48-8578-58b6b69e7da7:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 cc8cd65e-3234-4f98-a2ee-369c456b6ffa:/dev/nvme0n1p3]
Sep 09 19:54:54.399922 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.399871    2523 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 19:54:54.404906 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.404760    2523 manager.go:216] Machine: {Timestamp:2026-09-09 19:54:54.403454287 +0000 UTC m=+0.332067175 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097161 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27d9004dd67401fadd38d68b3c5ac6 SystemUUID:ec27d900-4dd6-7401-fadd-38d68b3c5ac6 BootID:3b1b8fc5-318a-4c9f-a556-bb4de896c523 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:29:16:a5:58:d5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:29:16:a5:58:d5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5a:c1:4a:e1:21:cb Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 19:54:54.404906 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.404904    2523 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 19:54:54.405008 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.404980    2523 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 19:54:54.406624 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.406607    2523 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 19:54:54.408077 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.408053    2523 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 19:54:54.408201 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.408081    2523 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-226.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 19:54:54.409100 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.409089    2523 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 19:54:54.409135 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.409102    2523 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 19:54:54.409135 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.409109    2523 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 19:54:54.409921 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.409911    2523 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 19:54:54.411117 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.411108    2523 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 19:54:54.413883 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.413872    2523 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 19:54:54.413989 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.413980    2523 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 19:54:54.414025 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.414016    2523 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 19:54:54.414061 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.414035    2523 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 19:54:54.414061 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.414049    2523 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 19:54:54.414061 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.414058    2523 kubelet.go:398] "Adding apiserver pod source"
Sep 09 19:54:54.414137 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.414066    2523 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 19:54:54.414379 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.414366    2523 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 19:54:54.414437 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.414389    2523 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 19:54:54.417467 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.417446    2523 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 19:54:54.419747 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.419724    2523 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 19:54:54.419791 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.419761    2523 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 19:54:54.420788 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.420778    2523 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 19:54:54.420821 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.420793    2523 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 19:54:54.420821 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.420799    2523 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 19:54:54.420821 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.420804    2523 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 19:54:54.420821 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.420815    2523 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 19:54:54.420924 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.420824    2523 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 19:54:54.420924 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.420841    2523 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 19:54:54.420924 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.420846    2523 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 19:54:54.420924 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.420854    2523 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 19:54:54.420924 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.420859    2523 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 19:54:54.420924 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.420867    2523 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 19:54:54.420924 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.420872    2523 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 19:54:54.420924 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.420893    2523 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 19:54:54.420924 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.420901    2523 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 19:54:54.424710 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.424697    2523 server.go:1268] "Started kubelet"
Sep 09 19:54:54.424827 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.424791    2523 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 19:54:54.424888 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.424773    2523 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 19:54:54.424888 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.424854    2523 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 19:54:54.428986 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.428965    2523 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 19:54:54.429270 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.429239    2523 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-226.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 19:54:54.429280 ip-10-0-132-226 systemd[1]: Started Kubernetes Kubelet.
Sep 09 19:54:54.430926 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:54.429611    2523 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 19:54:54.430926 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:54.430806    2523 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-226.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 19:54:54.431845 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.431459    2523 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 19:54:54.432748 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:54.432724    2523 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 19:54:54.435599 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.435586    2523 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 19:54:54.435686 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.435667    2523 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 19:54:54.435794 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:54.435776    2523 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-226.ec2.internal\" not found"
Sep 09 19:54:54.436223 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.436205    2523 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 19:54:54.436306 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:54.435342    2523 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-226.ec2.internal.18d3bf310f25ffde  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-226.ec2.internal,UID:ip-10-0-132-226.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-226.ec2.internal,},FirstTimestamp:2026-09-09 19:54:54.424670174 +0000 UTC m=+0.353283060,LastTimestamp:2026-09-09 19:54:54.424670174 +0000 UTC m=+0.353283060,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-226.ec2.internal,}"
Sep 09 19:54:54.436306 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.436277    2523 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 19:54:54.436306 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.436289    2523 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 19:54:54.436450 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.436400    2523 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 19:54:54.436450 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.436412    2523 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 19:54:54.436526 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.436479    2523 factory.go:55] Registering systemd factory
Sep 09 19:54:54.436526 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.436494    2523 factory.go:223] Registration of the systemd container factory successfully
Sep 09 19:54:54.438405 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.436759    2523 factory.go:153] Registering CRI-O factory
Sep 09 19:54:54.438405 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.436769    2523 factory.go:223] Registration of the crio container factory successfully
Sep 09 19:54:54.438405 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.436808    2523 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 19:54:54.438405 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.436822    2523 factory.go:103] Registering Raw factory
Sep 09 19:54:54.438405 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.436832    2523 manager.go:1196] Started watching for new ooms in manager
Sep 09 19:54:54.438405 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.437084    2523 manager.go:349] Starting recovery of all containers
Sep 09 19:54:54.444210 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:54.444184    2523 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 09 19:54:54.444307 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:54.444208    2523 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-226.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 09 19:54:54.444745 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.444728    2523 manager.go:354] Recovery completed
Sep 09 19:54:54.449878 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.449865    2523 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 19:54:54.451705 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.451676    2523 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-226.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 19:54:54.451779 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.451719    2523 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-226.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 19:54:54.451779 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.451733    2523 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-226.ec2.internal" event="NodeHasSufficientPID"
Sep 09 19:54:54.454195 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.454172    2523 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 19:54:54.454279 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.454197    2523 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 19:54:54.454279 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.454215    2523 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 19:54:54.454407 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.454388    2523 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 19:54:54.454501 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:54.454334    2523 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-226.ec2.internal.18d3bf3110c28200  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-226.ec2.internal,UID:ip-10-0-132-226.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-226.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-226.ec2.internal,},FirstTimestamp:2026-09-09 19:54:54.45170432 +0000 UTC m=+0.380317208,LastTimestamp:2026-09-09 19:54:54.45170432 +0000 UTC m=+0.380317208,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-226.ec2.internal,}"
Sep 09 19:54:54.455943 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.455923    2523 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-226.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 19:54:54.456018 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.455959    2523 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-226.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 19:54:54.456018 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.455989    2523 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-226.ec2.internal" event="NodeHasSufficientPID"
Sep 09 19:54:54.460597 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.460579    2523 policy_none.go:49] "None policy: Start"
Sep 09 19:54:54.460669 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.460601    2523 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 19:54:54.460669 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.460617    2523 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 19:54:54.462356 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.462254    2523 policy_none.go:47] "Start"
Sep 09 19:54:54.464583 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.464532    2523 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6jkbn"
Sep 09 19:54:54.466937 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:54.466839    2523 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-226.ec2.internal.18d3bf3110c2d5e6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-226.ec2.internal,UID:ip-10-0-132-226.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-226.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-226.ec2.internal,},FirstTimestamp:2026-09-09 19:54:54.451725798 +0000 UTC m=+0.380338685,LastTimestamp:2026-09-09 19:54:54.451725798 +0000 UTC m=+0.380338685,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-226.ec2.internal,}"
Sep 09 19:54:54.473533 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.473516    2523 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6jkbn"
Sep 09 19:54:54.475224 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:54.475166    2523 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-226.ec2.internal.18d3bf3110c308b1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-226.ec2.internal,UID:ip-10-0-132-226.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-132-226.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-132-226.ec2.internal,},FirstTimestamp:2026-09-09 19:54:54.451738801 +0000 UTC m=+0.380351687,LastTimestamp:2026-09-09 19:54:54.451738801 +0000 UTC m=+0.380351687,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-226.ec2.internal,}"
Sep 09 19:54:54.481602 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:54.481522    2523 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-226.ec2.internal.18d3bf3110c28200\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-226.ec2.internal.18d3bf3110c28200  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-226.ec2.internal,UID:ip-10-0-132-226.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-226.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-226.ec2.internal,},FirstTimestamp:2026-09-09 19:54:54.45170432 +0000 UTC m=+0.380317208,LastTimestamp:2026-09-09 19:54:54.455942237 +0000 UTC m=+0.384555123,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-226.ec2.internal,}"
Sep 09 19:54:54.513494 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.509149    2523 manager.go:337] "Starting Device Plugin manager"
Sep 09 19:54:54.513494 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:54.509184    2523 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 19:54:54.513494 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.509197    2523 server.go:87] "Starting device plugin registration server"
Sep 09 19:54:54.513494 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.509467    2523 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 19:54:54.513494 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.509478    2523 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 19:54:54.513494 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.509621    2523 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 19:54:54.513494 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.509716    2523 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 19:54:54.513494 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.509750    2523 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 19:54:54.513494 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:54.510270    2523 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 19:54:54.513494 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:54.510315    2523 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-226.ec2.internal\" not found"
Sep 09 19:54:54.544086 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.544055    2523 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 19:54:54.545268 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.545256    2523 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 19:54:54.545314 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.545295    2523 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 19:54:54.545314 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.545312    2523 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 19:54:54.545381 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:54.545371    2523 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 19:54:54.548308 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.548245    2523 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 19:54:54.610025 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.609971    2523 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 19:54:54.610761 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.610747    2523 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-226.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 19:54:54.610845 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.610772    2523 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-226.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 19:54:54.610845 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.610787    2523 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-226.ec2.internal" event="NodeHasSufficientPID"
Sep 09 19:54:54.610845 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.610813    2523 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-226.ec2.internal"
Sep 09 19:54:54.621015 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.620996    2523 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-226.ec2.internal"
Sep 09 19:54:54.621094 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:54.621019    2523 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-226.ec2.internal\": node \"ip-10-0-132-226.ec2.internal\" not found"
Sep 09 19:54:54.646266 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.646248    2523 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-226.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-226.ec2.internal"]
Sep 09 19:54:54.646347 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.646305    2523 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 19:54:54.648075 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:54.648058    2523 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-226.ec2.internal\" not found"
Sep 09 19:54:54.649713 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.649700    2523 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-226.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 19:54:54.649792 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.649725    2523 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-226.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 19:54:54.649792 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.649736    2523 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-226.ec2.internal" event="NodeHasSufficientPID"
Sep 09 19:54:54.650959 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.650947    2523 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 19:54:54.651103 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.651089    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-226.ec2.internal"
Sep 09 19:54:54.651165 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.651125    2523 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 19:54:54.651678 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.651662    2523 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-226.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 19:54:54.651763 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.651685    2523 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-226.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 19:54:54.651763 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.651699    2523 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-226.ec2.internal" event="NodeHasSufficientPID"
Sep 09 19:54:54.651763 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.651731    2523 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-226.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 19:54:54.651763 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.651754    2523 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-226.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 19:54:54.651938 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.651768    2523 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-226.ec2.internal" event="NodeHasSufficientPID"
Sep 09 19:54:54.653069 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.653053    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-226.ec2.internal"
Sep 09 19:54:54.653115 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.653086    2523 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 19:54:54.657334 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.657319    2523 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-226.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 19:54:54.657433 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.657338    2523 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-226.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 19:54:54.657433 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.657351    2523 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-226.ec2.internal" event="NodeHasSufficientPID"
Sep 09 19:54:54.670058 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:54.670039    2523 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-226.ec2.internal\" not found" node="ip-10-0-132-226.ec2.internal"
Sep 09 19:54:54.675633 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:54.675614    2523 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-226.ec2.internal\" not found" node="ip-10-0-132-226.ec2.internal"
Sep 09 19:54:54.748723 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:54.748703    2523 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-226.ec2.internal\" not found"
Sep 09 19:54:54.838035 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.838007    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/75ee1d831c2282f17a9072ea7f2d2163-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-226.ec2.internal\" (UID: \"75ee1d831c2282f17a9072ea7f2d2163\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-226.ec2.internal"
Sep 09 19:54:54.838035 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.838038    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/75ee1d831c2282f17a9072ea7f2d2163-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-226.ec2.internal\" (UID: \"75ee1d831c2282f17a9072ea7f2d2163\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-226.ec2.internal"
Sep 09 19:54:54.838160 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:54.838056    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f7c24ed75a93d3886eeb14da20c53cdf-config\") pod \"kube-apiserver-proxy-ip-10-0-132-226.ec2.internal\" (UID: \"f7c24ed75a93d3886eeb14da20c53cdf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-226.ec2.internal"
Sep 09 19:54:54.849117 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:54.849103    2523 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-226.ec2.internal\" not found"
Sep 09 19:54:56.195345 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:56.195319    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/75ee1d831c2282f17a9072ea7f2d2163-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-226.ec2.internal\" (UID: \"75ee1d831c2282f17a9072ea7f2d2163\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-226.ec2.internal"
Sep 09 19:54:56.195426 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:56.195351    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/75ee1d831c2282f17a9072ea7f2d2163-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-226.ec2.internal\" (UID: \"75ee1d831c2282f17a9072ea7f2d2163\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-226.ec2.internal"
Sep 09 19:54:56.195426 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:56.195368    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f7c24ed75a93d3886eeb14da20c53cdf-config\") pod \"kube-apiserver-proxy-ip-10-0-132-226.ec2.internal\" (UID: \"f7c24ed75a93d3886eeb14da20c53cdf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-226.ec2.internal"
Sep 09 19:54:56.195426 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:56.195422    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/75ee1d831c2282f17a9072ea7f2d2163-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-226.ec2.internal\" (UID: \"75ee1d831c2282f17a9072ea7f2d2163\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-226.ec2.internal"
Sep 09 19:54:56.195536 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:56.195436    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/75ee1d831c2282f17a9072ea7f2d2163-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-226.ec2.internal\" (UID: \"75ee1d831c2282f17a9072ea7f2d2163\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-226.ec2.internal"
Sep 09 19:54:56.195536 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:56.195489    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f7c24ed75a93d3886eeb14da20c53cdf-config\") pod \"kube-apiserver-proxy-ip-10-0-132-226.ec2.internal\" (UID: \"f7c24ed75a93d3886eeb14da20c53cdf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-226.ec2.internal"
Sep 09 19:54:56.206443 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:56.206424    2523 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-226.ec2.internal\" not found"
Sep 09 19:54:56.228579 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:56.228565    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-226.ec2.internal"
Sep 09 19:54:56.235367 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:56.235347    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-226.ec2.internal"
Sep 09 19:54:56.307213 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:56.307184    2523 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-226.ec2.internal\" not found"
Sep 09 19:54:56.407642 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:56.407623    2523 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-226.ec2.internal\" not found"
Sep 09 19:54:56.508207 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:56.508162    2523 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-226.ec2.internal\" not found"
Sep 09 19:54:56.608626 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:56.608601    2523 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-226.ec2.internal\" not found"
Sep 09 19:54:56.631755 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:56.631736    2523 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 19:54:56.632194 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:56.631853    2523 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 19:54:56.709373 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:56.709358    2523 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-226.ec2.internal\" not found"
Sep 09 19:54:56.734529 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:56.734500    2523 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 19:49:55 +0000 UTC" deadline="2028-04-06 20:00:56.89852937 +0000 UTC"
Sep 09 19:54:56.734529 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:56.734525    2523 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13800h6m0.164007479s"
Sep 09 19:54:56.809851 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:56.809832    2523 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-226.ec2.internal\" not found"
Sep 09 19:54:56.885552 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:56.885520    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7c24ed75a93d3886eeb14da20c53cdf.slice/crio-93d76232826c4a4694d170d69cc9e3cff8ccf67614b3bf9653b2ee59192e73d4 WatchSource:0}: Error finding container 93d76232826c4a4694d170d69cc9e3cff8ccf67614b3bf9653b2ee59192e73d4: Status 404 returned error can't find the container with id 93d76232826c4a4694d170d69cc9e3cff8ccf67614b3bf9653b2ee59192e73d4
Sep 09 19:54:56.890453 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:56.890440    2523 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 19:54:56.896199 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:56.896185    2523 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 19:54:56.896930 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:56.896913    2523 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-226.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 19:54:56.896986 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:56.896944    2523 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-226.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 19:54:56.896986 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:56.896953    2523 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-226.ec2.internal" event="NodeHasSufficientPID"
Sep 09 19:54:56.910611 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:56.910594    2523 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-226.ec2.internal\" not found"
Sep 09 19:54:57.011107 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:57.011087    2523 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-226.ec2.internal\" not found"
Sep 09 19:54:57.020695 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:57.020677    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75ee1d831c2282f17a9072ea7f2d2163.slice/crio-021edc0852545ebf1d36d9248ab5f9ef8958182b99eb9a35f0dcf49b30194465 WatchSource:0}: Error finding container 021edc0852545ebf1d36d9248ab5f9ef8958182b99eb9a35f0dcf49b30194465: Status 404 returned error can't find the container with id 021edc0852545ebf1d36d9248ab5f9ef8958182b99eb9a35f0dcf49b30194465
Sep 09 19:54:57.045510 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.045460    2523 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 19:54:57.111909 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:57.111889    2523 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-226.ec2.internal\" not found"
Sep 09 19:54:57.180872 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.180851    2523 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 19:54:57.203955 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.203936    2523 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 19:54:57.292382 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.292355    2523 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-226.ec2.internal"
Sep 09 19:54:57.302936 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.302872    2523 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 19:54:57.304061 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.304045    2523 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-226.ec2.internal"
Sep 09 19:54:57.316783 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.316763    2523 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 19:54:57.518163 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.518120    2523 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 19:54:57.671703 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.671623    2523 apiserver.go:52] "Watching apiserver"
Sep 09 19:54:57.679360 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.679338    2523 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 19:54:57.682096 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.682071    2523 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-xnmz6","openshift-multus/network-metrics-daemon-tmvpv","openshift-network-diagnostics/network-check-target-jcx8l","openshift-ovn-kubernetes/ovnkube-node-c72g2","kube-system/global-pull-secret-syncer-gn8cc","kube-system/konnectivity-agent-vkhvk","openshift-dns/node-resolver-w2tb2","openshift-image-registry/node-ca-ctlrg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-226.ec2.internal","openshift-multus/multus-additional-cni-plugins-4jptv","openshift-network-operator/iptables-alerter-m64jf","kube-system/kube-apiserver-proxy-ip-10-0-132-226.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l55pn","openshift-cluster-node-tuning-operator/tuned-ntkpv"]
Sep 09 19:54:57.683733 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.683712    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vkhvk"
Sep 09 19:54:57.684837 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.684813    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tmvpv"
Sep 09 19:54:57.684937 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:57.684896    2523 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tmvpv" podUID="21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8"
Sep 09 19:54:57.686792 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.686772    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-shm9q\""
Sep 09 19:54:57.687023 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.687006    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 19:54:57.687217 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.687204    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 19:54:57.687469 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.687455    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jcx8l"
Sep 09 19:54:57.687538 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:57.687517    2523 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jcx8l" podUID="d5e04e1e-56ef-4e29-89bd-eafcf7425d56"
Sep 09 19:54:57.688132 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.688063    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.689890 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.689868    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xnmz6"
Sep 09 19:54:57.690848 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.690535    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-w2tb2"
Sep 09 19:54:57.690848 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.690671    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 19:54:57.691577 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.691472    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 19:54:57.691577 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.691485    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 19:54:57.691577 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.691530    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 19:54:57.691577 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.691473    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 19:54:57.691781 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.691485    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mpsrs\""
Sep 09 19:54:57.691865 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.691847    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gn8cc"
Sep 09 19:54:57.691932 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:57.691916    2523 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gn8cc" podUID="b2cc1cab-a99d-4733-8623-2ccf3fb89961"
Sep 09 19:54:57.692825 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.692672    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 19:54:57.692899 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.692872    2523 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 19:54:57.693010 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.692993    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ctlrg"
Sep 09 19:54:57.694422 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.694403    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 19:54:57.694557 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.694536    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-lnqjj\""
Sep 09 19:54:57.694660 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.694607    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 19:54:57.694660 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.694538    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 19:54:57.694855 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.694839    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 19:54:57.694993 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.694981    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 19:54:57.695118 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.695100    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qwmls\""
Sep 09 19:54:57.695177 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.695102    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 19:54:57.695286 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.695270    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4jptv"
Sep 09 19:54:57.695514 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.695493    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 19:54:57.697258 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.697238    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hm5s7\""
Sep 09 19:54:57.697881 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.697859    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-m64jf"
Sep 09 19:54:57.701048 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.699568    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 19:54:57.701048 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.699910    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 19:54:57.701048 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.700327    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 19:54:57.701048 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.700771    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 19:54:57.701311 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.701102    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l55pn"
Sep 09 19:54:57.701407 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.701379    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 19:54:57.701600 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.701533    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 19:54:57.701600 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.701547    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 19:54:57.703056 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.702316    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dwwmk\""
Sep 09 19:54:57.703056 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.702348    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ntkpv"
Sep 09 19:54:57.703056 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.702498    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nn9df\""
Sep 09 19:54:57.703254 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.703201    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5ac07c25-2d29-4115-851c-1c3602361e6a-hostroot\") pod \"multus-xnmz6\" (UID: \"5ac07c25-2d29-4115-851c-1c3602361e6a\") " pod="openshift-multus/multus-xnmz6"
Sep 09 19:54:57.703254 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.703241    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5ac07c25-2d29-4115-851c-1c3602361e6a-host-run-multus-certs\") pod \"multus-xnmz6\" (UID: \"5ac07c25-2d29-4115-851c-1c3602361e6a\") " pod="openshift-multus/multus-xnmz6"
Sep 09 19:54:57.703353 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.703275    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ad886add-2c65-470b-8935-efe692cafcec-log-socket\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.703353 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.703307    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b2cc1cab-a99d-4733-8623-2ccf3fb89961-kubelet-config\") pod \"global-pull-secret-syncer-gn8cc\" (UID: \"b2cc1cab-a99d-4733-8623-2ccf3fb89961\") " pod="kube-system/global-pull-secret-syncer-gn8cc"
Sep 09 19:54:57.703353 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.703332    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8d8ec74b-cbcb-4c36-a904-21e6a6460f6b-serviceca\") pod \"node-ca-ctlrg\" (UID: \"8d8ec74b-cbcb-4c36-a904-21e6a6460f6b\") " pod="openshift-image-registry/node-ca-ctlrg"
Sep 09 19:54:57.703483 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.703390    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5ac07c25-2d29-4115-851c-1c3602361e6a-cni-binary-copy\") pod \"multus-xnmz6\" (UID: \"5ac07c25-2d29-4115-851c-1c3602361e6a\") " pod="openshift-multus/multus-xnmz6"
Sep 09 19:54:57.703535 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.703487    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5ac07c25-2d29-4115-851c-1c3602361e6a-host-run-k8s-cni-cncf-io\") pod \"multus-xnmz6\" (UID: \"5ac07c25-2d29-4115-851c-1c3602361e6a\") " pod="openshift-multus/multus-xnmz6"
Sep 09 19:54:57.703535 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.703526    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5ac07c25-2d29-4115-851c-1c3602361e6a-multus-daemon-config\") pod \"multus-xnmz6\" (UID: \"5ac07c25-2d29-4115-851c-1c3602361e6a\") " pod="openshift-multus/multus-xnmz6"
Sep 09 19:54:57.703624 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.703552    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8978aa77-ea38-48bd-b86f-fcaf6a8cfcbd-agent-certs\") pod \"konnectivity-agent-vkhvk\" (UID: \"8978aa77-ea38-48bd-b86f-fcaf6a8cfcbd\") " pod="kube-system/konnectivity-agent-vkhvk"
Sep 09 19:54:57.703624 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.703581    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ad886add-2c65-470b-8935-efe692cafcec-etc-openvswitch\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.703624 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.703612    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vq5l\" (UniqueName: \"kubernetes.io/projected/ad886add-2c65-470b-8935-efe692cafcec-kube-api-access-5vq5l\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.703752 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.703652    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 19:54:57.703752 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.703661    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4c8j\" (UniqueName: \"kubernetes.io/projected/8d8ec74b-cbcb-4c36-a904-21e6a6460f6b-kube-api-access-n4c8j\") pod \"node-ca-ctlrg\" (UID: \"8d8ec74b-cbcb-4c36-a904-21e6a6460f6b\") " pod="openshift-image-registry/node-ca-ctlrg"
Sep 09 19:54:57.703752 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.703694    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf8p6\" (UniqueName: \"kubernetes.io/projected/21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8-kube-api-access-jf8p6\") pod \"network-metrics-daemon-tmvpv\" (UID: \"21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8\") " pod="openshift-multus/network-metrics-daemon-tmvpv"
Sep 09 19:54:57.703752 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.703742    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ac07c25-2d29-4115-851c-1c3602361e6a-multus-cni-dir\") pod \"multus-xnmz6\" (UID: \"5ac07c25-2d29-4115-851c-1c3602361e6a\") " pod="openshift-multus/multus-xnmz6"
Sep 09 19:54:57.703958 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.703860    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5ac07c25-2d29-4115-851c-1c3602361e6a-host-var-lib-cni-bin\") pod \"multus-xnmz6\" (UID: \"5ac07c25-2d29-4115-851c-1c3602361e6a\") " pod="openshift-multus/multus-xnmz6"
Sep 09 19:54:57.703958 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.703906    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5ac07c25-2d29-4115-851c-1c3602361e6a-host-var-lib-cni-multus\") pod \"multus-xnmz6\" (UID: \"5ac07c25-2d29-4115-851c-1c3602361e6a\") " pod="openshift-multus/multus-xnmz6"
Sep 09 19:54:57.704117 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.703968    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad886add-2c65-470b-8935-efe692cafcec-host-kubelet\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.704117 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.704033    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ad886add-2c65-470b-8935-efe692cafcec-host-slash\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.704117 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.704072    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ad886add-2c65-470b-8935-efe692cafcec-run-ovn\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.704276 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.704114    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad886add-2c65-470b-8935-efe692cafcec-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.704276 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.704144    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ac07c25-2d29-4115-851c-1c3602361e6a-multus-conf-dir\") pod \"multus-xnmz6\" (UID: \"5ac07c25-2d29-4115-851c-1c3602361e6a\") " pod="openshift-multus/multus-xnmz6"
Sep 09 19:54:57.704380 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.704226    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ac07c25-2d29-4115-851c-1c3602361e6a-etc-kubernetes\") pod \"multus-xnmz6\" (UID: \"5ac07c25-2d29-4115-851c-1c3602361e6a\") " pod="openshift-multus/multus-xnmz6"
Sep 09 19:54:57.704380 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.704322    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k67zd\" (UniqueName: \"kubernetes.io/projected/5ac07c25-2d29-4115-851c-1c3602361e6a-kube-api-access-k67zd\") pod \"multus-xnmz6\" (UID: \"5ac07c25-2d29-4115-851c-1c3602361e6a\") " pod="openshift-multus/multus-xnmz6"
Sep 09 19:54:57.704380 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.704358    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngxjb\" (UniqueName: \"kubernetes.io/projected/8fe231fb-4f9c-478e-b2c5-95a65a627385-kube-api-access-ngxjb\") pod \"node-resolver-w2tb2\" (UID: \"8fe231fb-4f9c-478e-b2c5-95a65a627385\") " pod="openshift-dns/node-resolver-w2tb2"
Sep 09 19:54:57.704380 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.704325    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jd7rx\""
Sep 09 19:54:57.704572 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.704406    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfddc\" (UniqueName: \"kubernetes.io/projected/d5e04e1e-56ef-4e29-89bd-eafcf7425d56-kube-api-access-hfddc\") pod \"network-check-target-jcx8l\" (UID: \"d5e04e1e-56ef-4e29-89bd-eafcf7425d56\") " pod="openshift-network-diagnostics/network-check-target-jcx8l"
Sep 09 19:54:57.704572 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.704440    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ad886add-2c65-470b-8935-efe692cafcec-systemd-units\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.704572 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.704469    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ad886add-2c65-470b-8935-efe692cafcec-run-openvswitch\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.704572 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.704498    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ad886add-2c65-470b-8935-efe692cafcec-node-log\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.704572 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.704537    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad886add-2c65-470b-8935-efe692cafcec-host-run-ovn-kubernetes\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.704572 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.704565    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d8ec74b-cbcb-4c36-a904-21e6a6460f6b-host\") pod \"node-ca-ctlrg\" (UID: \"8d8ec74b-cbcb-4c36-a904-21e6a6460f6b\") " pod="openshift-image-registry/node-ca-ctlrg"
Sep 09 19:54:57.704851 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.704594    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5ac07c25-2d29-4115-851c-1c3602361e6a-host-run-netns\") pod \"multus-xnmz6\" (UID: \"5ac07c25-2d29-4115-851c-1c3602361e6a\") " pod="openshift-multus/multus-xnmz6"
Sep 09 19:54:57.704851 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.704639    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ad886add-2c65-470b-8935-efe692cafcec-host-cni-bin\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.704851 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.704686    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ad886add-2c65-470b-8935-efe692cafcec-host-cni-netd\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.704851 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.704691    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 19:54:57.704851 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.704729    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ad886add-2c65-470b-8935-efe692cafcec-ovnkube-config\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.704851 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.704800    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ad886add-2c65-470b-8935-efe692cafcec-ovn-node-metrics-cert\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.704851 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.704831    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ad886add-2c65-470b-8935-efe692cafcec-ovnkube-script-lib\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.705202 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.704875    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8fe231fb-4f9c-478e-b2c5-95a65a627385-hosts-file\") pod \"node-resolver-w2tb2\" (UID: \"8fe231fb-4f9c-478e-b2c5-95a65a627385\") " pod="openshift-dns/node-resolver-w2tb2"
Sep 09 19:54:57.705202 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.704967    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 19:54:57.705301 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.705240    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bpnzx\""
Sep 09 19:54:57.705686 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.705472    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fe231fb-4f9c-478e-b2c5-95a65a627385-tmp-dir\") pod \"node-resolver-w2tb2\" (UID: \"8fe231fb-4f9c-478e-b2c5-95a65a627385\") " pod="openshift-dns/node-resolver-w2tb2"
Sep 09 19:54:57.705686 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.705508    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8978aa77-ea38-48bd-b86f-fcaf6a8cfcbd-konnectivity-ca\") pod \"konnectivity-agent-vkhvk\" (UID: \"8978aa77-ea38-48bd-b86f-fcaf6a8cfcbd\") " pod="kube-system/konnectivity-agent-vkhvk"
Sep 09 19:54:57.705686 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.705527    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ad886add-2c65-470b-8935-efe692cafcec-host-run-netns\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.705686 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.705544    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ad886add-2c65-470b-8935-efe692cafcec-run-systemd\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.705686 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.705568    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ad886add-2c65-470b-8935-efe692cafcec-env-overrides\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.705686 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.705586    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8-metrics-certs\") pod \"network-metrics-daemon-tmvpv\" (UID: \"21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8\") " pod="openshift-multus/network-metrics-daemon-tmvpv"
Sep 09 19:54:57.705686 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.705600    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ac07c25-2d29-4115-851c-1c3602361e6a-system-cni-dir\") pod \"multus-xnmz6\" (UID: \"5ac07c25-2d29-4115-851c-1c3602361e6a\") " pod="openshift-multus/multus-xnmz6"
Sep 09 19:54:57.705686 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.705628    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ad886add-2c65-470b-8935-efe692cafcec-var-lib-openvswitch\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.705686 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.705652    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b2cc1cab-a99d-4733-8623-2ccf3fb89961-dbus\") pod \"global-pull-secret-syncer-gn8cc\" (UID: \"b2cc1cab-a99d-4733-8623-2ccf3fb89961\") " pod="kube-system/global-pull-secret-syncer-gn8cc"
Sep 09 19:54:57.706126 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.705670    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ac07c25-2d29-4115-851c-1c3602361e6a-cnibin\") pod \"multus-xnmz6\" (UID: \"5ac07c25-2d29-4115-851c-1c3602361e6a\") " pod="openshift-multus/multus-xnmz6"
Sep 09 19:54:57.706126 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.705845    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ac07c25-2d29-4115-851c-1c3602361e6a-host-var-lib-kubelet\") pod \"multus-xnmz6\" (UID: \"5ac07c25-2d29-4115-851c-1c3602361e6a\") " pod="openshift-multus/multus-xnmz6"
Sep 09 19:54:57.706126 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.705909    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2cc1cab-a99d-4733-8623-2ccf3fb89961-original-pull-secret\") pod \"global-pull-secret-syncer-gn8cc\" (UID: \"b2cc1cab-a99d-4733-8623-2ccf3fb89961\") " pod="kube-system/global-pull-secret-syncer-gn8cc"
Sep 09 19:54:57.706126 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.705939    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5ac07c25-2d29-4115-851c-1c3602361e6a-multus-socket-dir-parent\") pod \"multus-xnmz6\" (UID: \"5ac07c25-2d29-4115-851c-1c3602361e6a\") " pod="openshift-multus/multus-xnmz6"
Sep 09 19:54:57.706126 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.706061    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 19:54:57.706648 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.706430    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 19:54:57.713093 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.713074    2523 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 19:54:57.728446 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.728423    2523 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-6s76d"
Sep 09 19:54:57.734026 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.734007    2523 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6s76d"
Sep 09 19:54:57.794222 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.794199    2523 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 19:54:57.809588 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.806041    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfddc\" (UniqueName: \"kubernetes.io/projected/d5e04e1e-56ef-4e29-89bd-eafcf7425d56-kube-api-access-hfddc\") pod \"network-check-target-jcx8l\" (UID: \"d5e04e1e-56ef-4e29-89bd-eafcf7425d56\") " pod="openshift-network-diagnostics/network-check-target-jcx8l"
Sep 09 19:54:57.809588 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.806074    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ad886add-2c65-470b-8935-efe692cafcec-systemd-units\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.809588 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.806098    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad886add-2c65-470b-8935-efe692cafcec-host-run-ovn-kubernetes\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.809588 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.806139    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d8ec74b-cbcb-4c36-a904-21e6a6460f6b-host\") pod \"node-ca-ctlrg\" (UID: \"8d8ec74b-cbcb-4c36-a904-21e6a6460f6b\") " pod="openshift-image-registry/node-ca-ctlrg"
Sep 09 19:54:57.809588 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.806183    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5ac07c25-2d29-4115-851c-1c3602361e6a-host-run-netns\") pod \"multus-xnmz6\" (UID: \"5ac07c25-2d29-4115-851c-1c3602361e6a\") " pod="openshift-multus/multus-xnmz6"
Sep 09 19:54:57.809588 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.806196    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ad886add-2c65-470b-8935-efe692cafcec-systemd-units\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.809588 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.806214    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0eb194e9-a729-4845-86e2-e1011a9885c1-run\") pod \"tuned-ntkpv\" (UID: \"0eb194e9-a729-4845-86e2-e1011a9885c1\") " pod="openshift-cluster-node-tuning-operator/tuned-ntkpv"
Sep 09 19:54:57.809588 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.806228    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad886add-2c65-470b-8935-efe692cafcec-host-run-ovn-kubernetes\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.809588 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.806240    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0eb194e9-a729-4845-86e2-e1011a9885c1-var-lib-kubelet\") pod \"tuned-ntkpv\" (UID: \"0eb194e9-a729-4845-86e2-e1011a9885c1\") " pod="openshift-cluster-node-tuning-operator/tuned-ntkpv"
Sep 09 19:54:57.809588 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.806260    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d8ec74b-cbcb-4c36-a904-21e6a6460f6b-host\") pod \"node-ca-ctlrg\" (UID: \"8d8ec74b-cbcb-4c36-a904-21e6a6460f6b\") " pod="openshift-image-registry/node-ca-ctlrg"
Sep 09 19:54:57.809588 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.806269    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ad886add-2c65-470b-8935-efe692cafcec-host-cni-bin\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.809588 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.806273    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5ac07c25-2d29-4115-851c-1c3602361e6a-host-run-netns\") pod \"multus-xnmz6\" (UID: \"5ac07c25-2d29-4115-851c-1c3602361e6a\") " pod="openshift-multus/multus-xnmz6"
Sep 09 19:54:57.809588 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.806292    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fe231fb-4f9c-478e-b2c5-95a65a627385-tmp-dir\") pod \"node-resolver-w2tb2\" (UID: \"8fe231fb-4f9c-478e-b2c5-95a65a627385\") " pod="openshift-dns/node-resolver-w2tb2"
Sep 09 19:54:57.809588 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.806311    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ad886add-2c65-470b-8935-efe692cafcec-host-cni-bin\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.809588 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.806317    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/556a154a-15d2-4313-93b6-42580b64443c-cnibin\") pod \"multus-additional-cni-plugins-4jptv\" (UID: \"556a154a-15d2-4313-93b6-42580b64443c\") " pod="openshift-multus/multus-additional-cni-plugins-4jptv"
Sep 09 19:54:57.809588 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.806358    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0eb194e9-a729-4845-86e2-e1011a9885c1-etc-systemd\") pod \"tuned-ntkpv\" (UID: \"0eb194e9-a729-4845-86e2-e1011a9885c1\") " pod="openshift-cluster-node-tuning-operator/tuned-ntkpv"
Sep 09 19:54:57.809588 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.806382    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ad886add-2c65-470b-8935-efe692cafcec-host-run-netns\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.810464 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.806418    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ad886add-2c65-470b-8935-efe692cafcec-run-systemd\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.810464 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.806441    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ad886add-2c65-470b-8935-efe692cafcec-env-overrides\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.810464 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.806448    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ad886add-2c65-470b-8935-efe692cafcec-host-run-netns\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.810464 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.806464    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ac07c25-2d29-4115-851c-1c3602361e6a-system-cni-dir\") pod \"multus-xnmz6\" (UID: \"5ac07c25-2d29-4115-851c-1c3602361e6a\") " pod="openshift-multus/multus-xnmz6"
Sep 09 19:54:57.810464 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.806504    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ad886add-2c65-470b-8935-efe692cafcec-run-systemd\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.810464 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.806534    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1aa6d3e2-65bb-4992-9e19-e1825d7fe2aa-sys-fs\") pod \"aws-ebs-csi-driver-node-l55pn\" (UID: \"1aa6d3e2-65bb-4992-9e19-e1825d7fe2aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l55pn"
Sep 09 19:54:57.810464 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.806562    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ad886add-2c65-470b-8935-efe692cafcec-var-lib-openvswitch\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.810464 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.806570    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fe231fb-4f9c-478e-b2c5-95a65a627385-tmp-dir\") pod \"node-resolver-w2tb2\" (UID: \"8fe231fb-4f9c-478e-b2c5-95a65a627385\") " pod="openshift-dns/node-resolver-w2tb2"
Sep 09 19:54:57.810464 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.806584    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ac07c25-2d29-4115-851c-1c3602361e6a-system-cni-dir\") pod \"multus-xnmz6\" (UID: \"5ac07c25-2d29-4115-851c-1c3602361e6a\") " pod="openshift-multus/multus-xnmz6"
Sep 09 19:54:57.810464 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.806588    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/556a154a-15d2-4313-93b6-42580b64443c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4jptv\" (UID: \"556a154a-15d2-4313-93b6-42580b64443c\") " pod="openshift-multus/multus-additional-cni-plugins-4jptv"
Sep 09 19:54:57.810464 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.806628    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0eb194e9-a729-4845-86e2-e1011a9885c1-etc-sysctl-d\") pod \"tuned-ntkpv\" (UID: \"0eb194e9-a729-4845-86e2-e1011a9885c1\") " pod="openshift-cluster-node-tuning-operator/tuned-ntkpv"
Sep 09 19:54:57.810464 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.806634    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ad886add-2c65-470b-8935-efe692cafcec-var-lib-openvswitch\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.810464 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.806650    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0eb194e9-a729-4845-86e2-e1011a9885c1-host\") pod \"tuned-ntkpv\" (UID: \"0eb194e9-a729-4845-86e2-e1011a9885c1\") " pod="openshift-cluster-node-tuning-operator/tuned-ntkpv"
Sep 09 19:54:57.810464 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.806693    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2cc1cab-a99d-4733-8623-2ccf3fb89961-original-pull-secret\") pod \"global-pull-secret-syncer-gn8cc\" (UID: \"b2cc1cab-a99d-4733-8623-2ccf3fb89961\") " pod="kube-system/global-pull-secret-syncer-gn8cc"
Sep 09 19:54:57.810464 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.806720    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5ac07c25-2d29-4115-851c-1c3602361e6a-host-run-multus-certs\") pod \"multus-xnmz6\" (UID: \"5ac07c25-2d29-4115-851c-1c3602361e6a\") " pod="openshift-multus/multus-xnmz6"
Sep 09 19:54:57.810464 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.806744    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0eb194e9-a729-4845-86e2-e1011a9885c1-etc-sysctl-conf\") pod \"tuned-ntkpv\" (UID: \"0eb194e9-a729-4845-86e2-e1011a9885c1\") " pod="openshift-cluster-node-tuning-operator/tuned-ntkpv"
Sep 09 19:54:57.810464 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.806783    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0eb194e9-a729-4845-86e2-e1011a9885c1-sys\") pod \"tuned-ntkpv\" (UID: \"0eb194e9-a729-4845-86e2-e1011a9885c1\") " pod="openshift-cluster-node-tuning-operator/tuned-ntkpv"
Sep 09 19:54:57.811094 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.806810    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5ac07c25-2d29-4115-851c-1c3602361e6a-host-run-multus-certs\") pod \"multus-xnmz6\" (UID: \"5ac07c25-2d29-4115-851c-1c3602361e6a\") " pod="openshift-multus/multus-xnmz6"
Sep 09 19:54:57.811094 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.806813    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8d8ec74b-cbcb-4c36-a904-21e6a6460f6b-serviceca\") pod \"node-ca-ctlrg\" (UID: \"8d8ec74b-cbcb-4c36-a904-21e6a6460f6b\") " pod="openshift-image-registry/node-ca-ctlrg"
Sep 09 19:54:57.811094 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:57.806830    2523 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 19:54:57.811094 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.806844    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5ac07c25-2d29-4115-851c-1c3602361e6a-cni-binary-copy\") pod \"multus-xnmz6\" (UID: \"5ac07c25-2d29-4115-851c-1c3602361e6a\") " pod="openshift-multus/multus-xnmz6"
Sep 09 19:54:57.811094 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.806867    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5ac07c25-2d29-4115-851c-1c3602361e6a-multus-daemon-config\") pod \"multus-xnmz6\" (UID: \"5ac07c25-2d29-4115-851c-1c3602361e6a\") " pod="openshift-multus/multus-xnmz6"
Sep 09 19:54:57.811094 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:57.806915    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2cc1cab-a99d-4733-8623-2ccf3fb89961-original-pull-secret podName:b2cc1cab-a99d-4733-8623-2ccf3fb89961 nodeName:}" failed. No retries permitted until 2026-09-09 19:54:58.306881297 +0000 UTC m=+2.978900278 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b2cc1cab-a99d-4733-8623-2ccf3fb89961-original-pull-secret") pod "global-pull-secret-syncer-gn8cc" (UID: "b2cc1cab-a99d-4733-8623-2ccf3fb89961") : object "kube-system"/"original-pull-secret" not registered
Sep 09 19:54:57.811094 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.806933    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0eb194e9-a729-4845-86e2-e1011a9885c1-etc-kubernetes\") pod \"tuned-ntkpv\" (UID: \"0eb194e9-a729-4845-86e2-e1011a9885c1\") " pod="openshift-cluster-node-tuning-operator/tuned-ntkpv"
Sep 09 19:54:57.811094 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.806961    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8978aa77-ea38-48bd-b86f-fcaf6a8cfcbd-agent-certs\") pod \"konnectivity-agent-vkhvk\" (UID: \"8978aa77-ea38-48bd-b86f-fcaf6a8cfcbd\") " pod="kube-system/konnectivity-agent-vkhvk"
Sep 09 19:54:57.811094 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.806978    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ad886add-2c65-470b-8935-efe692cafcec-env-overrides\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.811094 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.806986    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ad886add-2c65-470b-8935-efe692cafcec-etc-openvswitch\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.811094 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.807015    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vq5l\" (UniqueName: \"kubernetes.io/projected/ad886add-2c65-470b-8935-efe692cafcec-kube-api-access-5vq5l\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.811094 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.807039    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf8p6\" (UniqueName: \"kubernetes.io/projected/21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8-kube-api-access-jf8p6\") pod \"network-metrics-daemon-tmvpv\" (UID: \"21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8\") " pod="openshift-multus/network-metrics-daemon-tmvpv"
Sep 09 19:54:57.811094 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.807062    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ac07c25-2d29-4115-851c-1c3602361e6a-multus-cni-dir\") pod \"multus-xnmz6\" (UID: \"5ac07c25-2d29-4115-851c-1c3602361e6a\") " pod="openshift-multus/multus-xnmz6"
Sep 09 19:54:57.811094 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.807085    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5ac07c25-2d29-4115-851c-1c3602361e6a-host-var-lib-cni-bin\") pod \"multus-xnmz6\" (UID: \"5ac07c25-2d29-4115-851c-1c3602361e6a\") " pod="openshift-multus/multus-xnmz6"
Sep 09 19:54:57.811094 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.807110    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5ac07c25-2d29-4115-851c-1c3602361e6a-host-var-lib-cni-multus\") pod \"multus-xnmz6\" (UID: \"5ac07c25-2d29-4115-851c-1c3602361e6a\") " pod="openshift-multus/multus-xnmz6"
Sep 09 19:54:57.811094 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.807136    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad886add-2c65-470b-8935-efe692cafcec-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.811094 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.807194    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ac07c25-2d29-4115-851c-1c3602361e6a-multus-conf-dir\") pod \"multus-xnmz6\" (UID: \"5ac07c25-2d29-4115-851c-1c3602361e6a\") " pod="openshift-multus/multus-xnmz6"
Sep 09 19:54:57.811701 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.807229    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ac07c25-2d29-4115-851c-1c3602361e6a-etc-kubernetes\") pod \"multus-xnmz6\" (UID: \"5ac07c25-2d29-4115-851c-1c3602361e6a\") " pod="openshift-multus/multus-xnmz6"
Sep 09 19:54:57.811701 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.807246    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8d8ec74b-cbcb-4c36-a904-21e6a6460f6b-serviceca\") pod \"node-ca-ctlrg\" (UID: \"8d8ec74b-cbcb-4c36-a904-21e6a6460f6b\") " pod="openshift-image-registry/node-ca-ctlrg"
Sep 09 19:54:57.811701 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.807253    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0eb194e9-a729-4845-86e2-e1011a9885c1-etc-modprobe-d\") pod \"tuned-ntkpv\" (UID: \"0eb194e9-a729-4845-86e2-e1011a9885c1\") " pod="openshift-cluster-node-tuning-operator/tuned-ntkpv"
Sep 09 19:54:57.811701 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.807276    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0eb194e9-a729-4845-86e2-e1011a9885c1-tmp\") pod \"tuned-ntkpv\" (UID: \"0eb194e9-a729-4845-86e2-e1011a9885c1\") " pod="openshift-cluster-node-tuning-operator/tuned-ntkpv"
Sep 09 19:54:57.811701 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.807309    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1aa6d3e2-65bb-4992-9e19-e1825d7fe2aa-registration-dir\") pod \"aws-ebs-csi-driver-node-l55pn\" (UID: \"1aa6d3e2-65bb-4992-9e19-e1825d7fe2aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l55pn"
Sep 09 19:54:57.811701 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.807339    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ad886add-2c65-470b-8935-efe692cafcec-run-openvswitch\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.811701 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.807364    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ad886add-2c65-470b-8935-efe692cafcec-node-log\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.811701 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.807389    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f6ccae8e-cfdd-44f3-8118-49e960b27d79-iptables-alerter-script\") pod \"iptables-alerter-m64jf\" (UID: \"f6ccae8e-cfdd-44f3-8118-49e960b27d79\") " pod="openshift-network-operator/iptables-alerter-m64jf"
Sep 09 19:54:57.811701 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.807411    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1aa6d3e2-65bb-4992-9e19-e1825d7fe2aa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l55pn\" (UID: \"1aa6d3e2-65bb-4992-9e19-e1825d7fe2aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l55pn"
Sep 09 19:54:57.811701 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.807365    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5ac07c25-2d29-4115-851c-1c3602361e6a-multus-daemon-config\") pod \"multus-xnmz6\" (UID: \"5ac07c25-2d29-4115-851c-1c3602361e6a\") " pod="openshift-multus/multus-xnmz6"
Sep 09 19:54:57.811701 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.807439    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5ac07c25-2d29-4115-851c-1c3602361e6a-cni-binary-copy\") pod \"multus-xnmz6\" (UID: \"5ac07c25-2d29-4115-851c-1c3602361e6a\") " pod="openshift-multus/multus-xnmz6"
Sep 09 19:54:57.811701 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.807444    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ad886add-2c65-470b-8935-efe692cafcec-host-cni-netd\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.811701 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.807515    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ad886add-2c65-470b-8935-efe692cafcec-run-openvswitch\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.811701 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.807528    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ad886add-2c65-470b-8935-efe692cafcec-ovnkube-config\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.811701 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.807543    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ad886add-2c65-470b-8935-efe692cafcec-etc-openvswitch\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.811701 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.807550    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ac07c25-2d29-4115-851c-1c3602361e6a-multus-conf-dir\") pod \"multus-xnmz6\" (UID: \"5ac07c25-2d29-4115-851c-1c3602361e6a\") " pod="openshift-multus/multus-xnmz6"
Sep 09 19:54:57.811701 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.807547    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ad886add-2c65-470b-8935-efe692cafcec-ovn-node-metrics-cert\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.812353 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.807551    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ac07c25-2d29-4115-851c-1c3602361e6a-etc-kubernetes\") pod \"multus-xnmz6\" (UID: \"5ac07c25-2d29-4115-851c-1c3602361e6a\") " pod="openshift-multus/multus-xnmz6"
Sep 09 19:54:57.812353 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.807515    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5ac07c25-2d29-4115-851c-1c3602361e6a-host-var-lib-cni-multus\") pod \"multus-xnmz6\" (UID: \"5ac07c25-2d29-4115-851c-1c3602361e6a\") " pod="openshift-multus/multus-xnmz6"
Sep 09 19:54:57.812353 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.807599    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ac07c25-2d29-4115-851c-1c3602361e6a-multus-cni-dir\") pod \"multus-xnmz6\" (UID: \"5ac07c25-2d29-4115-851c-1c3602361e6a\") " pod="openshift-multus/multus-xnmz6"
Sep 09 19:54:57.812353 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.807631    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad886add-2c65-470b-8935-efe692cafcec-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.812353 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.807668    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ad886add-2c65-470b-8935-efe692cafcec-ovnkube-script-lib\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.812353 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.807699    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8fe231fb-4f9c-478e-b2c5-95a65a627385-hosts-file\") pod \"node-resolver-w2tb2\" (UID: \"8fe231fb-4f9c-478e-b2c5-95a65a627385\") " pod="openshift-dns/node-resolver-w2tb2"
Sep 09 19:54:57.812353 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.807519    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ad886add-2c65-470b-8935-efe692cafcec-node-log\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.812353 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.807725    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8978aa77-ea38-48bd-b86f-fcaf6a8cfcbd-konnectivity-ca\") pod \"konnectivity-agent-vkhvk\" (UID: \"8978aa77-ea38-48bd-b86f-fcaf6a8cfcbd\") " pod="kube-system/konnectivity-agent-vkhvk"
Sep 09 19:54:57.812353 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.807765    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5ac07c25-2d29-4115-851c-1c3602361e6a-host-var-lib-cni-bin\") pod \"multus-xnmz6\" (UID: \"5ac07c25-2d29-4115-851c-1c3602361e6a\") " pod="openshift-multus/multus-xnmz6"
Sep 09 19:54:57.812353 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.807946    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ad886add-2c65-470b-8935-efe692cafcec-host-cni-netd\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.812353 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.807973    2523 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 19:54:57.812353 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.807984    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8-metrics-certs\") pod \"network-metrics-daemon-tmvpv\" (UID: \"21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8\") " pod="openshift-multus/network-metrics-daemon-tmvpv"
Sep 09 19:54:57.812353 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.808012    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0eb194e9-a729-4845-86e2-e1011a9885c1-etc-tuned\") pod \"tuned-ntkpv\" (UID: \"0eb194e9-a729-4845-86e2-e1011a9885c1\") " pod="openshift-cluster-node-tuning-operator/tuned-ntkpv"
Sep 09 19:54:57.812353 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.808035    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1aa6d3e2-65bb-4992-9e19-e1825d7fe2aa-socket-dir\") pod \"aws-ebs-csi-driver-node-l55pn\" (UID: \"1aa6d3e2-65bb-4992-9e19-e1825d7fe2aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l55pn"
Sep 09 19:54:57.812353 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.808065    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b2cc1cab-a99d-4733-8623-2ccf3fb89961-dbus\") pod \"global-pull-secret-syncer-gn8cc\" (UID: \"b2cc1cab-a99d-4733-8623-2ccf3fb89961\") " pod="kube-system/global-pull-secret-syncer-gn8cc"
Sep 09 19:54:57.812353 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.808137    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ac07c25-2d29-4115-851c-1c3602361e6a-cnibin\") pod \"multus-xnmz6\" (UID: \"5ac07c25-2d29-4115-851c-1c3602361e6a\") " pod="openshift-multus/multus-xnmz6"
Sep 09 19:54:57.812353 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.808141    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ad886add-2c65-470b-8935-efe692cafcec-ovnkube-config\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.812353 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.808184    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ac07c25-2d29-4115-851c-1c3602361e6a-host-var-lib-kubelet\") pod \"multus-xnmz6\" (UID: \"5ac07c25-2d29-4115-851c-1c3602361e6a\") " pod="openshift-multus/multus-xnmz6"
Sep 09 19:54:57.813229 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.808212    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/556a154a-15d2-4313-93b6-42580b64443c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4jptv\" (UID: \"556a154a-15d2-4313-93b6-42580b64443c\") " pod="openshift-multus/multus-additional-cni-plugins-4jptv"
Sep 09 19:54:57.813229 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.808237    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0eb194e9-a729-4845-86e2-e1011a9885c1-lib-modules\") pod \"tuned-ntkpv\" (UID: \"0eb194e9-a729-4845-86e2-e1011a9885c1\") " pod="openshift-cluster-node-tuning-operator/tuned-ntkpv"
Sep 09 19:54:57.813229 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.808262    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k95m\" (UniqueName: \"kubernetes.io/projected/0eb194e9-a729-4845-86e2-e1011a9885c1-kube-api-access-7k95m\") pod \"tuned-ntkpv\" (UID: \"0eb194e9-a729-4845-86e2-e1011a9885c1\") " pod="openshift-cluster-node-tuning-operator/tuned-ntkpv"
Sep 09 19:54:57.813229 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.808291    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5ac07c25-2d29-4115-851c-1c3602361e6a-multus-socket-dir-parent\") pod \"multus-xnmz6\" (UID: \"5ac07c25-2d29-4115-851c-1c3602361e6a\") " pod="openshift-multus/multus-xnmz6"
Sep 09 19:54:57.813229 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.808315    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5ac07c25-2d29-4115-851c-1c3602361e6a-hostroot\") pod \"multus-xnmz6\" (UID: \"5ac07c25-2d29-4115-851c-1c3602361e6a\") " pod="openshift-multus/multus-xnmz6"
Sep 09 19:54:57.813229 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.808237    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8fe231fb-4f9c-478e-b2c5-95a65a627385-hosts-file\") pod \"node-resolver-w2tb2\" (UID: \"8fe231fb-4f9c-478e-b2c5-95a65a627385\") " pod="openshift-dns/node-resolver-w2tb2"
Sep 09 19:54:57.813229 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.808361    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-226.ec2.internal" event={"ID":"75ee1d831c2282f17a9072ea7f2d2163","Type":"ContainerStarted","Data":"021edc0852545ebf1d36d9248ab5f9ef8958182b99eb9a35f0dcf49b30194465"}
Sep 09 19:54:57.813229 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.808391    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ad886add-2c65-470b-8935-efe692cafcec-ovnkube-script-lib\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.813229 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.808391    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/556a154a-15d2-4313-93b6-42580b64443c-cni-binary-copy\") pod \"multus-additional-cni-plugins-4jptv\" (UID: \"556a154a-15d2-4313-93b6-42580b64443c\") " pod="openshift-multus/multus-additional-cni-plugins-4jptv"
Sep 09 19:54:57.813229 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.808427    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/556a154a-15d2-4313-93b6-42580b64443c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4jptv\" (UID: \"556a154a-15d2-4313-93b6-42580b64443c\") " pod="openshift-multus/multus-additional-cni-plugins-4jptv"
Sep 09 19:54:57.813229 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.808426    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5ac07c25-2d29-4115-851c-1c3602361e6a-hostroot\") pod \"multus-xnmz6\" (UID: \"5ac07c25-2d29-4115-851c-1c3602361e6a\") " pod="openshift-multus/multus-xnmz6"
Sep 09 19:54:57.813229 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.808455    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0eb194e9-a729-4845-86e2-e1011a9885c1-etc-sysconfig\") pod \"tuned-ntkpv\" (UID: \"0eb194e9-a729-4845-86e2-e1011a9885c1\") " pod="openshift-cluster-node-tuning-operator/tuned-ntkpv"
Sep 09 19:54:57.813229 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.808469    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ac07c25-2d29-4115-851c-1c3602361e6a-cnibin\") pod \"multus-xnmz6\" (UID: \"5ac07c25-2d29-4115-851c-1c3602361e6a\") " pod="openshift-multus/multus-xnmz6"
Sep 09 19:54:57.813229 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.808479    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ac07c25-2d29-4115-851c-1c3602361e6a-host-var-lib-kubelet\") pod \"multus-xnmz6\" (UID: \"5ac07c25-2d29-4115-851c-1c3602361e6a\") " pod="openshift-multus/multus-xnmz6"
Sep 09 19:54:57.813229 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.808480    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgm4s\" (UniqueName: \"kubernetes.io/projected/f6ccae8e-cfdd-44f3-8118-49e960b27d79-kube-api-access-wgm4s\") pod \"iptables-alerter-m64jf\" (UID: \"f6ccae8e-cfdd-44f3-8118-49e960b27d79\") " pod="openshift-network-operator/iptables-alerter-m64jf"
Sep 09 19:54:57.813229 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.808513    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ad886add-2c65-470b-8935-efe692cafcec-log-socket\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.813229 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.808516    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b2cc1cab-a99d-4733-8623-2ccf3fb89961-dbus\") pod \"global-pull-secret-syncer-gn8cc\" (UID: \"b2cc1cab-a99d-4733-8623-2ccf3fb89961\") " pod="kube-system/global-pull-secret-syncer-gn8cc"
Sep 09 19:54:57.813810 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.808542    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b2cc1cab-a99d-4733-8623-2ccf3fb89961-kubelet-config\") pod \"global-pull-secret-syncer-gn8cc\" (UID: \"b2cc1cab-a99d-4733-8623-2ccf3fb89961\") " pod="kube-system/global-pull-secret-syncer-gn8cc"
Sep 09 19:54:57.813810 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.808569    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5ac07c25-2d29-4115-851c-1c3602361e6a-host-run-k8s-cni-cncf-io\") pod \"multus-xnmz6\" (UID: \"5ac07c25-2d29-4115-851c-1c3602361e6a\") " pod="openshift-multus/multus-xnmz6"
Sep 09 19:54:57.813810 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.808616    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ad886add-2c65-470b-8935-efe692cafcec-log-socket\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.813810 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:57.808619    2523 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:54:57.813810 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.808659    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5ac07c25-2d29-4115-851c-1c3602361e6a-multus-socket-dir-parent\") pod \"multus-xnmz6\" (UID: \"5ac07c25-2d29-4115-851c-1c3602361e6a\") " pod="openshift-multus/multus-xnmz6"
Sep 09 19:54:57.813810 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.808848    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1aa6d3e2-65bb-4992-9e19-e1825d7fe2aa-device-dir\") pod \"aws-ebs-csi-driver-node-l55pn\" (UID: \"1aa6d3e2-65bb-4992-9e19-e1825d7fe2aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l55pn"
Sep 09 19:54:57.813810 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.808874    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1aa6d3e2-65bb-4992-9e19-e1825d7fe2aa-etc-selinux\") pod \"aws-ebs-csi-driver-node-l55pn\" (UID: \"1aa6d3e2-65bb-4992-9e19-e1825d7fe2aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l55pn"
Sep 09 19:54:57.813810 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.808899    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/556a154a-15d2-4313-93b6-42580b64443c-system-cni-dir\") pod \"multus-additional-cni-plugins-4jptv\" (UID: \"556a154a-15d2-4313-93b6-42580b64443c\") " pod="openshift-multus/multus-additional-cni-plugins-4jptv"
Sep 09 19:54:57.813810 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.808929    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4c8j\" (UniqueName: \"kubernetes.io/projected/8d8ec74b-cbcb-4c36-a904-21e6a6460f6b-kube-api-access-n4c8j\") pod \"node-ca-ctlrg\" (UID: \"8d8ec74b-cbcb-4c36-a904-21e6a6460f6b\") " pod="openshift-image-registry/node-ca-ctlrg"
Sep 09 19:54:57.813810 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.808950    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8978aa77-ea38-48bd-b86f-fcaf6a8cfcbd-konnectivity-ca\") pod \"konnectivity-agent-vkhvk\" (UID: \"8978aa77-ea38-48bd-b86f-fcaf6a8cfcbd\") " pod="kube-system/konnectivity-agent-vkhvk"
Sep 09 19:54:57.813810 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.809006    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k67zd\" (UniqueName: \"kubernetes.io/projected/5ac07c25-2d29-4115-851c-1c3602361e6a-kube-api-access-k67zd\") pod \"multus-xnmz6\" (UID: \"5ac07c25-2d29-4115-851c-1c3602361e6a\") " pod="openshift-multus/multus-xnmz6"
Sep 09 19:54:57.813810 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.809033    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngxjb\" (UniqueName: \"kubernetes.io/projected/8fe231fb-4f9c-478e-b2c5-95a65a627385-kube-api-access-ngxjb\") pod \"node-resolver-w2tb2\" (UID: \"8fe231fb-4f9c-478e-b2c5-95a65a627385\") " pod="openshift-dns/node-resolver-w2tb2"
Sep 09 19:54:57.813810 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.809058    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad886add-2c65-470b-8935-efe692cafcec-host-kubelet\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.813810 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.809086    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ad886add-2c65-470b-8935-efe692cafcec-host-slash\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.813810 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.809109    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ad886add-2c65-470b-8935-efe692cafcec-run-ovn\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.813810 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.809162    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2n7x\" (UniqueName: \"kubernetes.io/projected/556a154a-15d2-4313-93b6-42580b64443c-kube-api-access-l2n7x\") pod \"multus-additional-cni-plugins-4jptv\" (UID: \"556a154a-15d2-4313-93b6-42580b64443c\") " pod="openshift-multus/multus-additional-cni-plugins-4jptv"
Sep 09 19:54:57.813810 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.809288    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b2cc1cab-a99d-4733-8623-2ccf3fb89961-kubelet-config\") pod \"global-pull-secret-syncer-gn8cc\" (UID: \"b2cc1cab-a99d-4733-8623-2ccf3fb89961\") " pod="kube-system/global-pull-secret-syncer-gn8cc"
Sep 09 19:54:57.814476 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:57.809336    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8-metrics-certs podName:21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8 nodeName:}" failed. No retries permitted until 2026-09-09 19:54:58.309320903 +0000 UTC m=+2.981339883 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8-metrics-certs") pod "network-metrics-daemon-tmvpv" (UID: "21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:54:57.814476 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.809359    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6ccae8e-cfdd-44f3-8118-49e960b27d79-host-slash\") pod \"iptables-alerter-m64jf\" (UID: \"f6ccae8e-cfdd-44f3-8118-49e960b27d79\") " pod="openshift-network-operator/iptables-alerter-m64jf"
Sep 09 19:54:57.814476 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.809368    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ad886add-2c65-470b-8935-efe692cafcec-host-slash\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.814476 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.809391    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66mmk\" (UniqueName: \"kubernetes.io/projected/1aa6d3e2-65bb-4992-9e19-e1825d7fe2aa-kube-api-access-66mmk\") pod \"aws-ebs-csi-driver-node-l55pn\" (UID: \"1aa6d3e2-65bb-4992-9e19-e1825d7fe2aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l55pn"
Sep 09 19:54:57.814476 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.809402    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ad886add-2c65-470b-8935-efe692cafcec-run-ovn\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.814476 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.809421    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5ac07c25-2d29-4115-851c-1c3602361e6a-host-run-k8s-cni-cncf-io\") pod \"multus-xnmz6\" (UID: \"5ac07c25-2d29-4115-851c-1c3602361e6a\") " pod="openshift-multus/multus-xnmz6"
Sep 09 19:54:57.814476 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.809559    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad886add-2c65-470b-8935-efe692cafcec-host-kubelet\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.814476 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.809725    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-226.ec2.internal" event={"ID":"f7c24ed75a93d3886eeb14da20c53cdf","Type":"ContainerStarted","Data":"93d76232826c4a4694d170d69cc9e3cff8ccf67614b3bf9653b2ee59192e73d4"}
Sep 09 19:54:57.814476 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.812465    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ad886add-2c65-470b-8935-efe692cafcec-ovn-node-metrics-cert\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.814476 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.812579    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8978aa77-ea38-48bd-b86f-fcaf6a8cfcbd-agent-certs\") pod \"konnectivity-agent-vkhvk\" (UID: \"8978aa77-ea38-48bd-b86f-fcaf6a8cfcbd\") " pod="kube-system/konnectivity-agent-vkhvk"
Sep 09 19:54:57.819819 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:57.819584    2523 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 19:54:57.819819 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:57.819611    2523 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 19:54:57.819819 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:57.819652    2523 projected.go:196] Error preparing data for projected volume kube-api-access-hfddc for pod openshift-network-diagnostics/network-check-target-jcx8l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:54:57.819819 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:57.819703    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5e04e1e-56ef-4e29-89bd-eafcf7425d56-kube-api-access-hfddc podName:d5e04e1e-56ef-4e29-89bd-eafcf7425d56 nodeName:}" failed. No retries permitted until 2026-09-09 19:54:58.319687051 +0000 UTC m=+2.991706042 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hfddc" (UniqueName: "kubernetes.io/projected/d5e04e1e-56ef-4e29-89bd-eafcf7425d56-kube-api-access-hfddc") pod "network-check-target-jcx8l" (UID: "d5e04e1e-56ef-4e29-89bd-eafcf7425d56") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:54:57.822804 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.822525    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4c8j\" (UniqueName: \"kubernetes.io/projected/8d8ec74b-cbcb-4c36-a904-21e6a6460f6b-kube-api-access-n4c8j\") pod \"node-ca-ctlrg\" (UID: \"8d8ec74b-cbcb-4c36-a904-21e6a6460f6b\") " pod="openshift-image-registry/node-ca-ctlrg"
Sep 09 19:54:57.823364 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.823346    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf8p6\" (UniqueName: \"kubernetes.io/projected/21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8-kube-api-access-jf8p6\") pod \"network-metrics-daemon-tmvpv\" (UID: \"21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8\") " pod="openshift-multus/network-metrics-daemon-tmvpv"
Sep 09 19:54:57.824535 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.824510    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngxjb\" (UniqueName: \"kubernetes.io/projected/8fe231fb-4f9c-478e-b2c5-95a65a627385-kube-api-access-ngxjb\") pod \"node-resolver-w2tb2\" (UID: \"8fe231fb-4f9c-478e-b2c5-95a65a627385\") " pod="openshift-dns/node-resolver-w2tb2"
Sep 09 19:54:57.824798 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.824778    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vq5l\" (UniqueName: \"kubernetes.io/projected/ad886add-2c65-470b-8935-efe692cafcec-kube-api-access-5vq5l\") pod \"ovnkube-node-c72g2\" (UID: \"ad886add-2c65-470b-8935-efe692cafcec\") " pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:57.827261 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.827239    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k67zd\" (UniqueName: \"kubernetes.io/projected/5ac07c25-2d29-4115-851c-1c3602361e6a-kube-api-access-k67zd\") pod \"multus-xnmz6\" (UID: \"5ac07c25-2d29-4115-851c-1c3602361e6a\") " pod="openshift-multus/multus-xnmz6"
Sep 09 19:54:57.910016 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.909982    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2n7x\" (UniqueName: \"kubernetes.io/projected/556a154a-15d2-4313-93b6-42580b64443c-kube-api-access-l2n7x\") pod \"multus-additional-cni-plugins-4jptv\" (UID: \"556a154a-15d2-4313-93b6-42580b64443c\") " pod="openshift-multus/multus-additional-cni-plugins-4jptv"
Sep 09 19:54:57.910184 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.910027    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6ccae8e-cfdd-44f3-8118-49e960b27d79-host-slash\") pod \"iptables-alerter-m64jf\" (UID: \"f6ccae8e-cfdd-44f3-8118-49e960b27d79\") " pod="openshift-network-operator/iptables-alerter-m64jf"
Sep 09 19:54:57.910184 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.910054    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66mmk\" (UniqueName: \"kubernetes.io/projected/1aa6d3e2-65bb-4992-9e19-e1825d7fe2aa-kube-api-access-66mmk\") pod \"aws-ebs-csi-driver-node-l55pn\" (UID: \"1aa6d3e2-65bb-4992-9e19-e1825d7fe2aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l55pn"
Sep 09 19:54:57.910184 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.910091    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0eb194e9-a729-4845-86e2-e1011a9885c1-run\") pod \"tuned-ntkpv\" (UID: \"0eb194e9-a729-4845-86e2-e1011a9885c1\") " pod="openshift-cluster-node-tuning-operator/tuned-ntkpv"
Sep 09 19:54:57.910184 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.910115    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0eb194e9-a729-4845-86e2-e1011a9885c1-var-lib-kubelet\") pod \"tuned-ntkpv\" (UID: \"0eb194e9-a729-4845-86e2-e1011a9885c1\") " pod="openshift-cluster-node-tuning-operator/tuned-ntkpv"
Sep 09 19:54:57.910184 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.910140    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/556a154a-15d2-4313-93b6-42580b64443c-cnibin\") pod \"multus-additional-cni-plugins-4jptv\" (UID: \"556a154a-15d2-4313-93b6-42580b64443c\") " pod="openshift-multus/multus-additional-cni-plugins-4jptv"
Sep 09 19:54:57.910184 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.910182    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0eb194e9-a729-4845-86e2-e1011a9885c1-etc-systemd\") pod \"tuned-ntkpv\" (UID: \"0eb194e9-a729-4845-86e2-e1011a9885c1\") " pod="openshift-cluster-node-tuning-operator/tuned-ntkpv"
Sep 09 19:54:57.910391 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.910209    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1aa6d3e2-65bb-4992-9e19-e1825d7fe2aa-sys-fs\") pod \"aws-ebs-csi-driver-node-l55pn\" (UID: \"1aa6d3e2-65bb-4992-9e19-e1825d7fe2aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l55pn"
Sep 09 19:54:57.910391 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.910233    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/556a154a-15d2-4313-93b6-42580b64443c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4jptv\" (UID: \"556a154a-15d2-4313-93b6-42580b64443c\") " pod="openshift-multus/multus-additional-cni-plugins-4jptv"
Sep 09 19:54:57.910391 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.910258    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0eb194e9-a729-4845-86e2-e1011a9885c1-etc-sysctl-d\") pod \"tuned-ntkpv\" (UID: \"0eb194e9-a729-4845-86e2-e1011a9885c1\") " pod="openshift-cluster-node-tuning-operator/tuned-ntkpv"
Sep 09 19:54:57.910391 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.910279    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0eb194e9-a729-4845-86e2-e1011a9885c1-host\") pod \"tuned-ntkpv\" (UID: \"0eb194e9-a729-4845-86e2-e1011a9885c1\") " pod="openshift-cluster-node-tuning-operator/tuned-ntkpv"
Sep 09 19:54:57.910391 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.910294    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0eb194e9-a729-4845-86e2-e1011a9885c1-var-lib-kubelet\") pod \"tuned-ntkpv\" (UID: \"0eb194e9-a729-4845-86e2-e1011a9885c1\") " pod="openshift-cluster-node-tuning-operator/tuned-ntkpv"
Sep 09 19:54:57.910391 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.910312    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0eb194e9-a729-4845-86e2-e1011a9885c1-etc-sysctl-conf\") pod \"tuned-ntkpv\" (UID: \"0eb194e9-a729-4845-86e2-e1011a9885c1\") " pod="openshift-cluster-node-tuning-operator/tuned-ntkpv"
Sep 09 19:54:57.910391 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.910334    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0eb194e9-a729-4845-86e2-e1011a9885c1-sys\") pod \"tuned-ntkpv\" (UID: \"0eb194e9-a729-4845-86e2-e1011a9885c1\") " pod="openshift-cluster-node-tuning-operator/tuned-ntkpv"
Sep 09 19:54:57.910391 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.910345    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6ccae8e-cfdd-44f3-8118-49e960b27d79-host-slash\") pod \"iptables-alerter-m64jf\" (UID: \"f6ccae8e-cfdd-44f3-8118-49e960b27d79\") " pod="openshift-network-operator/iptables-alerter-m64jf"
Sep 09 19:54:57.910391 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.910372    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0eb194e9-a729-4845-86e2-e1011a9885c1-etc-kubernetes\") pod \"tuned-ntkpv\" (UID: \"0eb194e9-a729-4845-86e2-e1011a9885c1\") " pod="openshift-cluster-node-tuning-operator/tuned-ntkpv"
Sep 09 19:54:57.910391 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.910392    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0eb194e9-a729-4845-86e2-e1011a9885c1-etc-modprobe-d\") pod \"tuned-ntkpv\" (UID: \"0eb194e9-a729-4845-86e2-e1011a9885c1\") " pod="openshift-cluster-node-tuning-operator/tuned-ntkpv"
Sep 09 19:54:57.910667 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.910411    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0eb194e9-a729-4845-86e2-e1011a9885c1-tmp\") pod \"tuned-ntkpv\" (UID: \"0eb194e9-a729-4845-86e2-e1011a9885c1\") " pod="openshift-cluster-node-tuning-operator/tuned-ntkpv"
Sep 09 19:54:57.910667 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.910424    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1aa6d3e2-65bb-4992-9e19-e1825d7fe2aa-registration-dir\") pod \"aws-ebs-csi-driver-node-l55pn\" (UID: \"1aa6d3e2-65bb-4992-9e19-e1825d7fe2aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l55pn"
Sep 09 19:54:57.910667 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.910442    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f6ccae8e-cfdd-44f3-8118-49e960b27d79-iptables-alerter-script\") pod \"iptables-alerter-m64jf\" (UID: \"f6ccae8e-cfdd-44f3-8118-49e960b27d79\") " pod="openshift-network-operator/iptables-alerter-m64jf"
Sep 09 19:54:57.910667 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.910444    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0eb194e9-a729-4845-86e2-e1011a9885c1-run\") pod \"tuned-ntkpv\" (UID: \"0eb194e9-a729-4845-86e2-e1011a9885c1\") " pod="openshift-cluster-node-tuning-operator/tuned-ntkpv"
Sep 09 19:54:57.910667 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.910459    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1aa6d3e2-65bb-4992-9e19-e1825d7fe2aa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l55pn\" (UID: \"1aa6d3e2-65bb-4992-9e19-e1825d7fe2aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l55pn"
Sep 09 19:54:57.910667 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.910508    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1aa6d3e2-65bb-4992-9e19-e1825d7fe2aa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l55pn\" (UID: \"1aa6d3e2-65bb-4992-9e19-e1825d7fe2aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l55pn"
Sep 09 19:54:57.910667 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.910543    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0eb194e9-a729-4845-86e2-e1011a9885c1-etc-sysctl-conf\") pod \"tuned-ntkpv\" (UID: \"0eb194e9-a729-4845-86e2-e1011a9885c1\") " pod="openshift-cluster-node-tuning-operator/tuned-ntkpv"
Sep 09 19:54:57.910667 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.910573    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0eb194e9-a729-4845-86e2-e1011a9885c1-etc-tuned\") pod \"tuned-ntkpv\" (UID: \"0eb194e9-a729-4845-86e2-e1011a9885c1\") " pod="openshift-cluster-node-tuning-operator/tuned-ntkpv"
Sep 09 19:54:57.910667 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.910589    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1aa6d3e2-65bb-4992-9e19-e1825d7fe2aa-socket-dir\") pod \"aws-ebs-csi-driver-node-l55pn\" (UID: \"1aa6d3e2-65bb-4992-9e19-e1825d7fe2aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l55pn"
Sep 09 19:54:57.910667 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.910608    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/556a154a-15d2-4313-93b6-42580b64443c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4jptv\" (UID: \"556a154a-15d2-4313-93b6-42580b64443c\") " pod="openshift-multus/multus-additional-cni-plugins-4jptv"
Sep 09 19:54:57.910667 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.910608    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0eb194e9-a729-4845-86e2-e1011a9885c1-etc-modprobe-d\") pod \"tuned-ntkpv\" (UID: \"0eb194e9-a729-4845-86e2-e1011a9885c1\") " pod="openshift-cluster-node-tuning-operator/tuned-ntkpv"
Sep 09 19:54:57.910667 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.910623    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0eb194e9-a729-4845-86e2-e1011a9885c1-lib-modules\") pod \"tuned-ntkpv\" (UID: \"0eb194e9-a729-4845-86e2-e1011a9885c1\") " pod="openshift-cluster-node-tuning-operator/tuned-ntkpv"
Sep 09 19:54:57.910667 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.910639    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k95m\" (UniqueName: \"kubernetes.io/projected/0eb194e9-a729-4845-86e2-e1011a9885c1-kube-api-access-7k95m\") pod \"tuned-ntkpv\" (UID: \"0eb194e9-a729-4845-86e2-e1011a9885c1\") " pod="openshift-cluster-node-tuning-operator/tuned-ntkpv"
Sep 09 19:54:57.910667 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.910655    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/556a154a-15d2-4313-93b6-42580b64443c-cni-binary-copy\") pod \"multus-additional-cni-plugins-4jptv\" (UID: \"556a154a-15d2-4313-93b6-42580b64443c\") " pod="openshift-multus/multus-additional-cni-plugins-4jptv"
Sep 09 19:54:57.910667 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.910667    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0eb194e9-a729-4845-86e2-e1011a9885c1-etc-kubernetes\") pod \"tuned-ntkpv\" (UID: \"0eb194e9-a729-4845-86e2-e1011a9885c1\") " pod="openshift-cluster-node-tuning-operator/tuned-ntkpv"
Sep 09 19:54:57.911349 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.910670    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/556a154a-15d2-4313-93b6-42580b64443c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4jptv\" (UID: \"556a154a-15d2-4313-93b6-42580b64443c\") " pod="openshift-multus/multus-additional-cni-plugins-4jptv"
Sep 09 19:54:57.911349 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.910704    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0eb194e9-a729-4845-86e2-e1011a9885c1-etc-sysconfig\") pod \"tuned-ntkpv\" (UID: \"0eb194e9-a729-4845-86e2-e1011a9885c1\") " pod="openshift-cluster-node-tuning-operator/tuned-ntkpv"
Sep 09 19:54:57.911349 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.910728    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgm4s\" (UniqueName: \"kubernetes.io/projected/f6ccae8e-cfdd-44f3-8118-49e960b27d79-kube-api-access-wgm4s\") pod \"iptables-alerter-m64jf\" (UID: \"f6ccae8e-cfdd-44f3-8118-49e960b27d79\") " pod="openshift-network-operator/iptables-alerter-m64jf"
Sep 09 19:54:57.911349 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.910757    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1aa6d3e2-65bb-4992-9e19-e1825d7fe2aa-device-dir\") pod \"aws-ebs-csi-driver-node-l55pn\" (UID: \"1aa6d3e2-65bb-4992-9e19-e1825d7fe2aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l55pn"
Sep 09 19:54:57.911349 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.910780    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1aa6d3e2-65bb-4992-9e19-e1825d7fe2aa-etc-selinux\") pod \"aws-ebs-csi-driver-node-l55pn\" (UID: \"1aa6d3e2-65bb-4992-9e19-e1825d7fe2aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l55pn"
Sep 09 19:54:57.911349 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.910796    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/556a154a-15d2-4313-93b6-42580b64443c-system-cni-dir\") pod \"multus-additional-cni-plugins-4jptv\" (UID: \"556a154a-15d2-4313-93b6-42580b64443c\") " pod="openshift-multus/multus-additional-cni-plugins-4jptv"
Sep 09 19:54:57.911349 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.910852    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/556a154a-15d2-4313-93b6-42580b64443c-system-cni-dir\") pod \"multus-additional-cni-plugins-4jptv\" (UID: \"556a154a-15d2-4313-93b6-42580b64443c\") " pod="openshift-multus/multus-additional-cni-plugins-4jptv"
Sep 09 19:54:57.911349 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.910886    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0eb194e9-a729-4845-86e2-e1011a9885c1-etc-sysconfig\") pod \"tuned-ntkpv\" (UID: \"0eb194e9-a729-4845-86e2-e1011a9885c1\") " pod="openshift-cluster-node-tuning-operator/tuned-ntkpv"
Sep 09 19:54:57.911349 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.911063    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1aa6d3e2-65bb-4992-9e19-e1825d7fe2aa-device-dir\") pod \"aws-ebs-csi-driver-node-l55pn\" (UID: \"1aa6d3e2-65bb-4992-9e19-e1825d7fe2aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l55pn"
Sep 09 19:54:57.911349 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.911095    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1aa6d3e2-65bb-4992-9e19-e1825d7fe2aa-etc-selinux\") pod \"aws-ebs-csi-driver-node-l55pn\" (UID: \"1aa6d3e2-65bb-4992-9e19-e1825d7fe2aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l55pn"
Sep 09 19:54:57.911349 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.911143    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/556a154a-15d2-4313-93b6-42580b64443c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4jptv\" (UID: \"556a154a-15d2-4313-93b6-42580b64443c\") " pod="openshift-multus/multus-additional-cni-plugins-4jptv"
Sep 09 19:54:57.911349 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.911275    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1aa6d3e2-65bb-4992-9e19-e1825d7fe2aa-socket-dir\") pod \"aws-ebs-csi-driver-node-l55pn\" (UID: \"1aa6d3e2-65bb-4992-9e19-e1825d7fe2aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l55pn"
Sep 09 19:54:57.911896 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.911379    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1aa6d3e2-65bb-4992-9e19-e1825d7fe2aa-registration-dir\") pod \"aws-ebs-csi-driver-node-l55pn\" (UID: \"1aa6d3e2-65bb-4992-9e19-e1825d7fe2aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l55pn"
Sep 09 19:54:57.911896 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.911502    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1aa6d3e2-65bb-4992-9e19-e1825d7fe2aa-sys-fs\") pod \"aws-ebs-csi-driver-node-l55pn\" (UID: \"1aa6d3e2-65bb-4992-9e19-e1825d7fe2aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l55pn"
Sep 09 19:54:57.911896 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.911550    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/556a154a-15d2-4313-93b6-42580b64443c-cnibin\") pod \"multus-additional-cni-plugins-4jptv\" (UID: \"556a154a-15d2-4313-93b6-42580b64443c\") " pod="openshift-multus/multus-additional-cni-plugins-4jptv"
Sep 09 19:54:57.911896 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.911592    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0eb194e9-a729-4845-86e2-e1011a9885c1-etc-systemd\") pod \"tuned-ntkpv\" (UID: \"0eb194e9-a729-4845-86e2-e1011a9885c1\") " pod="openshift-cluster-node-tuning-operator/tuned-ntkpv"
Sep 09 19:54:57.911896 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.911636    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0eb194e9-a729-4845-86e2-e1011a9885c1-sys\") pod \"tuned-ntkpv\" (UID: \"0eb194e9-a729-4845-86e2-e1011a9885c1\") " pod="openshift-cluster-node-tuning-operator/tuned-ntkpv"
Sep 09 19:54:57.911896 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.911690    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/556a154a-15d2-4313-93b6-42580b64443c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4jptv\" (UID: \"556a154a-15d2-4313-93b6-42580b64443c\") " pod="openshift-multus/multus-additional-cni-plugins-4jptv"
Sep 09 19:54:57.911896 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.911709    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0eb194e9-a729-4845-86e2-e1011a9885c1-lib-modules\") pod \"tuned-ntkpv\" (UID: \"0eb194e9-a729-4845-86e2-e1011a9885c1\") " pod="openshift-cluster-node-tuning-operator/tuned-ntkpv"
Sep 09 19:54:57.912236 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.911926    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0eb194e9-a729-4845-86e2-e1011a9885c1-etc-sysctl-d\") pod \"tuned-ntkpv\" (UID: \"0eb194e9-a729-4845-86e2-e1011a9885c1\") " pod="openshift-cluster-node-tuning-operator/tuned-ntkpv"
Sep 09 19:54:57.912236 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.912079    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/556a154a-15d2-4313-93b6-42580b64443c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4jptv\" (UID: \"556a154a-15d2-4313-93b6-42580b64443c\") " pod="openshift-multus/multus-additional-cni-plugins-4jptv"
Sep 09 19:54:57.912236 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.912080    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/556a154a-15d2-4313-93b6-42580b64443c-cni-binary-copy\") pod \"multus-additional-cni-plugins-4jptv\" (UID: \"556a154a-15d2-4313-93b6-42580b64443c\") " pod="openshift-multus/multus-additional-cni-plugins-4jptv"
Sep 09 19:54:57.912236 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.912118    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0eb194e9-a729-4845-86e2-e1011a9885c1-host\") pod \"tuned-ntkpv\" (UID: \"0eb194e9-a729-4845-86e2-e1011a9885c1\") " pod="openshift-cluster-node-tuning-operator/tuned-ntkpv"
Sep 09 19:54:57.912513 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.912488    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f6ccae8e-cfdd-44f3-8118-49e960b27d79-iptables-alerter-script\") pod \"iptables-alerter-m64jf\" (UID: \"f6ccae8e-cfdd-44f3-8118-49e960b27d79\") " pod="openshift-network-operator/iptables-alerter-m64jf"
Sep 09 19:54:57.913441 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.913420    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0eb194e9-a729-4845-86e2-e1011a9885c1-etc-tuned\") pod \"tuned-ntkpv\" (UID: \"0eb194e9-a729-4845-86e2-e1011a9885c1\") " pod="openshift-cluster-node-tuning-operator/tuned-ntkpv"
Sep 09 19:54:57.913730 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.913702    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0eb194e9-a729-4845-86e2-e1011a9885c1-tmp\") pod \"tuned-ntkpv\" (UID: \"0eb194e9-a729-4845-86e2-e1011a9885c1\") " pod="openshift-cluster-node-tuning-operator/tuned-ntkpv"
Sep 09 19:54:57.918931 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.918907    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66mmk\" (UniqueName: \"kubernetes.io/projected/1aa6d3e2-65bb-4992-9e19-e1825d7fe2aa-kube-api-access-66mmk\") pod \"aws-ebs-csi-driver-node-l55pn\" (UID: \"1aa6d3e2-65bb-4992-9e19-e1825d7fe2aa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l55pn"
Sep 09 19:54:57.919029 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.919006    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2n7x\" (UniqueName: \"kubernetes.io/projected/556a154a-15d2-4313-93b6-42580b64443c-kube-api-access-l2n7x\") pod \"multus-additional-cni-plugins-4jptv\" (UID: \"556a154a-15d2-4313-93b6-42580b64443c\") " pod="openshift-multus/multus-additional-cni-plugins-4jptv"
Sep 09 19:54:57.922668 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.922622    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgm4s\" (UniqueName: \"kubernetes.io/projected/f6ccae8e-cfdd-44f3-8118-49e960b27d79-kube-api-access-wgm4s\") pod \"iptables-alerter-m64jf\" (UID: \"f6ccae8e-cfdd-44f3-8118-49e960b27d79\") " pod="openshift-network-operator/iptables-alerter-m64jf"
Sep 09 19:54:57.926326 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.926306    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k95m\" (UniqueName: \"kubernetes.io/projected/0eb194e9-a729-4845-86e2-e1011a9885c1-kube-api-access-7k95m\") pod \"tuned-ntkpv\" (UID: \"0eb194e9-a729-4845-86e2-e1011a9885c1\") " pod="openshift-cluster-node-tuning-operator/tuned-ntkpv"
Sep 09 19:54:57.998448 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:57.998421    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vkhvk"
Sep 09 19:54:58.009356 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:58.009335    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:54:58.019972 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:58.019947    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xnmz6"
Sep 09 19:54:58.025439 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:58.025424    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-w2tb2"
Sep 09 19:54:58.033012 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:58.032995    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ctlrg"
Sep 09 19:54:58.040591 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:58.040576    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4jptv"
Sep 09 19:54:58.050028 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:58.050011    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-m64jf"
Sep 09 19:54:58.057591 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:58.057575    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l55pn"
Sep 09 19:54:58.063329 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:58.063312    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ntkpv"
Sep 09 19:54:58.314262 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:58.314185    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8-metrics-certs\") pod \"network-metrics-daemon-tmvpv\" (UID: \"21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8\") " pod="openshift-multus/network-metrics-daemon-tmvpv"
Sep 09 19:54:58.314404 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:58.314272    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2cc1cab-a99d-4733-8623-2ccf3fb89961-original-pull-secret\") pod \"global-pull-secret-syncer-gn8cc\" (UID: \"b2cc1cab-a99d-4733-8623-2ccf3fb89961\") " pod="kube-system/global-pull-secret-syncer-gn8cc"
Sep 09 19:54:58.314404 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:58.314342    2523 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:54:58.314404 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:58.314382    2523 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 19:54:58.314562 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:58.314419    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8-metrics-certs podName:21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8 nodeName:}" failed. No retries permitted until 2026-09-09 19:54:59.314402152 +0000 UTC m=+3.986421145 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8-metrics-certs") pod "network-metrics-daemon-tmvpv" (UID: "21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:54:58.314562 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:58.314436    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2cc1cab-a99d-4733-8623-2ccf3fb89961-original-pull-secret podName:b2cc1cab-a99d-4733-8623-2ccf3fb89961 nodeName:}" failed. No retries permitted until 2026-09-09 19:54:59.314427877 +0000 UTC m=+3.986446852 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b2cc1cab-a99d-4733-8623-2ccf3fb89961-original-pull-secret") pod "global-pull-secret-syncer-gn8cc" (UID: "b2cc1cab-a99d-4733-8623-2ccf3fb89961") : object "kube-system"/"original-pull-secret" not registered
Sep 09 19:54:58.414953 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:58.414924    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfddc\" (UniqueName: \"kubernetes.io/projected/d5e04e1e-56ef-4e29-89bd-eafcf7425d56-kube-api-access-hfddc\") pod \"network-check-target-jcx8l\" (UID: \"d5e04e1e-56ef-4e29-89bd-eafcf7425d56\") " pod="openshift-network-diagnostics/network-check-target-jcx8l"
Sep 09 19:54:58.415115 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:58.415052    2523 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 19:54:58.415115 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:58.415072    2523 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 19:54:58.415115 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:58.415083    2523 projected.go:196] Error preparing data for projected volume kube-api-access-hfddc for pod openshift-network-diagnostics/network-check-target-jcx8l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:54:58.415314 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:58.415142    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5e04e1e-56ef-4e29-89bd-eafcf7425d56-kube-api-access-hfddc podName:d5e04e1e-56ef-4e29-89bd-eafcf7425d56 nodeName:}" failed. No retries permitted until 2026-09-09 19:54:59.415126325 +0000 UTC m=+4.087145301 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hfddc" (UniqueName: "kubernetes.io/projected/d5e04e1e-56ef-4e29-89bd-eafcf7425d56-kube-api-access-hfddc") pod "network-check-target-jcx8l" (UID: "d5e04e1e-56ef-4e29-89bd-eafcf7425d56") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:54:58.736565 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:58.736488    2523 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 19:49:57 +0000 UTC" deadline="2028-02-14 11:32:14.942141028 +0000 UTC"
Sep 09 19:54:58.736565 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:58.736520    2523 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12543h37m16.205625803s"
Sep 09 19:54:58.803112 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:58.803082    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gn8cc"
Sep 09 19:54:58.803296 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:58.803232    2523 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gn8cc" podUID="b2cc1cab-a99d-4733-8623-2ccf3fb89961"
Sep 09 19:54:58.885122 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:58.885100    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fe231fb_4f9c_478e_b2c5_95a65a627385.slice/crio-92532108ffc4f9522506105e6e56ffd28fe73f5aa33c717b4437228e4b2faf09 WatchSource:0}: Error finding container 92532108ffc4f9522506105e6e56ffd28fe73f5aa33c717b4437228e4b2faf09: Status 404 returned error can't find the container with id 92532108ffc4f9522506105e6e56ffd28fe73f5aa33c717b4437228e4b2faf09
Sep 09 19:54:58.888204 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:58.888174    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d8ec74b_cbcb_4c36_a904_21e6a6460f6b.slice/crio-9e94ad436340dcf751e64333b3e42e914471372ada935141cda425f1d6de5109 WatchSource:0}: Error finding container 9e94ad436340dcf751e64333b3e42e914471372ada935141cda425f1d6de5109: Status 404 returned error can't find the container with id 9e94ad436340dcf751e64333b3e42e914471372ada935141cda425f1d6de5109
Sep 09 19:54:58.890676 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:58.890614    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8978aa77_ea38_48bd_b86f_fcaf6a8cfcbd.slice/crio-d3cfa2be4c89a41840e982cd4ce845559be7f9920b9595e4ba9e9f8cfc2922b4 WatchSource:0}: Error finding container d3cfa2be4c89a41840e982cd4ce845559be7f9920b9595e4ba9e9f8cfc2922b4: Status 404 returned error can't find the container with id d3cfa2be4c89a41840e982cd4ce845559be7f9920b9595e4ba9e9f8cfc2922b4
Sep 09 19:54:58.893390 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:58.893367    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aa6d3e2_65bb_4992_9e19_e1825d7fe2aa.slice/crio-22044fbaf557a4daaebccef35db4b4e1d277d305981f6f15e417c8147386b1c1 WatchSource:0}: Error finding container 22044fbaf557a4daaebccef35db4b4e1d277d305981f6f15e417c8147386b1c1: Status 404 returned error can't find the container with id 22044fbaf557a4daaebccef35db4b4e1d277d305981f6f15e417c8147386b1c1
Sep 09 19:54:58.895088 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:58.895068    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod556a154a_15d2_4313_93b6_42580b64443c.slice/crio-a16d86a68e02ff42764255947ab05a36fa19b5e10204b7f9d33ea607eeedd423 WatchSource:0}: Error finding container a16d86a68e02ff42764255947ab05a36fa19b5e10204b7f9d33ea607eeedd423: Status 404 returned error can't find the container with id a16d86a68e02ff42764255947ab05a36fa19b5e10204b7f9d33ea607eeedd423
Sep 09 19:54:58.896254 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:58.896036    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6ccae8e_cfdd_44f3_8118_49e960b27d79.slice/crio-24a9423c13a6414c03f4ec4d2a164b42a17227a7f1b8bac340b66cfe26379f00 WatchSource:0}: Error finding container 24a9423c13a6414c03f4ec4d2a164b42a17227a7f1b8bac340b66cfe26379f00: Status 404 returned error can't find the container with id 24a9423c13a6414c03f4ec4d2a164b42a17227a7f1b8bac340b66cfe26379f00
Sep 09 19:54:58.897414 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:58.897382    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad886add_2c65_470b_8935_efe692cafcec.slice/crio-425878faec9b7eb6208221fe3782bd931c416c83a95b2085f7497212d00ee3ec WatchSource:0}: Error finding container 425878faec9b7eb6208221fe3782bd931c416c83a95b2085f7497212d00ee3ec: Status 404 returned error can't find the container with id 425878faec9b7eb6208221fe3782bd931c416c83a95b2085f7497212d00ee3ec
Sep 09 19:54:58.897915 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:58.897765    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eb194e9_a729_4845_86e2_e1011a9885c1.slice/crio-589d660dadac447b9b508388df268186c720f4f643a6b09adce85f9f12c3b958 WatchSource:0}: Error finding container 589d660dadac447b9b508388df268186c720f4f643a6b09adce85f9f12c3b958: Status 404 returned error can't find the container with id 589d660dadac447b9b508388df268186c720f4f643a6b09adce85f9f12c3b958
Sep 09 19:54:58.899223 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:54:58.899197    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ac07c25_2d29_4115_851c_1c3602361e6a.slice/crio-ecbe0665abae43c55e697a2bfa34264f24fdc0aaee4c8cbd56b69996a9d5ce49 WatchSource:0}: Error finding container ecbe0665abae43c55e697a2bfa34264f24fdc0aaee4c8cbd56b69996a9d5ce49: Status 404 returned error can't find the container with id ecbe0665abae43c55e697a2bfa34264f24fdc0aaee4c8cbd56b69996a9d5ce49
Sep 09 19:54:59.322037 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:59.322001    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2cc1cab-a99d-4733-8623-2ccf3fb89961-original-pull-secret\") pod \"global-pull-secret-syncer-gn8cc\" (UID: \"b2cc1cab-a99d-4733-8623-2ccf3fb89961\") " pod="kube-system/global-pull-secret-syncer-gn8cc"
Sep 09 19:54:59.322200 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:59.322053    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8-metrics-certs\") pod \"network-metrics-daemon-tmvpv\" (UID: \"21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8\") " pod="openshift-multus/network-metrics-daemon-tmvpv"
Sep 09 19:54:59.322200 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:59.322170    2523 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 19:54:59.322200 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:59.322170    2523 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:54:59.322322 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:59.322232    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2cc1cab-a99d-4733-8623-2ccf3fb89961-original-pull-secret podName:b2cc1cab-a99d-4733-8623-2ccf3fb89961 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:01.322213855 +0000 UTC m=+5.994232844 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b2cc1cab-a99d-4733-8623-2ccf3fb89961-original-pull-secret") pod "global-pull-secret-syncer-gn8cc" (UID: "b2cc1cab-a99d-4733-8623-2ccf3fb89961") : object "kube-system"/"original-pull-secret" not registered
Sep 09 19:54:59.322322 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:59.322253    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8-metrics-certs podName:21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:01.322241792 +0000 UTC m=+5.994260779 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8-metrics-certs") pod "network-metrics-daemon-tmvpv" (UID: "21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:54:59.422752 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:59.422675    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfddc\" (UniqueName: \"kubernetes.io/projected/d5e04e1e-56ef-4e29-89bd-eafcf7425d56-kube-api-access-hfddc\") pod \"network-check-target-jcx8l\" (UID: \"d5e04e1e-56ef-4e29-89bd-eafcf7425d56\") " pod="openshift-network-diagnostics/network-check-target-jcx8l"
Sep 09 19:54:59.422883 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:59.422834    2523 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 19:54:59.422883 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:59.422855    2523 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 19:54:59.422883 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:59.422866    2523 projected.go:196] Error preparing data for projected volume kube-api-access-hfddc for pod openshift-network-diagnostics/network-check-target-jcx8l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:54:59.423028 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:59.422916    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5e04e1e-56ef-4e29-89bd-eafcf7425d56-kube-api-access-hfddc podName:d5e04e1e-56ef-4e29-89bd-eafcf7425d56 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:01.422898529 +0000 UTC m=+6.094917523 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hfddc" (UniqueName: "kubernetes.io/projected/d5e04e1e-56ef-4e29-89bd-eafcf7425d56-kube-api-access-hfddc") pod "network-check-target-jcx8l" (UID: "d5e04e1e-56ef-4e29-89bd-eafcf7425d56") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:54:59.737733 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:59.737620    2523 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 19:49:57 +0000 UTC" deadline="2028-03-15 06:35:00.820984272 +0000 UTC"
Sep 09 19:54:59.737733 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:59.737676    2523 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13258h40m1.083312266s"
Sep 09 19:54:59.804406 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:59.803312    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jcx8l"
Sep 09 19:54:59.804406 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:59.803330    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tmvpv"
Sep 09 19:54:59.804406 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:59.803426    2523 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jcx8l" podUID="d5e04e1e-56ef-4e29-89bd-eafcf7425d56"
Sep 09 19:54:59.804406 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:54:59.803562    2523 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tmvpv" podUID="21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8"
Sep 09 19:54:59.815015 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:59.814985    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c72g2" event={"ID":"ad886add-2c65-470b-8935-efe692cafcec","Type":"ContainerStarted","Data":"425878faec9b7eb6208221fe3782bd931c416c83a95b2085f7497212d00ee3ec"}
Sep 09 19:54:59.816731 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:59.816710    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-w2tb2" event={"ID":"8fe231fb-4f9c-478e-b2c5-95a65a627385","Type":"ContainerStarted","Data":"92532108ffc4f9522506105e6e56ffd28fe73f5aa33c717b4437228e4b2faf09"}
Sep 09 19:54:59.820692 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:59.820666    2523 generic.go:356] "Generic (PLEG): container finished" podID="75ee1d831c2282f17a9072ea7f2d2163" containerID="59e74a5c9a06ff5b4840f46a87b878c9bb8906745fea4ab4e7f9900be92a6867" exitCode=0
Sep 09 19:54:59.820798 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:59.820742    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-226.ec2.internal" event={"ID":"75ee1d831c2282f17a9072ea7f2d2163","Type":"ContainerDied","Data":"59e74a5c9a06ff5b4840f46a87b878c9bb8906745fea4ab4e7f9900be92a6867"}
Sep 09 19:54:59.826521 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:59.826473    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-m64jf" event={"ID":"f6ccae8e-cfdd-44f3-8118-49e960b27d79","Type":"ContainerStarted","Data":"24a9423c13a6414c03f4ec4d2a164b42a17227a7f1b8bac340b66cfe26379f00"}
Sep 09 19:54:59.830896 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:59.830868    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l55pn" event={"ID":"1aa6d3e2-65bb-4992-9e19-e1825d7fe2aa","Type":"ContainerStarted","Data":"22044fbaf557a4daaebccef35db4b4e1d277d305981f6f15e417c8147386b1c1"}
Sep 09 19:54:59.837004 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:59.836939    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vkhvk" event={"ID":"8978aa77-ea38-48bd-b86f-fcaf6a8cfcbd","Type":"ContainerStarted","Data":"d3cfa2be4c89a41840e982cd4ce845559be7f9920b9595e4ba9e9f8cfc2922b4"}
Sep 09 19:54:59.839600 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:59.839553    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ctlrg" event={"ID":"8d8ec74b-cbcb-4c36-a904-21e6a6460f6b","Type":"ContainerStarted","Data":"9e94ad436340dcf751e64333b3e42e914471372ada935141cda425f1d6de5109"}
Sep 09 19:54:59.845785 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:59.845758    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-226.ec2.internal" event={"ID":"f7c24ed75a93d3886eeb14da20c53cdf","Type":"ContainerStarted","Data":"bfce40c0cf2c708f5069a566fa79104f7ab39878156e57d1bd7cdd87bedd7541"}
Sep 09 19:54:59.854119 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:59.854093    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4jptv" event={"ID":"556a154a-15d2-4313-93b6-42580b64443c","Type":"ContainerStarted","Data":"a16d86a68e02ff42764255947ab05a36fa19b5e10204b7f9d33ea607eeedd423"}
Sep 09 19:54:59.858032 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:59.858006    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xnmz6" event={"ID":"5ac07c25-2d29-4115-851c-1c3602361e6a","Type":"ContainerStarted","Data":"ecbe0665abae43c55e697a2bfa34264f24fdc0aaee4c8cbd56b69996a9d5ce49"}
Sep 09 19:54:59.862352 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:59.862295    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-226.ec2.internal" podStartSLOduration=2.862280504 podStartE2EDuration="2.862280504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:54:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:54:59.862184857 +0000 UTC m=+4.534203854" watchObservedRunningTime="2026-09-09 19:54:59.862280504 +0000 UTC m=+4.534299493"
Sep 09 19:54:59.865101 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:54:59.865077    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ntkpv" event={"ID":"0eb194e9-a729-4845-86e2-e1011a9885c1","Type":"ContainerStarted","Data":"589d660dadac447b9b508388df268186c720f4f643a6b09adce85f9f12c3b958"}
Sep 09 19:55:00.802355 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:00.802251    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gn8cc"
Sep 09 19:55:00.803059 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:00.802381    2523 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gn8cc" podUID="b2cc1cab-a99d-4733-8623-2ccf3fb89961"
Sep 09 19:55:00.880332 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:00.880280    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-226.ec2.internal" event={"ID":"75ee1d831c2282f17a9072ea7f2d2163","Type":"ContainerStarted","Data":"30806eab1832e9807a7391f3fb6609a3a51a939e859bf5f4f3b03ff5372b6d6f"}
Sep 09 19:55:01.339163 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:01.338816    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2cc1cab-a99d-4733-8623-2ccf3fb89961-original-pull-secret\") pod \"global-pull-secret-syncer-gn8cc\" (UID: \"b2cc1cab-a99d-4733-8623-2ccf3fb89961\") " pod="kube-system/global-pull-secret-syncer-gn8cc"
Sep 09 19:55:01.339163 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:01.338882    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8-metrics-certs\") pod \"network-metrics-daemon-tmvpv\" (UID: \"21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8\") " pod="openshift-multus/network-metrics-daemon-tmvpv"
Sep 09 19:55:01.339163 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:01.338995    2523 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:01.339163 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:01.339013    2523 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:01.339163 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:01.339065    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2cc1cab-a99d-4733-8623-2ccf3fb89961-original-pull-secret podName:b2cc1cab-a99d-4733-8623-2ccf3fb89961 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:05.339046134 +0000 UTC m=+10.011065115 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b2cc1cab-a99d-4733-8623-2ccf3fb89961-original-pull-secret") pod "global-pull-secret-syncer-gn8cc" (UID: "b2cc1cab-a99d-4733-8623-2ccf3fb89961") : object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:01.339163 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:01.339120    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8-metrics-certs podName:21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:05.339102069 +0000 UTC m=+10.011121064 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8-metrics-certs") pod "network-metrics-daemon-tmvpv" (UID: "21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:01.439674 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:01.439637    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfddc\" (UniqueName: \"kubernetes.io/projected/d5e04e1e-56ef-4e29-89bd-eafcf7425d56-kube-api-access-hfddc\") pod \"network-check-target-jcx8l\" (UID: \"d5e04e1e-56ef-4e29-89bd-eafcf7425d56\") " pod="openshift-network-diagnostics/network-check-target-jcx8l"
Sep 09 19:55:01.439833 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:01.439819    2523 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 19:55:01.439894 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:01.439840    2523 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 19:55:01.439894 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:01.439853    2523 projected.go:196] Error preparing data for projected volume kube-api-access-hfddc for pod openshift-network-diagnostics/network-check-target-jcx8l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:01.439987 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:01.439908    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5e04e1e-56ef-4e29-89bd-eafcf7425d56-kube-api-access-hfddc podName:d5e04e1e-56ef-4e29-89bd-eafcf7425d56 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:05.439890539 +0000 UTC m=+10.111909529 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hfddc" (UniqueName: "kubernetes.io/projected/d5e04e1e-56ef-4e29-89bd-eafcf7425d56-kube-api-access-hfddc") pod "network-check-target-jcx8l" (UID: "d5e04e1e-56ef-4e29-89bd-eafcf7425d56") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:01.803326 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:01.803251    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jcx8l"
Sep 09 19:55:01.803735 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:01.803365    2523 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jcx8l" podUID="d5e04e1e-56ef-4e29-89bd-eafcf7425d56"
Sep 09 19:55:01.803811 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:01.803799    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tmvpv"
Sep 09 19:55:01.803929 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:01.803906    2523 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tmvpv" podUID="21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8"
Sep 09 19:55:02.802773 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:02.802743    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gn8cc"
Sep 09 19:55:02.802965 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:02.802870    2523 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gn8cc" podUID="b2cc1cab-a99d-4733-8623-2ccf3fb89961"
Sep 09 19:55:03.804287 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:03.804255    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jcx8l"
Sep 09 19:55:03.804715 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:03.804383    2523 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jcx8l" podUID="d5e04e1e-56ef-4e29-89bd-eafcf7425d56"
Sep 09 19:55:03.805010 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:03.804822    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tmvpv"
Sep 09 19:55:03.805010 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:03.804946    2523 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tmvpv" podUID="21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8"
Sep 09 19:55:04.803127 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:04.803053    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gn8cc"
Sep 09 19:55:04.803297 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:04.803200    2523 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gn8cc" podUID="b2cc1cab-a99d-4733-8623-2ccf3fb89961"
Sep 09 19:55:05.370653 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:05.370618    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8-metrics-certs\") pod \"network-metrics-daemon-tmvpv\" (UID: \"21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8\") " pod="openshift-multus/network-metrics-daemon-tmvpv"
Sep 09 19:55:05.371085 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:05.370686    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2cc1cab-a99d-4733-8623-2ccf3fb89961-original-pull-secret\") pod \"global-pull-secret-syncer-gn8cc\" (UID: \"b2cc1cab-a99d-4733-8623-2ccf3fb89961\") " pod="kube-system/global-pull-secret-syncer-gn8cc"
Sep 09 19:55:05.371085 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:05.370792    2523 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:05.371085 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:05.370848    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2cc1cab-a99d-4733-8623-2ccf3fb89961-original-pull-secret podName:b2cc1cab-a99d-4733-8623-2ccf3fb89961 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:13.370830547 +0000 UTC m=+18.042849538 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b2cc1cab-a99d-4733-8623-2ccf3fb89961-original-pull-secret") pod "global-pull-secret-syncer-gn8cc" (UID: "b2cc1cab-a99d-4733-8623-2ccf3fb89961") : object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:05.371296 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:05.371228    2523 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:05.371296 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:05.371276    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8-metrics-certs podName:21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:13.371261415 +0000 UTC m=+18.043280396 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8-metrics-certs") pod "network-metrics-daemon-tmvpv" (UID: "21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:05.471044 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:05.471006    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfddc\" (UniqueName: \"kubernetes.io/projected/d5e04e1e-56ef-4e29-89bd-eafcf7425d56-kube-api-access-hfddc\") pod \"network-check-target-jcx8l\" (UID: \"d5e04e1e-56ef-4e29-89bd-eafcf7425d56\") " pod="openshift-network-diagnostics/network-check-target-jcx8l"
Sep 09 19:55:05.471241 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:05.471200    2523 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 19:55:05.471241 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:05.471231    2523 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 19:55:05.471391 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:05.471247    2523 projected.go:196] Error preparing data for projected volume kube-api-access-hfddc for pod openshift-network-diagnostics/network-check-target-jcx8l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:05.471391 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:05.471303    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5e04e1e-56ef-4e29-89bd-eafcf7425d56-kube-api-access-hfddc podName:d5e04e1e-56ef-4e29-89bd-eafcf7425d56 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:13.471284673 +0000 UTC m=+18.143303652 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hfddc" (UniqueName: "kubernetes.io/projected/d5e04e1e-56ef-4e29-89bd-eafcf7425d56-kube-api-access-hfddc") pod "network-check-target-jcx8l" (UID: "d5e04e1e-56ef-4e29-89bd-eafcf7425d56") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:05.804379 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:05.803690    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jcx8l"
Sep 09 19:55:05.804379 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:05.803802    2523 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jcx8l" podUID="d5e04e1e-56ef-4e29-89bd-eafcf7425d56"
Sep 09 19:55:05.804379 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:05.804175    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tmvpv"
Sep 09 19:55:05.804379 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:05.804291    2523 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tmvpv" podUID="21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8"
Sep 09 19:55:06.802676 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:06.802641    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gn8cc"
Sep 09 19:55:06.803103 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:06.802790    2523 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gn8cc" podUID="b2cc1cab-a99d-4733-8623-2ccf3fb89961"
Sep 09 19:55:07.804317 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:07.804289    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jcx8l"
Sep 09 19:55:07.804723 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:07.804410    2523 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jcx8l" podUID="d5e04e1e-56ef-4e29-89bd-eafcf7425d56"
Sep 09 19:55:07.804785 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:07.804756    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tmvpv"
Sep 09 19:55:07.804890 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:07.804868    2523 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tmvpv" podUID="21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8"
Sep 09 19:55:08.802659 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:08.802618    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gn8cc"
Sep 09 19:55:08.802829 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:08.802726    2523 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gn8cc" podUID="b2cc1cab-a99d-4733-8623-2ccf3fb89961"
Sep 09 19:55:09.803164 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:09.802945    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jcx8l"
Sep 09 19:55:09.803535 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:09.803263    2523 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jcx8l" podUID="d5e04e1e-56ef-4e29-89bd-eafcf7425d56"
Sep 09 19:55:09.803593 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:09.803022    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tmvpv"
Sep 09 19:55:09.803693 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:09.803675    2523 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tmvpv" podUID="21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8"
Sep 09 19:55:10.802832 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:10.802803    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gn8cc"
Sep 09 19:55:10.803004 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:10.802904    2523 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gn8cc" podUID="b2cc1cab-a99d-4733-8623-2ccf3fb89961"
Sep 09 19:55:11.802271 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:11.802235    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jcx8l"
Sep 09 19:55:11.802729 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:11.802360    2523 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jcx8l" podUID="d5e04e1e-56ef-4e29-89bd-eafcf7425d56"
Sep 09 19:55:11.802729 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:11.802410    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tmvpv"
Sep 09 19:55:11.802729 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:11.802539    2523 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tmvpv" podUID="21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8"
Sep 09 19:55:12.803109 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:12.803079    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gn8cc"
Sep 09 19:55:12.803565 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:12.803221    2523 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gn8cc" podUID="b2cc1cab-a99d-4733-8623-2ccf3fb89961"
Sep 09 19:55:13.434138 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:13.434095    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8-metrics-certs\") pod \"network-metrics-daemon-tmvpv\" (UID: \"21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8\") " pod="openshift-multus/network-metrics-daemon-tmvpv"
Sep 09 19:55:13.434315 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:13.434200    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2cc1cab-a99d-4733-8623-2ccf3fb89961-original-pull-secret\") pod \"global-pull-secret-syncer-gn8cc\" (UID: \"b2cc1cab-a99d-4733-8623-2ccf3fb89961\") " pod="kube-system/global-pull-secret-syncer-gn8cc"
Sep 09 19:55:13.434315 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:13.434288    2523 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:13.434390 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:13.434315    2523 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:13.434390 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:13.434348    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8-metrics-certs podName:21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:29.434332768 +0000 UTC m=+34.106351767 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8-metrics-certs") pod "network-metrics-daemon-tmvpv" (UID: "21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:13.434390 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:13.434367    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2cc1cab-a99d-4733-8623-2ccf3fb89961-original-pull-secret podName:b2cc1cab-a99d-4733-8623-2ccf3fb89961 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:29.434353673 +0000 UTC m=+34.106372648 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b2cc1cab-a99d-4733-8623-2ccf3fb89961-original-pull-secret") pod "global-pull-secret-syncer-gn8cc" (UID: "b2cc1cab-a99d-4733-8623-2ccf3fb89961") : object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:13.534938 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:13.534905    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfddc\" (UniqueName: \"kubernetes.io/projected/d5e04e1e-56ef-4e29-89bd-eafcf7425d56-kube-api-access-hfddc\") pod \"network-check-target-jcx8l\" (UID: \"d5e04e1e-56ef-4e29-89bd-eafcf7425d56\") " pod="openshift-network-diagnostics/network-check-target-jcx8l"
Sep 09 19:55:13.535128 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:13.535110    2523 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 19:55:13.535192 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:13.535136    2523 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 19:55:13.535192 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:13.535164    2523 projected.go:196] Error preparing data for projected volume kube-api-access-hfddc for pod openshift-network-diagnostics/network-check-target-jcx8l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:13.535268 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:13.535225    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5e04e1e-56ef-4e29-89bd-eafcf7425d56-kube-api-access-hfddc podName:d5e04e1e-56ef-4e29-89bd-eafcf7425d56 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:29.53520689 +0000 UTC m=+34.207225879 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hfddc" (UniqueName: "kubernetes.io/projected/d5e04e1e-56ef-4e29-89bd-eafcf7425d56-kube-api-access-hfddc") pod "network-check-target-jcx8l" (UID: "d5e04e1e-56ef-4e29-89bd-eafcf7425d56") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:13.802622 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:13.802539    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jcx8l"
Sep 09 19:55:13.802622 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:13.802581    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tmvpv"
Sep 09 19:55:13.802788 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:13.802665    2523 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jcx8l" podUID="d5e04e1e-56ef-4e29-89bd-eafcf7425d56"
Sep 09 19:55:13.802840 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:13.802789    2523 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tmvpv" podUID="21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8"
Sep 09 19:55:14.802762 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:14.802731    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gn8cc"
Sep 09 19:55:14.803208 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:14.802842    2523 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gn8cc" podUID="b2cc1cab-a99d-4733-8623-2ccf3fb89961"
Sep 09 19:55:15.803133 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:15.803107    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tmvpv"
Sep 09 19:55:15.803600 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:15.803218    2523 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tmvpv" podUID="21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8"
Sep 09 19:55:15.803600 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:15.803317    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jcx8l"
Sep 09 19:55:15.803600 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:15.803423    2523 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jcx8l" podUID="d5e04e1e-56ef-4e29-89bd-eafcf7425d56"
Sep 09 19:55:16.802527 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:16.802501    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gn8cc"
Sep 09 19:55:16.802658 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:16.802598    2523 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gn8cc" podUID="b2cc1cab-a99d-4733-8623-2ccf3fb89961"
Sep 09 19:55:17.803130 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:17.802901    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jcx8l"
Sep 09 19:55:17.803794 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:17.802965    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tmvpv"
Sep 09 19:55:17.803794 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:17.803248    2523 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jcx8l" podUID="d5e04e1e-56ef-4e29-89bd-eafcf7425d56"
Sep 09 19:55:17.803794 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:17.803349    2523 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tmvpv" podUID="21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8"
Sep 09 19:55:17.919246 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:17.919175    2523 generic.go:356] "Generic (PLEG): container finished" podID="556a154a-15d2-4313-93b6-42580b64443c" containerID="27ebe75a416e234a2fb1c4d9619a4015a282d16a894000917411bf7b759633f5" exitCode=0
Sep 09 19:55:17.919393 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:17.919253    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4jptv" event={"ID":"556a154a-15d2-4313-93b6-42580b64443c","Type":"ContainerDied","Data":"27ebe75a416e234a2fb1c4d9619a4015a282d16a894000917411bf7b759633f5"}
Sep 09 19:55:17.921438 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:17.921413    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xnmz6" event={"ID":"5ac07c25-2d29-4115-851c-1c3602361e6a","Type":"ContainerStarted","Data":"8ad9e443194c95104886adfd42e7499deedb424b62ed86a231ebc56e94244d29"}
Sep 09 19:55:17.923332 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:17.923308    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ntkpv" event={"ID":"0eb194e9-a729-4845-86e2-e1011a9885c1","Type":"ContainerStarted","Data":"59a723e56ee947bd6d762caa18d765a6d8250d6d2096d03a73ea18f602acd27e"}
Sep 09 19:55:17.927189 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:17.927018    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c72g2" event={"ID":"ad886add-2c65-470b-8935-efe692cafcec","Type":"ContainerStarted","Data":"063ec2bd2db14ba367e606fc091a97f07a11d199fa2149741f8feeed5cc4cf1b"}
Sep 09 19:55:17.927189 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:17.927091    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c72g2" event={"ID":"ad886add-2c65-470b-8935-efe692cafcec","Type":"ContainerStarted","Data":"b640d4c8c033112583d5e3694b30f65582c3f7d528142fc50c247031ca791bbc"}
Sep 09 19:55:17.927189 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:17.927142    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c72g2" event={"ID":"ad886add-2c65-470b-8935-efe692cafcec","Type":"ContainerStarted","Data":"39fa12d62c92db66a23b0a1e324d7ee271982d12278d4f1e967707e2c1504ff1"}
Sep 09 19:55:17.927356 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:17.927211    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c72g2" event={"ID":"ad886add-2c65-470b-8935-efe692cafcec","Type":"ContainerStarted","Data":"e8dd12bb0fac624936a72708df22273f6663b80350bf47244a6962cf2e174c1d"}
Sep 09 19:55:17.927356 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:17.927258    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c72g2" event={"ID":"ad886add-2c65-470b-8935-efe692cafcec","Type":"ContainerStarted","Data":"02ca60114c5d6ce2b991cbe0eaf02151381bdcc79e8fa622992aed70e0376f29"}
Sep 09 19:55:17.927356 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:17.927304    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c72g2" event={"ID":"ad886add-2c65-470b-8935-efe692cafcec","Type":"ContainerStarted","Data":"ec18af2e2a1f07cf8a64fffe91b02dd5130e786b631e50c2becb765ee65f480e"}
Sep 09 19:55:17.928778 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:17.928755    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-w2tb2" event={"ID":"8fe231fb-4f9c-478e-b2c5-95a65a627385","Type":"ContainerStarted","Data":"1792db79363db8e1a716d2f2adc6c376c361e2c7ad75fac50080509af51a939f"}
Sep 09 19:55:17.930420 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:17.930393    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l55pn" event={"ID":"1aa6d3e2-65bb-4992-9e19-e1825d7fe2aa","Type":"ContainerStarted","Data":"3662f679eed807e26d5d1f4015cdfdbe582e56fee543d0a8dee31bee2323136a"}
Sep 09 19:55:17.931974 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:17.931947    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vkhvk" event={"ID":"8978aa77-ea38-48bd-b86f-fcaf6a8cfcbd","Type":"ContainerStarted","Data":"e8b93ae86434a252ea2755902cca7365384b3f9e1454d12dfe0bf8c328ece49f"}
Sep 09 19:55:17.933502 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:17.933485    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ctlrg" event={"ID":"8d8ec74b-cbcb-4c36-a904-21e6a6460f6b","Type":"ContainerStarted","Data":"e826da1dfd4cd4d25bc34e2706e9f9eaa421013e0adb8a4a0ce5f05752cb6482"}
Sep 09 19:55:17.942950 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:17.942911    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-226.ec2.internal" podStartSLOduration=20.942900204 podStartE2EDuration="20.942900204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:54:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:00.897282479 +0000 UTC m=+5.569301490" watchObservedRunningTime="2026-09-09 19:55:17.942900204 +0000 UTC m=+22.614919260"
Sep 09 19:55:17.967695 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:17.967652    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vkhvk" podStartSLOduration=4.961093954 podStartE2EDuration="22.967635513s" totalImagesPullingTime="18.006541559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:54:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:17.967446355 +0000 UTC m=+22.639465353" watchObservedRunningTime="2026-09-09 19:55:17.967635513 +0000 UTC m=+22.639654512"
Sep 09 19:55:17.990327 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:17.990279    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-xnmz6" podStartSLOduration=4.94079898 podStartE2EDuration="22.990265891s" totalImagesPullingTime="18.049466911s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:54:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:17.987910286 +0000 UTC m=+22.659929277" watchObservedRunningTime="2026-09-09 19:55:17.990265891 +0000 UTC m=+22.662284889"
Sep 09 19:55:18.004943 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:18.004913    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-w2tb2" podStartSLOduration=3.989182418 podStartE2EDuration="22.004902406s" totalImagesPullingTime="18.015719988s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:54:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:18.004644002 +0000 UTC m=+22.676662998" watchObservedRunningTime="2026-09-09 19:55:18.004902406 +0000 UTC m=+22.676921402"
Sep 09 19:55:18.021543 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:18.021507    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ctlrg" podStartSLOduration=4.014295751 podStartE2EDuration="22.021496432s" totalImagesPullingTime="18.007200681s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:54:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:18.021127897 +0000 UTC m=+22.693146895" watchObservedRunningTime="2026-09-09 19:55:18.021496432 +0000 UTC m=+22.693515425"
Sep 09 19:55:18.046914 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:18.046876    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ntkpv" podStartSLOduration=4.014861004 podStartE2EDuration="22.046866906s" totalImagesPullingTime="18.032005902s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:54:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:18.046794516 +0000 UTC m=+22.718813513" watchObservedRunningTime="2026-09-09 19:55:18.046866906 +0000 UTC m=+22.718885900"
Sep 09 19:55:18.401441 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:18.401302    2523 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 19:55:18.778334 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:18.778217    2523 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T19:55:18.401438361Z","UUID":"64f86cd0-f764-41ad-a2a5-866b2cdb6e83","Handler":null,"Name":"","Endpoint":""}
Sep 09 19:55:18.780317 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:18.780280    2523 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 19:55:18.780317 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:18.780313    2523 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 19:55:18.802585 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:18.802559    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gn8cc"
Sep 09 19:55:18.802696 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:18.802659    2523 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gn8cc" podUID="b2cc1cab-a99d-4733-8623-2ccf3fb89961"
Sep 09 19:55:18.936791 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:18.936764    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-m64jf" event={"ID":"f6ccae8e-cfdd-44f3-8118-49e960b27d79","Type":"ContainerStarted","Data":"427ca81e5bf6eef7d1497adc8988ea6f820bd5a747b69431a74ac1a0a4957486"}
Sep 09 19:55:18.938729 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:18.938697    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l55pn" event={"ID":"1aa6d3e2-65bb-4992-9e19-e1825d7fe2aa","Type":"ContainerStarted","Data":"5fa608df7b5d681d17c195701c7973709d128e171ad890d58403d712d7383c07"}
Sep 09 19:55:18.952209 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:18.952173    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-m64jf" podStartSLOduration=4.9507764850000004 podStartE2EDuration="22.952140861s" totalImagesPullingTime="18.001364376s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:54:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:18.952091583 +0000 UTC m=+23.624110579" watchObservedRunningTime="2026-09-09 19:55:18.952140861 +0000 UTC m=+23.624159857"
Sep 09 19:55:19.803072 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:19.802997    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tmvpv"
Sep 09 19:55:19.803236 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:19.803142    2523 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tmvpv" podUID="21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8"
Sep 09 19:55:19.803236 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:19.803193    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jcx8l"
Sep 09 19:55:19.803354 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:19.803296    2523 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jcx8l" podUID="d5e04e1e-56ef-4e29-89bd-eafcf7425d56"
Sep 09 19:55:19.943808 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:19.943779    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c72g2" event={"ID":"ad886add-2c65-470b-8935-efe692cafcec","Type":"ContainerStarted","Data":"6c6f8c9f9549e93d8f5a72bf47bab53817fce0a338e5bb66f79d259bdc2c6287"}
Sep 09 19:55:20.803018 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:20.802987    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gn8cc"
Sep 09 19:55:20.803208 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:20.803090    2523 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gn8cc" podUID="b2cc1cab-a99d-4733-8623-2ccf3fb89961"
Sep 09 19:55:21.802188 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:21.802142    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jcx8l"
Sep 09 19:55:21.802632 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:21.802276    2523 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jcx8l" podUID="d5e04e1e-56ef-4e29-89bd-eafcf7425d56"
Sep 09 19:55:21.802632 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:21.802341    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tmvpv"
Sep 09 19:55:21.802632 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:21.802431    2523 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tmvpv" podUID="21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8"
Sep 09 19:55:22.721634 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:22.721426    2523 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vkhvk"
Sep 09 19:55:22.721934 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:22.721913    2523 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vkhvk"
Sep 09 19:55:22.802713 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:22.802692    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gn8cc"
Sep 09 19:55:22.803118 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:22.802778    2523 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gn8cc" podUID="b2cc1cab-a99d-4733-8623-2ccf3fb89961"
Sep 09 19:55:22.951392 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:22.951340    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l55pn" event={"ID":"1aa6d3e2-65bb-4992-9e19-e1825d7fe2aa","Type":"ContainerStarted","Data":"1688cd18e87fb67a73ea40e640773a4de3b92080764d801cd06d7acde4534e17"}
Sep 09 19:55:22.953063 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:22.953043    2523 generic.go:356] "Generic (PLEG): container finished" podID="556a154a-15d2-4313-93b6-42580b64443c" containerID="e998e3c97afbf63bd92e183c1a253aba2f89b14fe948dfbaf678d09c0aa56504" exitCode=0
Sep 09 19:55:22.953168 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:22.953116    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4jptv" event={"ID":"556a154a-15d2-4313-93b6-42580b64443c","Type":"ContainerDied","Data":"e998e3c97afbf63bd92e183c1a253aba2f89b14fe948dfbaf678d09c0aa56504"}
Sep 09 19:55:22.959637 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:22.959614    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c72g2" event={"ID":"ad886add-2c65-470b-8935-efe692cafcec","Type":"ContainerStarted","Data":"4dc37519f325bc4d983e9a6b5303d85dcccbdd2604cf0c71abc7e5acc2c3b87c"}
Sep 09 19:55:22.959799 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:22.959784    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vkhvk"
Sep 09 19:55:22.960189 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:22.960170    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vkhvk"
Sep 09 19:55:22.971354 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:22.971319    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l55pn" podStartSLOduration=3.648933485 podStartE2EDuration="26.971309112s" totalImagesPullingTime="23.322375627s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:54:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:22.970815295 +0000 UTC m=+27.642834307" watchObservedRunningTime="2026-09-09 19:55:22.971309112 +0000 UTC m=+27.643328109"
Sep 09 19:55:23.001561 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:23.001528    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-c72g2" podStartSLOduration=9.940471014 podStartE2EDuration="28.001517382s" totalImagesPullingTime="18.061046368s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:54:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:23.000587013 +0000 UTC m=+27.672606040" watchObservedRunningTime="2026-09-09 19:55:23.001517382 +0000 UTC m=+27.673536362"
Sep 09 19:55:23.802857 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:23.802828    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tmvpv"
Sep 09 19:55:23.803485 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:23.802848    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jcx8l"
Sep 09 19:55:23.803485 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:23.802939    2523 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tmvpv" podUID="21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8"
Sep 09 19:55:23.803485 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:23.803133    2523 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jcx8l" podUID="d5e04e1e-56ef-4e29-89bd-eafcf7425d56"
Sep 09 19:55:23.964310 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:23.964239    2523 generic.go:356] "Generic (PLEG): container finished" podID="556a154a-15d2-4313-93b6-42580b64443c" containerID="1f25538132bdaaa4fb5a976f4d7dcf3b2d323d8264d84e965c2a13a1a48df807" exitCode=0
Sep 09 19:55:23.964426 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:23.964311    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4jptv" event={"ID":"556a154a-15d2-4313-93b6-42580b64443c","Type":"ContainerDied","Data":"1f25538132bdaaa4fb5a976f4d7dcf3b2d323d8264d84e965c2a13a1a48df807"}
Sep 09 19:55:23.965381 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:23.964943    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:55:23.965381 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:23.965020    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:55:23.965381 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:23.965061    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:55:23.979632 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:23.979609    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:55:23.979790 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:23.979778    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:55:24.088104 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:24.088080    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gn8cc"]
Sep 09 19:55:24.088253 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:24.088228    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gn8cc"
Sep 09 19:55:24.088380 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:24.088356    2523 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gn8cc" podUID="b2cc1cab-a99d-4733-8623-2ccf3fb89961"
Sep 09 19:55:24.091091 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:24.091071    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tmvpv"]
Sep 09 19:55:24.091191 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:24.091168    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tmvpv"
Sep 09 19:55:24.091254 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:24.091239    2523 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tmvpv" podUID="21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8"
Sep 09 19:55:24.104276 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:24.104253    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jcx8l"]
Sep 09 19:55:24.104354 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:24.104325    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jcx8l"
Sep 09 19:55:24.104433 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:24.104414    2523 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jcx8l" podUID="d5e04e1e-56ef-4e29-89bd-eafcf7425d56"
Sep 09 19:55:24.968615 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:24.968544    2523 generic.go:356] "Generic (PLEG): container finished" podID="556a154a-15d2-4313-93b6-42580b64443c" containerID="2042f17df229cd98aef76772b9e034fc46fe4f000efe646f91793cf4bf13e63e" exitCode=0
Sep 09 19:55:24.968975 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:24.968618    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4jptv" event={"ID":"556a154a-15d2-4313-93b6-42580b64443c","Type":"ContainerDied","Data":"2042f17df229cd98aef76772b9e034fc46fe4f000efe646f91793cf4bf13e63e"}
Sep 09 19:55:25.802865 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:25.802720    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jcx8l"
Sep 09 19:55:25.802990 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:25.802800    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gn8cc"
Sep 09 19:55:25.802990 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:25.802937    2523 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jcx8l" podUID="d5e04e1e-56ef-4e29-89bd-eafcf7425d56"
Sep 09 19:55:25.803112 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:25.802823    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tmvpv"
Sep 09 19:55:25.803112 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:25.803037    2523 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gn8cc" podUID="b2cc1cab-a99d-4733-8623-2ccf3fb89961"
Sep 09 19:55:25.803239 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:25.803170    2523 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tmvpv" podUID="21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8"
Sep 09 19:55:27.803020 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:27.802985    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gn8cc"
Sep 09 19:55:27.803455 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:27.802986    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jcx8l"
Sep 09 19:55:27.803455 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:27.803101    2523 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gn8cc" podUID="b2cc1cab-a99d-4733-8623-2ccf3fb89961"
Sep 09 19:55:27.803455 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:27.803111    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tmvpv"
Sep 09 19:55:27.803455 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:27.803231    2523 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jcx8l" podUID="d5e04e1e-56ef-4e29-89bd-eafcf7425d56"
Sep 09 19:55:27.803455 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:27.803335    2523 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tmvpv" podUID="21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8"
Sep 09 19:55:29.460900 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.460874    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8-metrics-certs\") pod \"network-metrics-daemon-tmvpv\" (UID: \"21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8\") " pod="openshift-multus/network-metrics-daemon-tmvpv"
Sep 09 19:55:29.461358 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.460952    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2cc1cab-a99d-4733-8623-2ccf3fb89961-original-pull-secret\") pod \"global-pull-secret-syncer-gn8cc\" (UID: \"b2cc1cab-a99d-4733-8623-2ccf3fb89961\") " pod="kube-system/global-pull-secret-syncer-gn8cc"
Sep 09 19:55:29.461358 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:29.461047    2523 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:29.461358 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:29.461064    2523 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:29.461358 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:29.461122    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8-metrics-certs podName:21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:01.461093072 +0000 UTC m=+66.133112069 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8-metrics-certs") pod "network-metrics-daemon-tmvpv" (UID: "21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:29.461358 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:29.461142    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2cc1cab-a99d-4733-8623-2ccf3fb89961-original-pull-secret podName:b2cc1cab-a99d-4733-8623-2ccf3fb89961 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:01.461132257 +0000 UTC m=+66.133151238 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b2cc1cab-a99d-4733-8623-2ccf3fb89961-original-pull-secret") pod "global-pull-secret-syncer-gn8cc" (UID: "b2cc1cab-a99d-4733-8623-2ccf3fb89961") : object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:29.561466 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.561438    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfddc\" (UniqueName: \"kubernetes.io/projected/d5e04e1e-56ef-4e29-89bd-eafcf7425d56-kube-api-access-hfddc\") pod \"network-check-target-jcx8l\" (UID: \"d5e04e1e-56ef-4e29-89bd-eafcf7425d56\") " pod="openshift-network-diagnostics/network-check-target-jcx8l"
Sep 09 19:55:29.561688 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:29.561597    2523 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 19:55:29.561688 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:29.561611    2523 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 19:55:29.561688 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:29.561620    2523 projected.go:196] Error preparing data for projected volume kube-api-access-hfddc for pod openshift-network-diagnostics/network-check-target-jcx8l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:29.561688 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:29.561672    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d5e04e1e-56ef-4e29-89bd-eafcf7425d56-kube-api-access-hfddc podName:d5e04e1e-56ef-4e29-89bd-eafcf7425d56 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:01.561657084 +0000 UTC m=+66.233676080 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-hfddc" (UniqueName: "kubernetes.io/projected/d5e04e1e-56ef-4e29-89bd-eafcf7425d56-kube-api-access-hfddc") pod "network-check-target-jcx8l" (UID: "d5e04e1e-56ef-4e29-89bd-eafcf7425d56") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:29.664003 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.663978    2523 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-226.ec2.internal" event="NodeReady"
Sep 09 19:55:29.664252 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.664095    2523 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 19:55:29.704592 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.704528    2523 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-bcc64b9dc-p7sz5"]
Sep 09 19:55:29.720740 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.720714    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bcc64b9dc-p7sz5"
Sep 09 19:55:29.722953 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.722629    2523 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zr55l"]
Sep 09 19:55:29.723453 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.723287    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 09 19:55:29.723453 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.723278    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 09 19:55:29.723453 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.723327    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-j6mws\""
Sep 09 19:55:29.723639 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.723628    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 09 19:55:29.729614 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.729580    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 09 19:55:29.737383 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.737209    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-bcc64b9dc-p7sz5"]
Sep 09 19:55:29.737383 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.737246    2523 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rq89t"]
Sep 09 19:55:29.737383 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.737341    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zr55l"
Sep 09 19:55:29.740323 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.740307    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 19:55:29.740475 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.740455    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 19:55:29.740576 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.740514    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-shzs9\""
Sep 09 19:55:29.757883 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.757865    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rq89t"]
Sep 09 19:55:29.757971 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.757897    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zr55l"]
Sep 09 19:55:29.758028 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.757980    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rq89t"
Sep 09 19:55:29.760472 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.760454    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 19:55:29.760702 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.760688    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 19:55:29.760763 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.760749    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-jdq5v\""
Sep 09 19:55:29.760815 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.760803    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 19:55:29.802689 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.802663    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jcx8l"
Sep 09 19:55:29.802689 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.802681    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gn8cc"
Sep 09 19:55:29.802883 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.802765    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tmvpv"
Sep 09 19:55:29.805577 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.805539    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7r2s2\""
Sep 09 19:55:29.805577 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.805564    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 19:55:29.805948 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.805786    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rfksv\""
Sep 09 19:55:29.805948 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.805823    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 19:55:29.805948 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.805830    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 19:55:29.805948 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.805897    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 19:55:29.863422 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.863398    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fb48971c-63be-4de1-b19e-0b8c1aadbc58-installation-pull-secrets\") pod \"image-registry-bcc64b9dc-p7sz5\" (UID: \"fb48971c-63be-4de1-b19e-0b8c1aadbc58\") " pod="openshift-image-registry/image-registry-bcc64b9dc-p7sz5"
Sep 09 19:55:29.863548 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.863429    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fb48971c-63be-4de1-b19e-0b8c1aadbc58-registry-certificates\") pod \"image-registry-bcc64b9dc-p7sz5\" (UID: \"fb48971c-63be-4de1-b19e-0b8c1aadbc58\") " pod="openshift-image-registry/image-registry-bcc64b9dc-p7sz5"
Sep 09 19:55:29.863548 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.863450    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fb48971c-63be-4de1-b19e-0b8c1aadbc58-registry-tls\") pod \"image-registry-bcc64b9dc-p7sz5\" (UID: \"fb48971c-63be-4de1-b19e-0b8c1aadbc58\") " pod="openshift-image-registry/image-registry-bcc64b9dc-p7sz5"
Sep 09 19:55:29.863548 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.863484    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fb48971c-63be-4de1-b19e-0b8c1aadbc58-bound-sa-token\") pod \"image-registry-bcc64b9dc-p7sz5\" (UID: \"fb48971c-63be-4de1-b19e-0b8c1aadbc58\") " pod="openshift-image-registry/image-registry-bcc64b9dc-p7sz5"
Sep 09 19:55:29.863548 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.863511    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d3bbfe8-34df-41e1-b959-766ff5100634-metrics-tls\") pod \"dns-default-zr55l\" (UID: \"1d3bbfe8-34df-41e1-b959-766ff5100634\") " pod="openshift-dns/dns-default-zr55l"
Sep 09 19:55:29.863763 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.863570    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbj5l\" (UniqueName: \"kubernetes.io/projected/8029d5a2-cb5c-4ab4-93cf-d9da4f3bcaea-kube-api-access-kbj5l\") pod \"ingress-canary-rq89t\" (UID: \"8029d5a2-cb5c-4ab4-93cf-d9da4f3bcaea\") " pod="openshift-ingress-canary/ingress-canary-rq89t"
Sep 09 19:55:29.863763 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.863645    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8029d5a2-cb5c-4ab4-93cf-d9da4f3bcaea-cert\") pod \"ingress-canary-rq89t\" (UID: \"8029d5a2-cb5c-4ab4-93cf-d9da4f3bcaea\") " pod="openshift-ingress-canary/ingress-canary-rq89t"
Sep 09 19:55:29.863763 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.863686    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fb48971c-63be-4de1-b19e-0b8c1aadbc58-ca-trust-extracted\") pod \"image-registry-bcc64b9dc-p7sz5\" (UID: \"fb48971c-63be-4de1-b19e-0b8c1aadbc58\") " pod="openshift-image-registry/image-registry-bcc64b9dc-p7sz5"
Sep 09 19:55:29.863763 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.863726    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d3bbfe8-34df-41e1-b959-766ff5100634-tmp-dir\") pod \"dns-default-zr55l\" (UID: \"1d3bbfe8-34df-41e1-b959-766ff5100634\") " pod="openshift-dns/dns-default-zr55l"
Sep 09 19:55:29.863946 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.863765    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fb48971c-63be-4de1-b19e-0b8c1aadbc58-image-registry-private-configuration\") pod \"image-registry-bcc64b9dc-p7sz5\" (UID: \"fb48971c-63be-4de1-b19e-0b8c1aadbc58\") " pod="openshift-image-registry/image-registry-bcc64b9dc-p7sz5"
Sep 09 19:55:29.863946 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.863790    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brzwp\" (UniqueName: \"kubernetes.io/projected/1d3bbfe8-34df-41e1-b959-766ff5100634-kube-api-access-brzwp\") pod \"dns-default-zr55l\" (UID: \"1d3bbfe8-34df-41e1-b959-766ff5100634\") " pod="openshift-dns/dns-default-zr55l"
Sep 09 19:55:29.863946 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.863818    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1d3bbfe8-34df-41e1-b959-766ff5100634-config-volume\") pod \"dns-default-zr55l\" (UID: \"1d3bbfe8-34df-41e1-b959-766ff5100634\") " pod="openshift-dns/dns-default-zr55l"
Sep 09 19:55:29.863946 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.863855    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fb48971c-63be-4de1-b19e-0b8c1aadbc58-trusted-ca\") pod \"image-registry-bcc64b9dc-p7sz5\" (UID: \"fb48971c-63be-4de1-b19e-0b8c1aadbc58\") " pod="openshift-image-registry/image-registry-bcc64b9dc-p7sz5"
Sep 09 19:55:29.863946 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.863884    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsvj2\" (UniqueName: \"kubernetes.io/projected/fb48971c-63be-4de1-b19e-0b8c1aadbc58-kube-api-access-jsvj2\") pod \"image-registry-bcc64b9dc-p7sz5\" (UID: \"fb48971c-63be-4de1-b19e-0b8c1aadbc58\") " pod="openshift-image-registry/image-registry-bcc64b9dc-p7sz5"
Sep 09 19:55:29.965061 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.965002    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fb48971c-63be-4de1-b19e-0b8c1aadbc58-registry-certificates\") pod \"image-registry-bcc64b9dc-p7sz5\" (UID: \"fb48971c-63be-4de1-b19e-0b8c1aadbc58\") " pod="openshift-image-registry/image-registry-bcc64b9dc-p7sz5"
Sep 09 19:55:29.965061 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.965032    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fb48971c-63be-4de1-b19e-0b8c1aadbc58-registry-tls\") pod \"image-registry-bcc64b9dc-p7sz5\" (UID: \"fb48971c-63be-4de1-b19e-0b8c1aadbc58\") " pod="openshift-image-registry/image-registry-bcc64b9dc-p7sz5"
Sep 09 19:55:29.965061 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.965057    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fb48971c-63be-4de1-b19e-0b8c1aadbc58-bound-sa-token\") pod \"image-registry-bcc64b9dc-p7sz5\" (UID: \"fb48971c-63be-4de1-b19e-0b8c1aadbc58\") " pod="openshift-image-registry/image-registry-bcc64b9dc-p7sz5"
Sep 09 19:55:29.965334 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.965073    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d3bbfe8-34df-41e1-b959-766ff5100634-metrics-tls\") pod \"dns-default-zr55l\" (UID: \"1d3bbfe8-34df-41e1-b959-766ff5100634\") " pod="openshift-dns/dns-default-zr55l"
Sep 09 19:55:29.965334 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.965095    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbj5l\" (UniqueName: \"kubernetes.io/projected/8029d5a2-cb5c-4ab4-93cf-d9da4f3bcaea-kube-api-access-kbj5l\") pod \"ingress-canary-rq89t\" (UID: \"8029d5a2-cb5c-4ab4-93cf-d9da4f3bcaea\") " pod="openshift-ingress-canary/ingress-canary-rq89t"
Sep 09 19:55:29.965334 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:29.965216    2523 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 19:55:29.965334 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:29.965233    2523 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 19:55:29.965334 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:29.965313    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d3bbfe8-34df-41e1-b959-766ff5100634-metrics-tls podName:1d3bbfe8-34df-41e1-b959-766ff5100634 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:30.465290115 +0000 UTC m=+35.137309106 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1d3bbfe8-34df-41e1-b959-766ff5100634-metrics-tls") pod "dns-default-zr55l" (UID: "1d3bbfe8-34df-41e1-b959-766ff5100634") : secret "dns-default-metrics-tls" not found
Sep 09 19:55:29.965588 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:29.965237    2523 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bcc64b9dc-p7sz5: secret "image-registry-tls" not found
Sep 09 19:55:29.965588 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:29.965441    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb48971c-63be-4de1-b19e-0b8c1aadbc58-registry-tls podName:fb48971c-63be-4de1-b19e-0b8c1aadbc58 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:30.465422919 +0000 UTC m=+35.137441912 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fb48971c-63be-4de1-b19e-0b8c1aadbc58-registry-tls") pod "image-registry-bcc64b9dc-p7sz5" (UID: "fb48971c-63be-4de1-b19e-0b8c1aadbc58") : secret "image-registry-tls" not found
Sep 09 19:55:29.965588 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.965506    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8029d5a2-cb5c-4ab4-93cf-d9da4f3bcaea-cert\") pod \"ingress-canary-rq89t\" (UID: \"8029d5a2-cb5c-4ab4-93cf-d9da4f3bcaea\") " pod="openshift-ingress-canary/ingress-canary-rq89t"
Sep 09 19:55:29.965588 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.965571    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fb48971c-63be-4de1-b19e-0b8c1aadbc58-ca-trust-extracted\") pod \"image-registry-bcc64b9dc-p7sz5\" (UID: \"fb48971c-63be-4de1-b19e-0b8c1aadbc58\") " pod="openshift-image-registry/image-registry-bcc64b9dc-p7sz5"
Sep 09 19:55:29.965780 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.965616    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d3bbfe8-34df-41e1-b959-766ff5100634-tmp-dir\") pod \"dns-default-zr55l\" (UID: \"1d3bbfe8-34df-41e1-b959-766ff5100634\") " pod="openshift-dns/dns-default-zr55l"
Sep 09 19:55:29.965780 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.965642    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fb48971c-63be-4de1-b19e-0b8c1aadbc58-registry-certificates\") pod \"image-registry-bcc64b9dc-p7sz5\" (UID: \"fb48971c-63be-4de1-b19e-0b8c1aadbc58\") " pod="openshift-image-registry/image-registry-bcc64b9dc-p7sz5"
Sep 09 19:55:29.965780 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.965655    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fb48971c-63be-4de1-b19e-0b8c1aadbc58-image-registry-private-configuration\") pod \"image-registry-bcc64b9dc-p7sz5\" (UID: \"fb48971c-63be-4de1-b19e-0b8c1aadbc58\") " pod="openshift-image-registry/image-registry-bcc64b9dc-p7sz5"
Sep 09 19:55:29.965780 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.965696    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brzwp\" (UniqueName: \"kubernetes.io/projected/1d3bbfe8-34df-41e1-b959-766ff5100634-kube-api-access-brzwp\") pod \"dns-default-zr55l\" (UID: \"1d3bbfe8-34df-41e1-b959-766ff5100634\") " pod="openshift-dns/dns-default-zr55l"
Sep 09 19:55:29.965780 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.965729    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1d3bbfe8-34df-41e1-b959-766ff5100634-config-volume\") pod \"dns-default-zr55l\" (UID: \"1d3bbfe8-34df-41e1-b959-766ff5100634\") " pod="openshift-dns/dns-default-zr55l"
Sep 09 19:55:29.965780 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.965758    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fb48971c-63be-4de1-b19e-0b8c1aadbc58-trusted-ca\") pod \"image-registry-bcc64b9dc-p7sz5\" (UID: \"fb48971c-63be-4de1-b19e-0b8c1aadbc58\") " pod="openshift-image-registry/image-registry-bcc64b9dc-p7sz5"
Sep 09 19:55:29.966046 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.965782    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsvj2\" (UniqueName: \"kubernetes.io/projected/fb48971c-63be-4de1-b19e-0b8c1aadbc58-kube-api-access-jsvj2\") pod \"image-registry-bcc64b9dc-p7sz5\" (UID: \"fb48971c-63be-4de1-b19e-0b8c1aadbc58\") " pod="openshift-image-registry/image-registry-bcc64b9dc-p7sz5"
Sep 09 19:55:29.966046 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.965832    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fb48971c-63be-4de1-b19e-0b8c1aadbc58-installation-pull-secrets\") pod \"image-registry-bcc64b9dc-p7sz5\" (UID: \"fb48971c-63be-4de1-b19e-0b8c1aadbc58\") " pod="openshift-image-registry/image-registry-bcc64b9dc-p7sz5"
Sep 09 19:55:29.966046 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.965838    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fb48971c-63be-4de1-b19e-0b8c1aadbc58-ca-trust-extracted\") pod \"image-registry-bcc64b9dc-p7sz5\" (UID: \"fb48971c-63be-4de1-b19e-0b8c1aadbc58\") " pod="openshift-image-registry/image-registry-bcc64b9dc-p7sz5"
Sep 09 19:55:29.966046 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.965972    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d3bbfe8-34df-41e1-b959-766ff5100634-tmp-dir\") pod \"dns-default-zr55l\" (UID: \"1d3bbfe8-34df-41e1-b959-766ff5100634\") " pod="openshift-dns/dns-default-zr55l"
Sep 09 19:55:29.966635 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.966589    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1d3bbfe8-34df-41e1-b959-766ff5100634-config-volume\") pod \"dns-default-zr55l\" (UID: \"1d3bbfe8-34df-41e1-b959-766ff5100634\") " pod="openshift-dns/dns-default-zr55l"
Sep 09 19:55:29.966816 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.966797    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fb48971c-63be-4de1-b19e-0b8c1aadbc58-trusted-ca\") pod \"image-registry-bcc64b9dc-p7sz5\" (UID: \"fb48971c-63be-4de1-b19e-0b8c1aadbc58\") " pod="openshift-image-registry/image-registry-bcc64b9dc-p7sz5"
Sep 09 19:55:29.969856 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.969831    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8029d5a2-cb5c-4ab4-93cf-d9da4f3bcaea-cert\") pod \"ingress-canary-rq89t\" (UID: \"8029d5a2-cb5c-4ab4-93cf-d9da4f3bcaea\") " pod="openshift-ingress-canary/ingress-canary-rq89t"
Sep 09 19:55:29.969947 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.969838    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fb48971c-63be-4de1-b19e-0b8c1aadbc58-installation-pull-secrets\") pod \"image-registry-bcc64b9dc-p7sz5\" (UID: \"fb48971c-63be-4de1-b19e-0b8c1aadbc58\") " pod="openshift-image-registry/image-registry-bcc64b9dc-p7sz5"
Sep 09 19:55:29.969947 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.969883    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fb48971c-63be-4de1-b19e-0b8c1aadbc58-image-registry-private-configuration\") pod \"image-registry-bcc64b9dc-p7sz5\" (UID: \"fb48971c-63be-4de1-b19e-0b8c1aadbc58\") " pod="openshift-image-registry/image-registry-bcc64b9dc-p7sz5"
Sep 09 19:55:29.975346 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.975296    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brzwp\" (UniqueName: \"kubernetes.io/projected/1d3bbfe8-34df-41e1-b959-766ff5100634-kube-api-access-brzwp\") pod \"dns-default-zr55l\" (UID: \"1d3bbfe8-34df-41e1-b959-766ff5100634\") " pod="openshift-dns/dns-default-zr55l"
Sep 09 19:55:29.975717 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.975696    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsvj2\" (UniqueName: \"kubernetes.io/projected/fb48971c-63be-4de1-b19e-0b8c1aadbc58-kube-api-access-jsvj2\") pod \"image-registry-bcc64b9dc-p7sz5\" (UID: \"fb48971c-63be-4de1-b19e-0b8c1aadbc58\") " pod="openshift-image-registry/image-registry-bcc64b9dc-p7sz5"
Sep 09 19:55:29.976592 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.976571    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbj5l\" (UniqueName: \"kubernetes.io/projected/8029d5a2-cb5c-4ab4-93cf-d9da4f3bcaea-kube-api-access-kbj5l\") pod \"ingress-canary-rq89t\" (UID: \"8029d5a2-cb5c-4ab4-93cf-d9da4f3bcaea\") " pod="openshift-ingress-canary/ingress-canary-rq89t"
Sep 09 19:55:29.977243 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:29.977220    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fb48971c-63be-4de1-b19e-0b8c1aadbc58-bound-sa-token\") pod \"image-registry-bcc64b9dc-p7sz5\" (UID: \"fb48971c-63be-4de1-b19e-0b8c1aadbc58\") " pod="openshift-image-registry/image-registry-bcc64b9dc-p7sz5"
Sep 09 19:55:30.066904 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:30.066870    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rq89t"
Sep 09 19:55:30.469966 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:30.469930    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fb48971c-63be-4de1-b19e-0b8c1aadbc58-registry-tls\") pod \"image-registry-bcc64b9dc-p7sz5\" (UID: \"fb48971c-63be-4de1-b19e-0b8c1aadbc58\") " pod="openshift-image-registry/image-registry-bcc64b9dc-p7sz5"
Sep 09 19:55:30.469966 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:30.469973    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d3bbfe8-34df-41e1-b959-766ff5100634-metrics-tls\") pod \"dns-default-zr55l\" (UID: \"1d3bbfe8-34df-41e1-b959-766ff5100634\") " pod="openshift-dns/dns-default-zr55l"
Sep 09 19:55:30.470459 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:30.470097    2523 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 19:55:30.470459 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:30.470108    2523 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 19:55:30.470459 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:30.470126    2523 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bcc64b9dc-p7sz5: secret "image-registry-tls" not found
Sep 09 19:55:30.470459 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:30.470172    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d3bbfe8-34df-41e1-b959-766ff5100634-metrics-tls podName:1d3bbfe8-34df-41e1-b959-766ff5100634 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:31.470136911 +0000 UTC m=+36.142155893 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1d3bbfe8-34df-41e1-b959-766ff5100634-metrics-tls") pod "dns-default-zr55l" (UID: "1d3bbfe8-34df-41e1-b959-766ff5100634") : secret "dns-default-metrics-tls" not found
Sep 09 19:55:30.470459 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:30.470212    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb48971c-63be-4de1-b19e-0b8c1aadbc58-registry-tls podName:fb48971c-63be-4de1-b19e-0b8c1aadbc58 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:31.470194997 +0000 UTC m=+36.142213985 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fb48971c-63be-4de1-b19e-0b8c1aadbc58-registry-tls") pod "image-registry-bcc64b9dc-p7sz5" (UID: "fb48971c-63be-4de1-b19e-0b8c1aadbc58") : secret "image-registry-tls" not found
Sep 09 19:55:31.477908 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:31.477741    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fb48971c-63be-4de1-b19e-0b8c1aadbc58-registry-tls\") pod \"image-registry-bcc64b9dc-p7sz5\" (UID: \"fb48971c-63be-4de1-b19e-0b8c1aadbc58\") " pod="openshift-image-registry/image-registry-bcc64b9dc-p7sz5"
Sep 09 19:55:31.478224 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:31.477926    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d3bbfe8-34df-41e1-b959-766ff5100634-metrics-tls\") pod \"dns-default-zr55l\" (UID: \"1d3bbfe8-34df-41e1-b959-766ff5100634\") " pod="openshift-dns/dns-default-zr55l"
Sep 09 19:55:31.478224 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:31.477854    2523 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 19:55:31.478224 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:31.477986    2523 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bcc64b9dc-p7sz5: secret "image-registry-tls" not found
Sep 09 19:55:31.478224 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:31.478018    2523 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 19:55:31.478224 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:31.478041    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb48971c-63be-4de1-b19e-0b8c1aadbc58-registry-tls podName:fb48971c-63be-4de1-b19e-0b8c1aadbc58 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:33.478026131 +0000 UTC m=+38.150045106 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fb48971c-63be-4de1-b19e-0b8c1aadbc58-registry-tls") pod "image-registry-bcc64b9dc-p7sz5" (UID: "fb48971c-63be-4de1-b19e-0b8c1aadbc58") : secret "image-registry-tls" not found
Sep 09 19:55:31.478224 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:31.478062    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d3bbfe8-34df-41e1-b959-766ff5100634-metrics-tls podName:1d3bbfe8-34df-41e1-b959-766ff5100634 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:33.478050619 +0000 UTC m=+38.150069594 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1d3bbfe8-34df-41e1-b959-766ff5100634-metrics-tls") pod "dns-default-zr55l" (UID: "1d3bbfe8-34df-41e1-b959-766ff5100634") : secret "dns-default-metrics-tls" not found
Sep 09 19:55:31.613946 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:31.613922    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rq89t"]
Sep 09 19:55:31.617990 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:55:31.617960    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8029d5a2_cb5c_4ab4_93cf_d9da4f3bcaea.slice/crio-2a6a931e1309294c9d49c458228e37336744192d05d57ce0ec2138400617ce17 WatchSource:0}: Error finding container 2a6a931e1309294c9d49c458228e37336744192d05d57ce0ec2138400617ce17: Status 404 returned error can't find the container with id 2a6a931e1309294c9d49c458228e37336744192d05d57ce0ec2138400617ce17
Sep 09 19:55:31.986314 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:31.986287    2523 generic.go:356] "Generic (PLEG): container finished" podID="556a154a-15d2-4313-93b6-42580b64443c" containerID="f776b4cc38518f76d4c900d0753ae0ba3082edff465da8de87f218538573e666" exitCode=0
Sep 09 19:55:31.986423 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:31.986359    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4jptv" event={"ID":"556a154a-15d2-4313-93b6-42580b64443c","Type":"ContainerDied","Data":"f776b4cc38518f76d4c900d0753ae0ba3082edff465da8de87f218538573e666"}
Sep 09 19:55:31.987604 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:31.987585    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rq89t" event={"ID":"8029d5a2-cb5c-4ab4-93cf-d9da4f3bcaea","Type":"ContainerStarted","Data":"2a6a931e1309294c9d49c458228e37336744192d05d57ce0ec2138400617ce17"}
Sep 09 19:55:32.994321 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:32.994278    2523 generic.go:356] "Generic (PLEG): container finished" podID="556a154a-15d2-4313-93b6-42580b64443c" containerID="072bd93c1bd945a9d332b53dcd8c5228a38533554a836b94c2b14b97f090b782" exitCode=0
Sep 09 19:55:32.994906 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:32.994332    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4jptv" event={"ID":"556a154a-15d2-4313-93b6-42580b64443c","Type":"ContainerDied","Data":"072bd93c1bd945a9d332b53dcd8c5228a38533554a836b94c2b14b97f090b782"}
Sep 09 19:55:33.493136 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:33.493091    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fb48971c-63be-4de1-b19e-0b8c1aadbc58-registry-tls\") pod \"image-registry-bcc64b9dc-p7sz5\" (UID: \"fb48971c-63be-4de1-b19e-0b8c1aadbc58\") " pod="openshift-image-registry/image-registry-bcc64b9dc-p7sz5"
Sep 09 19:55:33.493317 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:33.493169    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d3bbfe8-34df-41e1-b959-766ff5100634-metrics-tls\") pod \"dns-default-zr55l\" (UID: \"1d3bbfe8-34df-41e1-b959-766ff5100634\") " pod="openshift-dns/dns-default-zr55l"
Sep 09 19:55:33.493317 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:33.493266    2523 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 19:55:33.493317 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:33.493283    2523 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bcc64b9dc-p7sz5: secret "image-registry-tls" not found
Sep 09 19:55:33.493317 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:33.493288    2523 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 19:55:33.493503 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:33.493343    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d3bbfe8-34df-41e1-b959-766ff5100634-metrics-tls podName:1d3bbfe8-34df-41e1-b959-766ff5100634 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:37.493329175 +0000 UTC m=+42.165348150 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1d3bbfe8-34df-41e1-b959-766ff5100634-metrics-tls") pod "dns-default-zr55l" (UID: "1d3bbfe8-34df-41e1-b959-766ff5100634") : secret "dns-default-metrics-tls" not found
Sep 09 19:55:33.493503 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:33.493358    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb48971c-63be-4de1-b19e-0b8c1aadbc58-registry-tls podName:fb48971c-63be-4de1-b19e-0b8c1aadbc58 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:37.493352102 +0000 UTC m=+42.165371077 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fb48971c-63be-4de1-b19e-0b8c1aadbc58-registry-tls") pod "image-registry-bcc64b9dc-p7sz5" (UID: "fb48971c-63be-4de1-b19e-0b8c1aadbc58") : secret "image-registry-tls" not found
Sep 09 19:55:34.001643 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:34.001561    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4jptv" event={"ID":"556a154a-15d2-4313-93b6-42580b64443c","Type":"ContainerStarted","Data":"fab5c04bac54152064d3a48c79d6abb33696ccacbc0e27c277ad9c9e0e0759fe"}
Sep 09 19:55:34.025373 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:34.025313    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4jptv" podStartSLOduration=7.882142957 podStartE2EDuration="38.025294303s" totalImagesPullingTime="30.143151346s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:54:56 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:34.024039315 +0000 UTC m=+38.696058313" watchObservedRunningTime="2026-09-09 19:55:34.025294303 +0000 UTC m=+38.697313301"
Sep 09 19:55:35.005237 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:35.004998    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rq89t" event={"ID":"8029d5a2-cb5c-4ab4-93cf-d9da4f3bcaea","Type":"ContainerStarted","Data":"850fa7e5c33fdcb3e788430e35fe1edc5a0604bc31b8f1f7456e571efc1fcffc"}
Sep 09 19:55:35.035924 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:35.035881    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rq89t" podStartSLOduration=3.267748504 podStartE2EDuration="6.035866133s" totalImagesPullingTime="2.768117629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:55:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:35.034935078 +0000 UTC m=+39.706954076" watchObservedRunningTime="2026-09-09 19:55:35.035866133 +0000 UTC m=+39.707885127"
Sep 09 19:55:37.520810 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:37.520775    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fb48971c-63be-4de1-b19e-0b8c1aadbc58-registry-tls\") pod \"image-registry-bcc64b9dc-p7sz5\" (UID: \"fb48971c-63be-4de1-b19e-0b8c1aadbc58\") " pod="openshift-image-registry/image-registry-bcc64b9dc-p7sz5"
Sep 09 19:55:37.520810 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:37.520815    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d3bbfe8-34df-41e1-b959-766ff5100634-metrics-tls\") pod \"dns-default-zr55l\" (UID: \"1d3bbfe8-34df-41e1-b959-766ff5100634\") " pod="openshift-dns/dns-default-zr55l"
Sep 09 19:55:37.521332 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:37.520904    2523 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 19:55:37.521332 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:37.520951    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d3bbfe8-34df-41e1-b959-766ff5100634-metrics-tls podName:1d3bbfe8-34df-41e1-b959-766ff5100634 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:45.520938404 +0000 UTC m=+50.192957378 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1d3bbfe8-34df-41e1-b959-766ff5100634-metrics-tls") pod "dns-default-zr55l" (UID: "1d3bbfe8-34df-41e1-b959-766ff5100634") : secret "dns-default-metrics-tls" not found
Sep 09 19:55:37.521332 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:37.520953    2523 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 19:55:37.521332 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:37.520976    2523 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bcc64b9dc-p7sz5: secret "image-registry-tls" not found
Sep 09 19:55:37.521332 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:37.521025    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb48971c-63be-4de1-b19e-0b8c1aadbc58-registry-tls podName:fb48971c-63be-4de1-b19e-0b8c1aadbc58 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:45.521008711 +0000 UTC m=+50.193027693 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fb48971c-63be-4de1-b19e-0b8c1aadbc58-registry-tls") pod "image-registry-bcc64b9dc-p7sz5" (UID: "fb48971c-63be-4de1-b19e-0b8c1aadbc58") : secret "image-registry-tls" not found
Sep 09 19:55:45.573143 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:45.573100    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fb48971c-63be-4de1-b19e-0b8c1aadbc58-registry-tls\") pod \"image-registry-bcc64b9dc-p7sz5\" (UID: \"fb48971c-63be-4de1-b19e-0b8c1aadbc58\") " pod="openshift-image-registry/image-registry-bcc64b9dc-p7sz5"
Sep 09 19:55:45.573556 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:45.573186    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d3bbfe8-34df-41e1-b959-766ff5100634-metrics-tls\") pod \"dns-default-zr55l\" (UID: \"1d3bbfe8-34df-41e1-b959-766ff5100634\") " pod="openshift-dns/dns-default-zr55l"
Sep 09 19:55:45.573556 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:45.573244    2523 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 19:55:45.573556 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:45.573262    2523 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bcc64b9dc-p7sz5: secret "image-registry-tls" not found
Sep 09 19:55:45.573556 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:45.573289    2523 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 19:55:45.573556 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:45.573314    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb48971c-63be-4de1-b19e-0b8c1aadbc58-registry-tls podName:fb48971c-63be-4de1-b19e-0b8c1aadbc58 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:01.57329997 +0000 UTC m=+66.245318948 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fb48971c-63be-4de1-b19e-0b8c1aadbc58-registry-tls") pod "image-registry-bcc64b9dc-p7sz5" (UID: "fb48971c-63be-4de1-b19e-0b8c1aadbc58") : secret "image-registry-tls" not found
Sep 09 19:55:45.573556 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:45.573340    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d3bbfe8-34df-41e1-b959-766ff5100634-metrics-tls podName:1d3bbfe8-34df-41e1-b959-766ff5100634 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:01.573324559 +0000 UTC m=+66.245343541 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1d3bbfe8-34df-41e1-b959-766ff5100634-metrics-tls") pod "dns-default-zr55l" (UID: "1d3bbfe8-34df-41e1-b959-766ff5100634") : secret "dns-default-metrics-tls" not found
Sep 09 19:55:49.727145 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:49.727116    2523 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-b9f9c"]
Sep 09 19:55:49.728900 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:49.728885    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-b9f9c"
Sep 09 19:55:49.731455 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:49.731427    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 09 19:55:49.731455 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:49.731427    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 09 19:55:49.731626 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:49.731571    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 09 19:55:49.731626 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:49.731604    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-jwnsp\""
Sep 09 19:55:49.743862 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:49.743840    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-b9f9c"]
Sep 09 19:55:49.804257 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:49.804238    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f26e26cf-30ac-4401-af29-627376841db9-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-b9f9c\" (UID: \"f26e26cf-30ac-4401-af29-627376841db9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-b9f9c"
Sep 09 19:55:49.804358 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:49.804279    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sldm8\" (UniqueName: \"kubernetes.io/projected/f26e26cf-30ac-4401-af29-627376841db9-kube-api-access-sldm8\") pod \"cluster-samples-operator-7cdcfcf76c-b9f9c\" (UID: \"f26e26cf-30ac-4401-af29-627376841db9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-b9f9c"
Sep 09 19:55:49.814485 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:49.814465    2523 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-bxjtn"]
Sep 09 19:55:49.816232 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:49.816218    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-bxjtn"
Sep 09 19:55:49.818610 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:49.818587    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-g6mct\""
Sep 09 19:55:49.826048 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:49.826023    2523 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8zpvv"]
Sep 09 19:55:49.828101 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:49.828083    2523 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-fn5ll"]
Sep 09 19:55:49.828232 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:49.828216    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8zpvv"
Sep 09 19:55:49.829901 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:49.829883    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-bxjtn"]
Sep 09 19:55:49.829983 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:49.829954    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-fn5ll"
Sep 09 19:55:49.831906 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:49.831875    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 09 19:55:49.831906 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:49.831888    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 19:55:49.832051 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:49.831944    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 09 19:55:49.832235 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:49.832219    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 19:55:49.832314 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:49.832274    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-nkkmp\""
Sep 09 19:55:49.832418 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:49.832403    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 19:55:49.832483 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:49.832469    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-x2bqp\""
Sep 09 19:55:49.832483 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:49.832477    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 09 19:55:49.832582 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:49.832478    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 19:55:49.832740 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:49.832724    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 09 19:55:49.839052 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:49.839036    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 09 19:55:49.843756 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:49.843738    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8zpvv"]
Sep 09 19:55:49.844656 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:49.844637    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-fn5ll"]
Sep 09 19:55:49.904789 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:49.904765    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/72f24ca2-96db-441c-a051-9fbf6f361baa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-8zpvv\" (UID: \"72f24ca2-96db-441c-a051-9fbf6f361baa\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8zpvv"
Sep 09 19:55:49.904880 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:49.904795    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4568x\" (UniqueName: \"kubernetes.io/projected/a37fe983-a166-436c-82c3-cec3965409dc-kube-api-access-4568x\") pod \"network-check-source-685456655f-bxjtn\" (UID: \"a37fe983-a166-436c-82c3-cec3965409dc\") " pod="openshift-network-diagnostics/network-check-source-685456655f-bxjtn"
Sep 09 19:55:49.904880 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:49.904818    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd5hx\" (UniqueName: \"kubernetes.io/projected/72f24ca2-96db-441c-a051-9fbf6f361baa-kube-api-access-nd5hx\") pod \"cluster-monitoring-operator-6cdcc96965-8zpvv\" (UID: \"72f24ca2-96db-441c-a051-9fbf6f361baa\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8zpvv"
Sep 09 19:55:49.904880 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:49.904843    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e6f3e8a-9da3-48f8-a081-dfcfa2917b40-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-fn5ll\" (UID: \"9e6f3e8a-9da3-48f8-a081-dfcfa2917b40\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fn5ll"
Sep 09 19:55:49.904975 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:49.904919    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9e6f3e8a-9da3-48f8-a081-dfcfa2917b40-snapshots\") pod \"insights-operator-76bcd6ccf5-fn5ll\" (UID: \"9e6f3e8a-9da3-48f8-a081-dfcfa2917b40\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fn5ll"
Sep 09 19:55:49.904975 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:49.904959    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e6f3e8a-9da3-48f8-a081-dfcfa2917b40-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-fn5ll\" (UID: \"9e6f3e8a-9da3-48f8-a081-dfcfa2917b40\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fn5ll"
Sep 09 19:55:49.905036 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:49.904996    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f26e26cf-30ac-4401-af29-627376841db9-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-b9f9c\" (UID: \"f26e26cf-30ac-4401-af29-627376841db9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-b9f9c"
Sep 09 19:55:49.905036 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:49.905018    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w5zg\" (UniqueName: \"kubernetes.io/projected/9e6f3e8a-9da3-48f8-a081-dfcfa2917b40-kube-api-access-5w5zg\") pod \"insights-operator-76bcd6ccf5-fn5ll\" (UID: \"9e6f3e8a-9da3-48f8-a081-dfcfa2917b40\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fn5ll"
Sep 09 19:55:49.905096 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:49.905035    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/72f24ca2-96db-441c-a051-9fbf6f361baa-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-8zpvv\" (UID: \"72f24ca2-96db-441c-a051-9fbf6f361baa\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8zpvv"
Sep 09 19:55:49.905096 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:49.905069    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sldm8\" (UniqueName: \"kubernetes.io/projected/f26e26cf-30ac-4401-af29-627376841db9-kube-api-access-sldm8\") pod \"cluster-samples-operator-7cdcfcf76c-b9f9c\" (UID: \"f26e26cf-30ac-4401-af29-627376841db9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-b9f9c"
Sep 09 19:55:49.905096 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:49.905084    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9e6f3e8a-9da3-48f8-a081-dfcfa2917b40-serving-cert\") pod \"insights-operator-76bcd6ccf5-fn5ll\" (UID: \"9e6f3e8a-9da3-48f8-a081-dfcfa2917b40\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fn5ll"
Sep 09 19:55:49.905283 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:49.905101    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e6f3e8a-9da3-48f8-a081-dfcfa2917b40-tmp\") pod \"insights-operator-76bcd6ccf5-fn5ll\" (UID: \"9e6f3e8a-9da3-48f8-a081-dfcfa2917b40\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fn5ll"
Sep 09 19:55:49.905283 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:49.905110    2523 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 19:55:49.905283 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:49.905189    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f26e26cf-30ac-4401-af29-627376841db9-samples-operator-tls podName:f26e26cf-30ac-4401-af29-627376841db9 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:50.405170289 +0000 UTC m=+55.077189281 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f26e26cf-30ac-4401-af29-627376841db9-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-b9f9c" (UID: "f26e26cf-30ac-4401-af29-627376841db9") : secret "samples-operator-tls" not found
Sep 09 19:55:49.914660 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:49.914642    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sldm8\" (UniqueName: \"kubernetes.io/projected/f26e26cf-30ac-4401-af29-627376841db9-kube-api-access-sldm8\") pod \"cluster-samples-operator-7cdcfcf76c-b9f9c\" (UID: \"f26e26cf-30ac-4401-af29-627376841db9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-b9f9c"
Sep 09 19:55:50.005741 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:50.005697    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w5zg\" (UniqueName: \"kubernetes.io/projected/9e6f3e8a-9da3-48f8-a081-dfcfa2917b40-kube-api-access-5w5zg\") pod \"insights-operator-76bcd6ccf5-fn5ll\" (UID: \"9e6f3e8a-9da3-48f8-a081-dfcfa2917b40\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fn5ll"
Sep 09 19:55:50.005741 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:50.005724    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/72f24ca2-96db-441c-a051-9fbf6f361baa-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-8zpvv\" (UID: \"72f24ca2-96db-441c-a051-9fbf6f361baa\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8zpvv"
Sep 09 19:55:50.005890 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:50.005751    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9e6f3e8a-9da3-48f8-a081-dfcfa2917b40-serving-cert\") pod \"insights-operator-76bcd6ccf5-fn5ll\" (UID: \"9e6f3e8a-9da3-48f8-a081-dfcfa2917b40\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fn5ll"
Sep 09 19:55:50.005890 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:50.005766    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e6f3e8a-9da3-48f8-a081-dfcfa2917b40-tmp\") pod \"insights-operator-76bcd6ccf5-fn5ll\" (UID: \"9e6f3e8a-9da3-48f8-a081-dfcfa2917b40\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fn5ll"
Sep 09 19:55:50.005890 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:50.005797    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/72f24ca2-96db-441c-a051-9fbf6f361baa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-8zpvv\" (UID: \"72f24ca2-96db-441c-a051-9fbf6f361baa\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8zpvv"
Sep 09 19:55:50.005890 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:50.005846    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4568x\" (UniqueName: \"kubernetes.io/projected/a37fe983-a166-436c-82c3-cec3965409dc-kube-api-access-4568x\") pod \"network-check-source-685456655f-bxjtn\" (UID: \"a37fe983-a166-436c-82c3-cec3965409dc\") " pod="openshift-network-diagnostics/network-check-source-685456655f-bxjtn"
Sep 09 19:55:50.005890 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:50.005867    2523 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 19:55:50.006135 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:50.005888    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd5hx\" (UniqueName: \"kubernetes.io/projected/72f24ca2-96db-441c-a051-9fbf6f361baa-kube-api-access-nd5hx\") pod \"cluster-monitoring-operator-6cdcc96965-8zpvv\" (UID: \"72f24ca2-96db-441c-a051-9fbf6f361baa\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8zpvv"
Sep 09 19:55:50.006135 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:50.005915    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72f24ca2-96db-441c-a051-9fbf6f361baa-cluster-monitoring-operator-tls podName:72f24ca2-96db-441c-a051-9fbf6f361baa nodeName:}" failed. No retries permitted until 2026-09-09 19:55:50.505901131 +0000 UTC m=+55.177920106 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/72f24ca2-96db-441c-a051-9fbf6f361baa-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-8zpvv" (UID: "72f24ca2-96db-441c-a051-9fbf6f361baa") : secret "cluster-monitoring-operator-tls" not found
Sep 09 19:55:50.006135 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:50.005931    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e6f3e8a-9da3-48f8-a081-dfcfa2917b40-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-fn5ll\" (UID: \"9e6f3e8a-9da3-48f8-a081-dfcfa2917b40\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fn5ll"
Sep 09 19:55:50.006135 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:50.005979    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9e6f3e8a-9da3-48f8-a081-dfcfa2917b40-snapshots\") pod \"insights-operator-76bcd6ccf5-fn5ll\" (UID: \"9e6f3e8a-9da3-48f8-a081-dfcfa2917b40\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fn5ll"
Sep 09 19:55:50.006135 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:50.006016    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e6f3e8a-9da3-48f8-a081-dfcfa2917b40-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-fn5ll\" (UID: \"9e6f3e8a-9da3-48f8-a081-dfcfa2917b40\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fn5ll"
Sep 09 19:55:50.006135 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:50.006131    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e6f3e8a-9da3-48f8-a081-dfcfa2917b40-tmp\") pod \"insights-operator-76bcd6ccf5-fn5ll\" (UID: \"9e6f3e8a-9da3-48f8-a081-dfcfa2917b40\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fn5ll"
Sep 09 19:55:50.006512 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:50.006492    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9e6f3e8a-9da3-48f8-a081-dfcfa2917b40-snapshots\") pod \"insights-operator-76bcd6ccf5-fn5ll\" (UID: \"9e6f3e8a-9da3-48f8-a081-dfcfa2917b40\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fn5ll"
Sep 09 19:55:50.006512 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:50.006503    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/72f24ca2-96db-441c-a051-9fbf6f361baa-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-8zpvv\" (UID: \"72f24ca2-96db-441c-a051-9fbf6f361baa\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8zpvv"
Sep 09 19:55:50.006620 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:50.006593    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e6f3e8a-9da3-48f8-a081-dfcfa2917b40-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-fn5ll\" (UID: \"9e6f3e8a-9da3-48f8-a081-dfcfa2917b40\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fn5ll"
Sep 09 19:55:50.007301 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:50.007281    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e6f3e8a-9da3-48f8-a081-dfcfa2917b40-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-fn5ll\" (UID: \"9e6f3e8a-9da3-48f8-a081-dfcfa2917b40\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fn5ll"
Sep 09 19:55:50.008234 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:50.008219    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9e6f3e8a-9da3-48f8-a081-dfcfa2917b40-serving-cert\") pod \"insights-operator-76bcd6ccf5-fn5ll\" (UID: \"9e6f3e8a-9da3-48f8-a081-dfcfa2917b40\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fn5ll"
Sep 09 19:55:50.014678 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:50.014656    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4568x\" (UniqueName: \"kubernetes.io/projected/a37fe983-a166-436c-82c3-cec3965409dc-kube-api-access-4568x\") pod \"network-check-source-685456655f-bxjtn\" (UID: \"a37fe983-a166-436c-82c3-cec3965409dc\") " pod="openshift-network-diagnostics/network-check-source-685456655f-bxjtn"
Sep 09 19:55:50.014990 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:50.014968    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w5zg\" (UniqueName: \"kubernetes.io/projected/9e6f3e8a-9da3-48f8-a081-dfcfa2917b40-kube-api-access-5w5zg\") pod \"insights-operator-76bcd6ccf5-fn5ll\" (UID: \"9e6f3e8a-9da3-48f8-a081-dfcfa2917b40\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fn5ll"
Sep 09 19:55:50.015103 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:50.015088    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd5hx\" (UniqueName: \"kubernetes.io/projected/72f24ca2-96db-441c-a051-9fbf6f361baa-kube-api-access-nd5hx\") pod \"cluster-monitoring-operator-6cdcc96965-8zpvv\" (UID: \"72f24ca2-96db-441c-a051-9fbf6f361baa\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8zpvv"
Sep 09 19:55:50.123646 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:50.123627    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-bxjtn"
Sep 09 19:55:50.145371 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:50.145352    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-fn5ll"
Sep 09 19:55:50.394713 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:50.394689    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-bxjtn"]
Sep 09 19:55:50.397674 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:55:50.397644    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda37fe983_a166_436c_82c3_cec3965409dc.slice/crio-ca8fee15a15b0dd78417ed8a2bd157f844b4ef6b070994a51a3d741a8f258f11 WatchSource:0}: Error finding container ca8fee15a15b0dd78417ed8a2bd157f844b4ef6b070994a51a3d741a8f258f11: Status 404 returned error can't find the container with id ca8fee15a15b0dd78417ed8a2bd157f844b4ef6b070994a51a3d741a8f258f11
Sep 09 19:55:50.409043 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:50.409022    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f26e26cf-30ac-4401-af29-627376841db9-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-b9f9c\" (UID: \"f26e26cf-30ac-4401-af29-627376841db9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-b9f9c"
Sep 09 19:55:50.409192 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:50.409177    2523 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 19:55:50.409267 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:50.409244    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f26e26cf-30ac-4401-af29-627376841db9-samples-operator-tls podName:f26e26cf-30ac-4401-af29-627376841db9 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:51.409221724 +0000 UTC m=+56.081240716 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f26e26cf-30ac-4401-af29-627376841db9-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-b9f9c" (UID: "f26e26cf-30ac-4401-af29-627376841db9") : secret "samples-operator-tls" not found
Sep 09 19:55:50.423658 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:50.423636    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-fn5ll"]
Sep 09 19:55:50.426311 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:55:50.426287    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e6f3e8a_9da3_48f8_a081_dfcfa2917b40.slice/crio-78f4ecbee4b2773ce277254280fb4e84bb43dfeb2ef38a8bf54e33cf5c7a6506 WatchSource:0}: Error finding container 78f4ecbee4b2773ce277254280fb4e84bb43dfeb2ef38a8bf54e33cf5c7a6506: Status 404 returned error can't find the container with id 78f4ecbee4b2773ce277254280fb4e84bb43dfeb2ef38a8bf54e33cf5c7a6506
Sep 09 19:55:50.510110 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:50.510087    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/72f24ca2-96db-441c-a051-9fbf6f361baa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-8zpvv\" (UID: \"72f24ca2-96db-441c-a051-9fbf6f361baa\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8zpvv"
Sep 09 19:55:50.510232 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:50.510220    2523 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 19:55:50.510281 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:50.510267    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72f24ca2-96db-441c-a051-9fbf6f361baa-cluster-monitoring-operator-tls podName:72f24ca2-96db-441c-a051-9fbf6f361baa nodeName:}" failed. No retries permitted until 2026-09-09 19:55:51.510254657 +0000 UTC m=+56.182273632 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/72f24ca2-96db-441c-a051-9fbf6f361baa-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-8zpvv" (UID: "72f24ca2-96db-441c-a051-9fbf6f361baa") : secret "cluster-monitoring-operator-tls" not found
Sep 09 19:55:51.041167 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:51.041111    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-fn5ll" event={"ID":"9e6f3e8a-9da3-48f8-a081-dfcfa2917b40","Type":"ContainerStarted","Data":"78f4ecbee4b2773ce277254280fb4e84bb43dfeb2ef38a8bf54e33cf5c7a6506"}
Sep 09 19:55:51.042977 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:51.042913    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-bxjtn" event={"ID":"a37fe983-a166-436c-82c3-cec3965409dc","Type":"ContainerStarted","Data":"ca8fee15a15b0dd78417ed8a2bd157f844b4ef6b070994a51a3d741a8f258f11"}
Sep 09 19:55:51.418982 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:51.418946    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f26e26cf-30ac-4401-af29-627376841db9-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-b9f9c\" (UID: \"f26e26cf-30ac-4401-af29-627376841db9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-b9f9c"
Sep 09 19:55:51.419252 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:51.419118    2523 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 19:55:51.419252 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:51.419206    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f26e26cf-30ac-4401-af29-627376841db9-samples-operator-tls podName:f26e26cf-30ac-4401-af29-627376841db9 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:53.419189431 +0000 UTC m=+58.091208430 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f26e26cf-30ac-4401-af29-627376841db9-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-b9f9c" (UID: "f26e26cf-30ac-4401-af29-627376841db9") : secret "samples-operator-tls" not found
Sep 09 19:55:51.519785 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:51.519720    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/72f24ca2-96db-441c-a051-9fbf6f361baa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-8zpvv\" (UID: \"72f24ca2-96db-441c-a051-9fbf6f361baa\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8zpvv"
Sep 09 19:55:51.519918 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:51.519880    2523 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 19:55:51.519976 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:51.519948    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72f24ca2-96db-441c-a051-9fbf6f361baa-cluster-monitoring-operator-tls podName:72f24ca2-96db-441c-a051-9fbf6f361baa nodeName:}" failed. No retries permitted until 2026-09-09 19:55:53.519926972 +0000 UTC m=+58.191945976 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/72f24ca2-96db-441c-a051-9fbf6f361baa-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-8zpvv" (UID: "72f24ca2-96db-441c-a051-9fbf6f361baa") : secret "cluster-monitoring-operator-tls" not found
Sep 09 19:55:53.438126 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:53.438088    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f26e26cf-30ac-4401-af29-627376841db9-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-b9f9c\" (UID: \"f26e26cf-30ac-4401-af29-627376841db9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-b9f9c"
Sep 09 19:55:53.438525 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:53.438232    2523 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 19:55:53.438525 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:53.438296    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f26e26cf-30ac-4401-af29-627376841db9-samples-operator-tls podName:f26e26cf-30ac-4401-af29-627376841db9 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:57.438280937 +0000 UTC m=+62.110299911 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f26e26cf-30ac-4401-af29-627376841db9-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-b9f9c" (UID: "f26e26cf-30ac-4401-af29-627376841db9") : secret "samples-operator-tls" not found
Sep 09 19:55:53.538916 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:53.538876    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/72f24ca2-96db-441c-a051-9fbf6f361baa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-8zpvv\" (UID: \"72f24ca2-96db-441c-a051-9fbf6f361baa\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8zpvv"
Sep 09 19:55:53.539088 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:53.539043    2523 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 19:55:53.539141 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:53.539108    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72f24ca2-96db-441c-a051-9fbf6f361baa-cluster-monitoring-operator-tls podName:72f24ca2-96db-441c-a051-9fbf6f361baa nodeName:}" failed. No retries permitted until 2026-09-09 19:55:57.539092169 +0000 UTC m=+62.211111145 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/72f24ca2-96db-441c-a051-9fbf6f361baa-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-8zpvv" (UID: "72f24ca2-96db-441c-a051-9fbf6f361baa") : secret "cluster-monitoring-operator-tls" not found
Sep 09 19:55:54.052716 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:54.052631    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-fn5ll" event={"ID":"9e6f3e8a-9da3-48f8-a081-dfcfa2917b40","Type":"ContainerStarted","Data":"f6cb2b8d676e26fb78dea4d506eb8e57a41bc7ed44f983c2a4e9c4594a6a4821"}
Sep 09 19:55:54.054412 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:54.054384    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-bxjtn" event={"ID":"a37fe983-a166-436c-82c3-cec3965409dc","Type":"ContainerStarted","Data":"5b00b1ff36ceccb605f82de11dded7245a7ae6ad6fe2543fa68290d5f2b90d88"}
Sep 09 19:55:54.069586 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:54.069537    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-fn5ll" podStartSLOduration=1.6966035640000001 podStartE2EDuration="5.069512938s" totalImagesPullingTime="3.372909374s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:55:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:54.068377956 +0000 UTC m=+58.740396953" watchObservedRunningTime="2026-09-09 19:55:54.069512938 +0000 UTC m=+58.741531971"
Sep 09 19:55:54.083116 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:54.083068    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-bxjtn" podStartSLOduration=1.656664065 podStartE2EDuration="5.083049465s" totalImagesPullingTime="3.4263854s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:55:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:54.081793529 +0000 UTC m=+58.753812526" watchObservedRunningTime="2026-09-09 19:55:54.083049465 +0000 UTC m=+58.755068463"
Sep 09 19:55:55.259278 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:55.259245    2523 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-z7hmm"]
Sep 09 19:55:55.261531 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:55.261510    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-z7hmm"
Sep 09 19:55:55.263740 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:55.263720    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 19:55:55.263821 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:55.263724    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-57xxp\""
Sep 09 19:55:55.264604 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:55.264587    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 19:55:55.271439 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:55.271421    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-z7hmm"]
Sep 09 19:55:55.353647 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:55.353621    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t9kv\" (UniqueName: \"kubernetes.io/projected/27f5043a-c597-4d58-9a73-c0b1cd9071b3-kube-api-access-9t9kv\") pod \"insights-runtime-extractor-z7hmm\" (UID: \"27f5043a-c597-4d58-9a73-c0b1cd9071b3\") " pod="openshift-insights/insights-runtime-extractor-z7hmm"
Sep 09 19:55:55.353741 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:55.353657    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/27f5043a-c597-4d58-9a73-c0b1cd9071b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z7hmm\" (UID: \"27f5043a-c597-4d58-9a73-c0b1cd9071b3\") " pod="openshift-insights/insights-runtime-extractor-z7hmm"
Sep 09 19:55:55.353741 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:55.353685    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/27f5043a-c597-4d58-9a73-c0b1cd9071b3-crio-socket\") pod \"insights-runtime-extractor-z7hmm\" (UID: \"27f5043a-c597-4d58-9a73-c0b1cd9071b3\") " pod="openshift-insights/insights-runtime-extractor-z7hmm"
Sep 09 19:55:55.353822 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:55.353784    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/27f5043a-c597-4d58-9a73-c0b1cd9071b3-data-volume\") pod \"insights-runtime-extractor-z7hmm\" (UID: \"27f5043a-c597-4d58-9a73-c0b1cd9071b3\") " pod="openshift-insights/insights-runtime-extractor-z7hmm"
Sep 09 19:55:55.353857 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:55.353832    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/27f5043a-c597-4d58-9a73-c0b1cd9071b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z7hmm\" (UID: \"27f5043a-c597-4d58-9a73-c0b1cd9071b3\") " pod="openshift-insights/insights-runtime-extractor-z7hmm"
Sep 09 19:55:55.455001 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:55.454975    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/27f5043a-c597-4d58-9a73-c0b1cd9071b3-data-volume\") pod \"insights-runtime-extractor-z7hmm\" (UID: \"27f5043a-c597-4d58-9a73-c0b1cd9071b3\") " pod="openshift-insights/insights-runtime-extractor-z7hmm"
Sep 09 19:55:55.455087 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:55.455026    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/27f5043a-c597-4d58-9a73-c0b1cd9071b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z7hmm\" (UID: \"27f5043a-c597-4d58-9a73-c0b1cd9071b3\") " pod="openshift-insights/insights-runtime-extractor-z7hmm"
Sep 09 19:55:55.455087 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:55.455061    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t9kv\" (UniqueName: \"kubernetes.io/projected/27f5043a-c597-4d58-9a73-c0b1cd9071b3-kube-api-access-9t9kv\") pod \"insights-runtime-extractor-z7hmm\" (UID: \"27f5043a-c597-4d58-9a73-c0b1cd9071b3\") " pod="openshift-insights/insights-runtime-extractor-z7hmm"
Sep 09 19:55:55.455198 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:55.455087    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/27f5043a-c597-4d58-9a73-c0b1cd9071b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z7hmm\" (UID: \"27f5043a-c597-4d58-9a73-c0b1cd9071b3\") " pod="openshift-insights/insights-runtime-extractor-z7hmm"
Sep 09 19:55:55.455198 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:55.455109    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/27f5043a-c597-4d58-9a73-c0b1cd9071b3-crio-socket\") pod \"insights-runtime-extractor-z7hmm\" (UID: \"27f5043a-c597-4d58-9a73-c0b1cd9071b3\") " pod="openshift-insights/insights-runtime-extractor-z7hmm"
Sep 09 19:55:55.455296 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:55.455211    2523 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 19:55:55.455296 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:55.455229    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/27f5043a-c597-4d58-9a73-c0b1cd9071b3-crio-socket\") pod \"insights-runtime-extractor-z7hmm\" (UID: \"27f5043a-c597-4d58-9a73-c0b1cd9071b3\") " pod="openshift-insights/insights-runtime-extractor-z7hmm"
Sep 09 19:55:55.455296 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:55.455263    2523 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 19:55:55.455418 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:55.455297    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/27f5043a-c597-4d58-9a73-c0b1cd9071b3-data-volume\") pod \"insights-runtime-extractor-z7hmm\" (UID: \"27f5043a-c597-4d58-9a73-c0b1cd9071b3\") " pod="openshift-insights/insights-runtime-extractor-z7hmm"
Sep 09 19:55:55.455418 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:55.455284    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/27f5043a-c597-4d58-9a73-c0b1cd9071b3-kube-rbac-proxy-cm podName:27f5043a-c597-4d58-9a73-c0b1cd9071b3 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:55.95526549 +0000 UTC m=+60.627284465 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/27f5043a-c597-4d58-9a73-c0b1cd9071b3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-z7hmm" (UID: "27f5043a-c597-4d58-9a73-c0b1cd9071b3") : configmap "kube-rbac-proxy" not found
Sep 09 19:55:55.455418 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:55.455347    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27f5043a-c597-4d58-9a73-c0b1cd9071b3-insights-runtime-extractor-tls podName:27f5043a-c597-4d58-9a73-c0b1cd9071b3 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:55.95532965 +0000 UTC m=+60.627348629 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/27f5043a-c597-4d58-9a73-c0b1cd9071b3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-z7hmm" (UID: "27f5043a-c597-4d58-9a73-c0b1cd9071b3") : secret "insights-runtime-extractor-tls" not found
Sep 09 19:55:55.465456 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:55.465437    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t9kv\" (UniqueName: \"kubernetes.io/projected/27f5043a-c597-4d58-9a73-c0b1cd9071b3-kube-api-access-9t9kv\") pod \"insights-runtime-extractor-z7hmm\" (UID: \"27f5043a-c597-4d58-9a73-c0b1cd9071b3\") " pod="openshift-insights/insights-runtime-extractor-z7hmm"
Sep 09 19:55:55.958333 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:55.958302    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/27f5043a-c597-4d58-9a73-c0b1cd9071b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z7hmm\" (UID: \"27f5043a-c597-4d58-9a73-c0b1cd9071b3\") " pod="openshift-insights/insights-runtime-extractor-z7hmm"
Sep 09 19:55:55.958471 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:55.958397    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/27f5043a-c597-4d58-9a73-c0b1cd9071b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z7hmm\" (UID: \"27f5043a-c597-4d58-9a73-c0b1cd9071b3\") " pod="openshift-insights/insights-runtime-extractor-z7hmm"
Sep 09 19:55:55.958471 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:55.958425    2523 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 19:55:55.958540 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:55.958472    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27f5043a-c597-4d58-9a73-c0b1cd9071b3-insights-runtime-extractor-tls podName:27f5043a-c597-4d58-9a73-c0b1cd9071b3 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:56.958458553 +0000 UTC m=+61.630477527 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/27f5043a-c597-4d58-9a73-c0b1cd9071b3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-z7hmm" (UID: "27f5043a-c597-4d58-9a73-c0b1cd9071b3") : secret "insights-runtime-extractor-tls" not found
Sep 09 19:55:55.958540 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:55.958475    2523 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 19:55:55.958540 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:55.958511    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/27f5043a-c597-4d58-9a73-c0b1cd9071b3-kube-rbac-proxy-cm podName:27f5043a-c597-4d58-9a73-c0b1cd9071b3 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:56.95849979 +0000 UTC m=+61.630518765 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/27f5043a-c597-4d58-9a73-c0b1cd9071b3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-z7hmm" (UID: "27f5043a-c597-4d58-9a73-c0b1cd9071b3") : configmap "kube-rbac-proxy" not found
Sep 09 19:55:55.980832 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:55.980807    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c72g2"
Sep 09 19:55:56.966846 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:56.966818    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/27f5043a-c597-4d58-9a73-c0b1cd9071b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z7hmm\" (UID: \"27f5043a-c597-4d58-9a73-c0b1cd9071b3\") " pod="openshift-insights/insights-runtime-extractor-z7hmm"
Sep 09 19:55:56.967275 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:56.966890    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/27f5043a-c597-4d58-9a73-c0b1cd9071b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z7hmm\" (UID: \"27f5043a-c597-4d58-9a73-c0b1cd9071b3\") " pod="openshift-insights/insights-runtime-extractor-z7hmm"
Sep 09 19:55:56.967275 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:56.966971    2523 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 19:55:56.967275 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:56.966977    2523 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 19:55:56.967275 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:56.967018    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/27f5043a-c597-4d58-9a73-c0b1cd9071b3-kube-rbac-proxy-cm podName:27f5043a-c597-4d58-9a73-c0b1cd9071b3 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:58.967005466 +0000 UTC m=+63.639024441 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/27f5043a-c597-4d58-9a73-c0b1cd9071b3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-z7hmm" (UID: "27f5043a-c597-4d58-9a73-c0b1cd9071b3") : configmap "kube-rbac-proxy" not found
Sep 09 19:55:56.967275 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:56.967041    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27f5043a-c597-4d58-9a73-c0b1cd9071b3-insights-runtime-extractor-tls podName:27f5043a-c597-4d58-9a73-c0b1cd9071b3 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:58.96702357 +0000 UTC m=+63.639042552 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/27f5043a-c597-4d58-9a73-c0b1cd9071b3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-z7hmm" (UID: "27f5043a-c597-4d58-9a73-c0b1cd9071b3") : secret "insights-runtime-extractor-tls" not found
Sep 09 19:55:57.407079 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:57.407053    2523 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-w2tb2_8fe231fb-4f9c-478e-b2c5-95a65a627385/dns-node-resolver/0.log"
Sep 09 19:55:57.473934 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:57.473909    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f26e26cf-30ac-4401-af29-627376841db9-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-b9f9c\" (UID: \"f26e26cf-30ac-4401-af29-627376841db9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-b9f9c"
Sep 09 19:55:57.474086 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:57.474052    2523 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 19:55:57.474129 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:57.474106    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f26e26cf-30ac-4401-af29-627376841db9-samples-operator-tls podName:f26e26cf-30ac-4401-af29-627376841db9 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:05.47409177 +0000 UTC m=+70.146110744 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f26e26cf-30ac-4401-af29-627376841db9-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-b9f9c" (UID: "f26e26cf-30ac-4401-af29-627376841db9") : secret "samples-operator-tls" not found
Sep 09 19:55:57.574989 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:57.574966    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/72f24ca2-96db-441c-a051-9fbf6f361baa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-8zpvv\" (UID: \"72f24ca2-96db-441c-a051-9fbf6f361baa\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8zpvv"
Sep 09 19:55:57.575123 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:57.575107    2523 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 19:55:57.575192 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:57.575177    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72f24ca2-96db-441c-a051-9fbf6f361baa-cluster-monitoring-operator-tls podName:72f24ca2-96db-441c-a051-9fbf6f361baa nodeName:}" failed. No retries permitted until 2026-09-09 19:56:05.575163396 +0000 UTC m=+70.247182384 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/72f24ca2-96db-441c-a051-9fbf6f361baa-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-8zpvv" (UID: "72f24ca2-96db-441c-a051-9fbf6f361baa") : secret "cluster-monitoring-operator-tls" not found
Sep 09 19:55:58.007875 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:58.007853    2523 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ctlrg_8d8ec74b-cbcb-4c36-a904-21e6a6460f6b/node-ca/0.log"
Sep 09 19:55:58.987305 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:58.987276    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/27f5043a-c597-4d58-9a73-c0b1cd9071b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z7hmm\" (UID: \"27f5043a-c597-4d58-9a73-c0b1cd9071b3\") " pod="openshift-insights/insights-runtime-extractor-z7hmm"
Sep 09 19:55:58.987446 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:58.987349    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/27f5043a-c597-4d58-9a73-c0b1cd9071b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z7hmm\" (UID: \"27f5043a-c597-4d58-9a73-c0b1cd9071b3\") " pod="openshift-insights/insights-runtime-extractor-z7hmm"
Sep 09 19:55:58.987446 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:58.987420    2523 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 19:55:58.987575 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:58.987452    2523 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 19:55:58.987575 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:58.987491    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27f5043a-c597-4d58-9a73-c0b1cd9071b3-insights-runtime-extractor-tls podName:27f5043a-c597-4d58-9a73-c0b1cd9071b3 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:02.987476899 +0000 UTC m=+67.659495874 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/27f5043a-c597-4d58-9a73-c0b1cd9071b3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-z7hmm" (UID: "27f5043a-c597-4d58-9a73-c0b1cd9071b3") : secret "insights-runtime-extractor-tls" not found
Sep 09 19:55:58.987575 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:55:58.987506    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/27f5043a-c597-4d58-9a73-c0b1cd9071b3-kube-rbac-proxy-cm podName:27f5043a-c597-4d58-9a73-c0b1cd9071b3 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:02.98749932 +0000 UTC m=+67.659518294 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/27f5043a-c597-4d58-9a73-c0b1cd9071b3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-z7hmm" (UID: "27f5043a-c597-4d58-9a73-c0b1cd9071b3") : configmap "kube-rbac-proxy" not found
Sep 09 19:55:59.207744 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:55:59.207722    2523 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rq89t_8029d5a2-cb5c-4ab4-93cf-d9da4f3bcaea/serve-healthcheck-canary/0.log"
Sep 09 19:56:01.505268 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:01.505235    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8-metrics-certs\") pod \"network-metrics-daemon-tmvpv\" (UID: \"21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8\") " pod="openshift-multus/network-metrics-daemon-tmvpv"
Sep 09 19:56:01.505685 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:01.505293    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2cc1cab-a99d-4733-8623-2ccf3fb89961-original-pull-secret\") pod \"global-pull-secret-syncer-gn8cc\" (UID: \"b2cc1cab-a99d-4733-8623-2ccf3fb89961\") " pod="kube-system/global-pull-secret-syncer-gn8cc"
Sep 09 19:56:01.508359 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:01.508342    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 19:56:01.508407 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:01.508386    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 19:56:01.515781 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:56:01.515760    2523 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 19:56:01.515829 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:56:01.515814    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8-metrics-certs podName:21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8 nodeName:}" failed. No retries permitted until 2026-09-09 19:57:05.515800852 +0000 UTC m=+130.187819826 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8-metrics-certs") pod "network-metrics-daemon-tmvpv" (UID: "21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8") : secret "metrics-daemon-secret" not found
Sep 09 19:56:01.518095 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:01.518075    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2cc1cab-a99d-4733-8623-2ccf3fb89961-original-pull-secret\") pod \"global-pull-secret-syncer-gn8cc\" (UID: \"b2cc1cab-a99d-4733-8623-2ccf3fb89961\") " pod="kube-system/global-pull-secret-syncer-gn8cc"
Sep 09 19:56:01.605651 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:01.605622    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfddc\" (UniqueName: \"kubernetes.io/projected/d5e04e1e-56ef-4e29-89bd-eafcf7425d56-kube-api-access-hfddc\") pod \"network-check-target-jcx8l\" (UID: \"d5e04e1e-56ef-4e29-89bd-eafcf7425d56\") " pod="openshift-network-diagnostics/network-check-target-jcx8l"
Sep 09 19:56:01.605764 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:01.605690    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fb48971c-63be-4de1-b19e-0b8c1aadbc58-registry-tls\") pod \"image-registry-bcc64b9dc-p7sz5\" (UID: \"fb48971c-63be-4de1-b19e-0b8c1aadbc58\") " pod="openshift-image-registry/image-registry-bcc64b9dc-p7sz5"
Sep 09 19:56:01.605764 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:01.605713    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d3bbfe8-34df-41e1-b959-766ff5100634-metrics-tls\") pod \"dns-default-zr55l\" (UID: \"1d3bbfe8-34df-41e1-b959-766ff5100634\") " pod="openshift-dns/dns-default-zr55l"
Sep 09 19:56:01.605832 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:56:01.605767    2523 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 19:56:01.605832 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:56:01.605784    2523 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bcc64b9dc-p7sz5: secret "image-registry-tls" not found
Sep 09 19:56:01.605832 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:56:01.605797    2523 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 19:56:01.605917 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:56:01.605838    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb48971c-63be-4de1-b19e-0b8c1aadbc58-registry-tls podName:fb48971c-63be-4de1-b19e-0b8c1aadbc58 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:33.605822528 +0000 UTC m=+98.277841519 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fb48971c-63be-4de1-b19e-0b8c1aadbc58-registry-tls") pod "image-registry-bcc64b9dc-p7sz5" (UID: "fb48971c-63be-4de1-b19e-0b8c1aadbc58") : secret "image-registry-tls" not found
Sep 09 19:56:01.605917 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:56:01.605855    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d3bbfe8-34df-41e1-b959-766ff5100634-metrics-tls podName:1d3bbfe8-34df-41e1-b959-766ff5100634 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:33.605846277 +0000 UTC m=+98.277865257 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1d3bbfe8-34df-41e1-b959-766ff5100634-metrics-tls") pod "dns-default-zr55l" (UID: "1d3bbfe8-34df-41e1-b959-766ff5100634") : secret "dns-default-metrics-tls" not found
Sep 09 19:56:01.607946 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:01.607930    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfddc\" (UniqueName: \"kubernetes.io/projected/d5e04e1e-56ef-4e29-89bd-eafcf7425d56-kube-api-access-hfddc\") pod \"network-check-target-jcx8l\" (UID: \"d5e04e1e-56ef-4e29-89bd-eafcf7425d56\") " pod="openshift-network-diagnostics/network-check-target-jcx8l"
Sep 09 19:56:01.616278 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:01.616258    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rfksv\""
Sep 09 19:56:01.621613 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:01.621600    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gn8cc"
Sep 09 19:56:01.623736 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:01.623721    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jcx8l"
Sep 09 19:56:01.898438 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:01.898401    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gn8cc"]
Sep 09 19:56:01.900964 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:01.900942    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jcx8l"]
Sep 09 19:56:01.901371 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:56:01.901334    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2cc1cab_a99d_4733_8623_2ccf3fb89961.slice/crio-271d41bb022ec3d065da12ec895bf8d9cc8a2dce5bb843665d6bbf2fd5081239 WatchSource:0}: Error finding container 271d41bb022ec3d065da12ec895bf8d9cc8a2dce5bb843665d6bbf2fd5081239: Status 404 returned error can't find the container with id 271d41bb022ec3d065da12ec895bf8d9cc8a2dce5bb843665d6bbf2fd5081239
Sep 09 19:56:01.903540 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:56:01.903488    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5e04e1e_56ef_4e29_89bd_eafcf7425d56.slice/crio-18ca3266ec71f6a45d2e6aef4dde7b7e1a178b1795a17ad0cbba1507cfcbc144 WatchSource:0}: Error finding container 18ca3266ec71f6a45d2e6aef4dde7b7e1a178b1795a17ad0cbba1507cfcbc144: Status 404 returned error can't find the container with id 18ca3266ec71f6a45d2e6aef4dde7b7e1a178b1795a17ad0cbba1507cfcbc144
Sep 09 19:56:02.073923 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:02.073887    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jcx8l" event={"ID":"d5e04e1e-56ef-4e29-89bd-eafcf7425d56","Type":"ContainerStarted","Data":"d3decd2034c4898f99c268821a440eb6c27a03585b06ae15fa00907013f950c9"}
Sep 09 19:56:02.074057 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:02.074038    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jcx8l"
Sep 09 19:56:02.074123 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:02.074085    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jcx8l" event={"ID":"d5e04e1e-56ef-4e29-89bd-eafcf7425d56","Type":"ContainerStarted","Data":"18ca3266ec71f6a45d2e6aef4dde7b7e1a178b1795a17ad0cbba1507cfcbc144"}
Sep 09 19:56:02.075011 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:02.074989    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gn8cc" event={"ID":"b2cc1cab-a99d-4733-8623-2ccf3fb89961","Type":"ContainerStarted","Data":"271d41bb022ec3d065da12ec895bf8d9cc8a2dce5bb843665d6bbf2fd5081239"}
Sep 09 19:56:02.097778 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:02.097738    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jcx8l" podStartSLOduration=67.097726611 podStartE2EDuration="1m7.097726611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:54:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:02.097193358 +0000 UTC m=+66.769212355" watchObservedRunningTime="2026-09-09 19:56:02.097726611 +0000 UTC m=+66.769745608"
Sep 09 19:56:03.018172 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:03.018115    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/27f5043a-c597-4d58-9a73-c0b1cd9071b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z7hmm\" (UID: \"27f5043a-c597-4d58-9a73-c0b1cd9071b3\") " pod="openshift-insights/insights-runtime-extractor-z7hmm"
Sep 09 19:56:03.018603 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:03.018226    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/27f5043a-c597-4d58-9a73-c0b1cd9071b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z7hmm\" (UID: \"27f5043a-c597-4d58-9a73-c0b1cd9071b3\") " pod="openshift-insights/insights-runtime-extractor-z7hmm"
Sep 09 19:56:03.018603 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:56:03.018234    2523 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 19:56:03.018603 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:56:03.018354    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/27f5043a-c597-4d58-9a73-c0b1cd9071b3-kube-rbac-proxy-cm podName:27f5043a-c597-4d58-9a73-c0b1cd9071b3 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:11.018332883 +0000 UTC m=+75.690351864 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/27f5043a-c597-4d58-9a73-c0b1cd9071b3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-z7hmm" (UID: "27f5043a-c597-4d58-9a73-c0b1cd9071b3") : configmap "kube-rbac-proxy" not found
Sep 09 19:56:03.018603 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:56:03.018353    2523 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 19:56:03.018603 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:56:03.018421    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27f5043a-c597-4d58-9a73-c0b1cd9071b3-insights-runtime-extractor-tls podName:27f5043a-c597-4d58-9a73-c0b1cd9071b3 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:11.018402498 +0000 UTC m=+75.690421488 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/27f5043a-c597-4d58-9a73-c0b1cd9071b3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-z7hmm" (UID: "27f5043a-c597-4d58-9a73-c0b1cd9071b3") : secret "insights-runtime-extractor-tls" not found
Sep 09 19:56:05.539231 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:05.539197    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f26e26cf-30ac-4401-af29-627376841db9-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-b9f9c\" (UID: \"f26e26cf-30ac-4401-af29-627376841db9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-b9f9c"
Sep 09 19:56:05.541848 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:05.541820    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f26e26cf-30ac-4401-af29-627376841db9-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-b9f9c\" (UID: \"f26e26cf-30ac-4401-af29-627376841db9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-b9f9c"
Sep 09 19:56:05.640257 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:05.640214    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/72f24ca2-96db-441c-a051-9fbf6f361baa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-8zpvv\" (UID: \"72f24ca2-96db-441c-a051-9fbf6f361baa\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8zpvv"
Sep 09 19:56:05.640416 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:56:05.640359    2523 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 19:56:05.640482 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:56:05.640436    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72f24ca2-96db-441c-a051-9fbf6f361baa-cluster-monitoring-operator-tls podName:72f24ca2-96db-441c-a051-9fbf6f361baa nodeName:}" failed. No retries permitted until 2026-09-09 19:56:21.640419328 +0000 UTC m=+86.312438307 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/72f24ca2-96db-441c-a051-9fbf6f361baa-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-8zpvv" (UID: "72f24ca2-96db-441c-a051-9fbf6f361baa") : secret "cluster-monitoring-operator-tls" not found
Sep 09 19:56:05.640735 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:05.640593    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-jwnsp\""
Sep 09 19:56:05.648251 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:05.648233    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-b9f9c"
Sep 09 19:56:06.353368 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:06.353294    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-b9f9c"]
Sep 09 19:56:07.087932 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:07.087893    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gn8cc" event={"ID":"b2cc1cab-a99d-4733-8623-2ccf3fb89961","Type":"ContainerStarted","Data":"6e06e9f243fa124f682568d1ee801951e26450bcc24fef9d678bbb46cbd49a43"}
Sep 09 19:56:07.089092 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:07.089056    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-b9f9c" event={"ID":"f26e26cf-30ac-4401-af29-627376841db9","Type":"ContainerStarted","Data":"d0f4ded801ccb98dfc9e96f5630eadf969e49af315a855f7de4719153025daf0"}
Sep 09 19:56:07.106506 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:07.106462    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-gn8cc" podStartSLOduration=66.559520895 podStartE2EDuration="1m11.106448209s" totalImagesPullingTime="4.546927314s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:54:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:07.106193643 +0000 UTC m=+71.778212640" watchObservedRunningTime="2026-09-09 19:56:07.106448209 +0000 UTC m=+71.778467267"
Sep 09 19:56:09.097518 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:09.097473    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-b9f9c" event={"ID":"f26e26cf-30ac-4401-af29-627376841db9","Type":"ContainerStarted","Data":"d8130c4fe8998234fba3324866cbd58579b65899760eb24a33630b2b96ca798c"}
Sep 09 19:56:09.097881 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:09.097538    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-b9f9c" event={"ID":"f26e26cf-30ac-4401-af29-627376841db9","Type":"ContainerStarted","Data":"a9f4c7549ca55338a9b2c2114b93324d176ae805267c3a4e289d6dbe9e11239c"}
Sep 09 19:56:09.114005 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:09.113966    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-b9f9c" podStartSLOduration=18.213351659 podStartE2EDuration="20.113951157s" totalImagesPullingTime="1.900599498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:55:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:09.113137139 +0000 UTC m=+73.785156151" watchObservedRunningTime="2026-09-09 19:56:09.113951157 +0000 UTC m=+73.785970150"
Sep 09 19:56:11.083912 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:11.083876    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/27f5043a-c597-4d58-9a73-c0b1cd9071b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z7hmm\" (UID: \"27f5043a-c597-4d58-9a73-c0b1cd9071b3\") " pod="openshift-insights/insights-runtime-extractor-z7hmm"
Sep 09 19:56:11.084314 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:11.083954    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/27f5043a-c597-4d58-9a73-c0b1cd9071b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z7hmm\" (UID: \"27f5043a-c597-4d58-9a73-c0b1cd9071b3\") " pod="openshift-insights/insights-runtime-extractor-z7hmm"
Sep 09 19:56:11.084314 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:56:11.084006    2523 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 19:56:11.084314 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:56:11.084064    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27f5043a-c597-4d58-9a73-c0b1cd9071b3-insights-runtime-extractor-tls podName:27f5043a-c597-4d58-9a73-c0b1cd9071b3 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:27.084050405 +0000 UTC m=+91.756069379 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/27f5043a-c597-4d58-9a73-c0b1cd9071b3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-z7hmm" (UID: "27f5043a-c597-4d58-9a73-c0b1cd9071b3") : secret "insights-runtime-extractor-tls" not found
Sep 09 19:56:11.084439 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:11.084385    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/27f5043a-c597-4d58-9a73-c0b1cd9071b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z7hmm\" (UID: \"27f5043a-c597-4d58-9a73-c0b1cd9071b3\") " pod="openshift-insights/insights-runtime-extractor-z7hmm"
Sep 09 19:56:21.655278 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:21.655248    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/72f24ca2-96db-441c-a051-9fbf6f361baa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-8zpvv\" (UID: \"72f24ca2-96db-441c-a051-9fbf6f361baa\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8zpvv"
Sep 09 19:56:21.657658 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:21.657639    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/72f24ca2-96db-441c-a051-9fbf6f361baa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-8zpvv\" (UID: \"72f24ca2-96db-441c-a051-9fbf6f361baa\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8zpvv"
Sep 09 19:56:21.941513 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:21.941456    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-nkkmp\""
Sep 09 19:56:21.949092 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:21.949069    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8zpvv"
Sep 09 19:56:22.237639 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:56:22.237615    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72f24ca2_96db_441c_a051_9fbf6f361baa.slice/crio-e10973d51e47f441be5ba49bcb0fae8c3593ac6767d1e270868a9b01d8f0b5fc WatchSource:0}: Error finding container e10973d51e47f441be5ba49bcb0fae8c3593ac6767d1e270868a9b01d8f0b5fc: Status 404 returned error can't find the container with id e10973d51e47f441be5ba49bcb0fae8c3593ac6767d1e270868a9b01d8f0b5fc
Sep 09 19:56:22.245453 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:22.243543    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8zpvv"]
Sep 09 19:56:23.142439 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:23.142406    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8zpvv" event={"ID":"72f24ca2-96db-441c-a051-9fbf6f361baa","Type":"ContainerStarted","Data":"e10973d51e47f441be5ba49bcb0fae8c3593ac6767d1e270868a9b01d8f0b5fc"}
Sep 09 19:56:23.655551 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:23.655525    2523 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-qx2rd"]
Sep 09 19:56:23.660657 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:23.660642    2523 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8595546d49-z6rhf"]
Sep 09 19:56:23.660783 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:23.660766    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-qx2rd"
Sep 09 19:56:23.664636 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:23.664618    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8595546d49-z6rhf"
Sep 09 19:56:23.664821 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:23.664799    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 09 19:56:23.664965 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:23.664826    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 09 19:56:23.665053 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:23.664980    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-m4zjr\""
Sep 09 19:56:23.668564 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:23.668541    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 09 19:56:23.668659 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:23.668597    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 09 19:56:23.668714 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:23.668606    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 09 19:56:23.668763 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:23.668737    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 09 19:56:23.668814 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:23.668761    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 09 19:56:23.668814 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:23.668767    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 09 19:56:23.668903 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:23.668851    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 09 19:56:23.677877 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:23.677855    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-qx2rd"]
Sep 09 19:56:23.679413 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:23.679397    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8595546d49-z6rhf"]
Sep 09 19:56:23.769690 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:23.769664    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/041dfad6-0f95-4e92-9cb3-68bc5df83ff4-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-8595546d49-z6rhf\" (UID: \"041dfad6-0f95-4e92-9cb3-68bc5df83ff4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8595546d49-z6rhf"
Sep 09 19:56:23.769690 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:23.769691    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b6011a45-1105-4276-85a6-2145931807ba-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-qx2rd\" (UID: \"b6011a45-1105-4276-85a6-2145931807ba\") " pod="openshift-network-console/networking-console-plugin-79977f9844-qx2rd"
Sep 09 19:56:23.769854 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:23.769712    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b6011a45-1105-4276-85a6-2145931807ba-nginx-conf\") pod \"networking-console-plugin-79977f9844-qx2rd\" (UID: \"b6011a45-1105-4276-85a6-2145931807ba\") " pod="openshift-network-console/networking-console-plugin-79977f9844-qx2rd"
Sep 09 19:56:23.769854 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:23.769757    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/041dfad6-0f95-4e92-9cb3-68bc5df83ff4-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-8595546d49-z6rhf\" (UID: \"041dfad6-0f95-4e92-9cb3-68bc5df83ff4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8595546d49-z6rhf"
Sep 09 19:56:23.769854 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:23.769831    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h55qd\" (UniqueName: \"kubernetes.io/projected/041dfad6-0f95-4e92-9cb3-68bc5df83ff4-kube-api-access-h55qd\") pod \"cluster-proxy-proxy-agent-8595546d49-z6rhf\" (UID: \"041dfad6-0f95-4e92-9cb3-68bc5df83ff4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8595546d49-z6rhf"
Sep 09 19:56:23.769957 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:23.769864    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/041dfad6-0f95-4e92-9cb3-68bc5df83ff4-additional-service-ca\") pod \"cluster-proxy-proxy-agent-8595546d49-z6rhf\" (UID: \"041dfad6-0f95-4e92-9cb3-68bc5df83ff4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8595546d49-z6rhf"
Sep 09 19:56:23.769957 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:23.769891    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/041dfad6-0f95-4e92-9cb3-68bc5df83ff4-hub\") pod \"cluster-proxy-proxy-agent-8595546d49-z6rhf\" (UID: \"041dfad6-0f95-4e92-9cb3-68bc5df83ff4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8595546d49-z6rhf"
Sep 09 19:56:23.769957 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:23.769938    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/041dfad6-0f95-4e92-9cb3-68bc5df83ff4-ca\") pod \"cluster-proxy-proxy-agent-8595546d49-z6rhf\" (UID: \"041dfad6-0f95-4e92-9cb3-68bc5df83ff4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8595546d49-z6rhf"
Sep 09 19:56:23.870434 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:23.870403    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h55qd\" (UniqueName: \"kubernetes.io/projected/041dfad6-0f95-4e92-9cb3-68bc5df83ff4-kube-api-access-h55qd\") pod \"cluster-proxy-proxy-agent-8595546d49-z6rhf\" (UID: \"041dfad6-0f95-4e92-9cb3-68bc5df83ff4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8595546d49-z6rhf"
Sep 09 19:56:23.870557 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:23.870441    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/041dfad6-0f95-4e92-9cb3-68bc5df83ff4-additional-service-ca\") pod \"cluster-proxy-proxy-agent-8595546d49-z6rhf\" (UID: \"041dfad6-0f95-4e92-9cb3-68bc5df83ff4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8595546d49-z6rhf"
Sep 09 19:56:23.870557 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:23.870469    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/041dfad6-0f95-4e92-9cb3-68bc5df83ff4-hub\") pod \"cluster-proxy-proxy-agent-8595546d49-z6rhf\" (UID: \"041dfad6-0f95-4e92-9cb3-68bc5df83ff4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8595546d49-z6rhf"
Sep 09 19:56:23.870557 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:23.870508    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/041dfad6-0f95-4e92-9cb3-68bc5df83ff4-ca\") pod \"cluster-proxy-proxy-agent-8595546d49-z6rhf\" (UID: \"041dfad6-0f95-4e92-9cb3-68bc5df83ff4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8595546d49-z6rhf"
Sep 09 19:56:23.870557 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:23.870551    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/041dfad6-0f95-4e92-9cb3-68bc5df83ff4-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-8595546d49-z6rhf\" (UID: \"041dfad6-0f95-4e92-9cb3-68bc5df83ff4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8595546d49-z6rhf"
Sep 09 19:56:23.870763 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:23.870575    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b6011a45-1105-4276-85a6-2145931807ba-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-qx2rd\" (UID: \"b6011a45-1105-4276-85a6-2145931807ba\") " pod="openshift-network-console/networking-console-plugin-79977f9844-qx2rd"
Sep 09 19:56:23.870763 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:23.870605    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b6011a45-1105-4276-85a6-2145931807ba-nginx-conf\") pod \"networking-console-plugin-79977f9844-qx2rd\" (UID: \"b6011a45-1105-4276-85a6-2145931807ba\") " pod="openshift-network-console/networking-console-plugin-79977f9844-qx2rd"
Sep 09 19:56:23.870860 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:23.870802    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/041dfad6-0f95-4e92-9cb3-68bc5df83ff4-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-8595546d49-z6rhf\" (UID: \"041dfad6-0f95-4e92-9cb3-68bc5df83ff4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8595546d49-z6rhf"
Sep 09 19:56:23.871307 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:23.871280    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/041dfad6-0f95-4e92-9cb3-68bc5df83ff4-additional-service-ca\") pod \"cluster-proxy-proxy-agent-8595546d49-z6rhf\" (UID: \"041dfad6-0f95-4e92-9cb3-68bc5df83ff4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8595546d49-z6rhf"
Sep 09 19:56:23.871433 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:23.871341    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b6011a45-1105-4276-85a6-2145931807ba-nginx-conf\") pod \"networking-console-plugin-79977f9844-qx2rd\" (UID: \"b6011a45-1105-4276-85a6-2145931807ba\") " pod="openshift-network-console/networking-console-plugin-79977f9844-qx2rd"
Sep 09 19:56:23.873385 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:23.873354    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/041dfad6-0f95-4e92-9cb3-68bc5df83ff4-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-8595546d49-z6rhf\" (UID: \"041dfad6-0f95-4e92-9cb3-68bc5df83ff4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8595546d49-z6rhf"
Sep 09 19:56:23.873691 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:23.873669    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b6011a45-1105-4276-85a6-2145931807ba-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-qx2rd\" (UID: \"b6011a45-1105-4276-85a6-2145931807ba\") " pod="openshift-network-console/networking-console-plugin-79977f9844-qx2rd"
Sep 09 19:56:23.873778 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:23.873671    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/041dfad6-0f95-4e92-9cb3-68bc5df83ff4-ca\") pod \"cluster-proxy-proxy-agent-8595546d49-z6rhf\" (UID: \"041dfad6-0f95-4e92-9cb3-68bc5df83ff4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8595546d49-z6rhf"
Sep 09 19:56:23.873839 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:23.873788    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/041dfad6-0f95-4e92-9cb3-68bc5df83ff4-hub\") pod \"cluster-proxy-proxy-agent-8595546d49-z6rhf\" (UID: \"041dfad6-0f95-4e92-9cb3-68bc5df83ff4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8595546d49-z6rhf"
Sep 09 19:56:23.873891 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:23.873857    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/041dfad6-0f95-4e92-9cb3-68bc5df83ff4-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-8595546d49-z6rhf\" (UID: \"041dfad6-0f95-4e92-9cb3-68bc5df83ff4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8595546d49-z6rhf"
Sep 09 19:56:23.890739 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:23.890719    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h55qd\" (UniqueName: \"kubernetes.io/projected/041dfad6-0f95-4e92-9cb3-68bc5df83ff4-kube-api-access-h55qd\") pod \"cluster-proxy-proxy-agent-8595546d49-z6rhf\" (UID: \"041dfad6-0f95-4e92-9cb3-68bc5df83ff4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8595546d49-z6rhf"
Sep 09 19:56:23.971133 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:23.971078    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-qx2rd"
Sep 09 19:56:23.983091 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:23.983072    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8595546d49-z6rhf"
Sep 09 19:56:24.252728 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:24.252705    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-qx2rd"]
Sep 09 19:56:24.255007 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:56:24.254979    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6011a45_1105_4276_85a6_2145931807ba.slice/crio-5841e4b58c69d320f00aa868f9d33e35e09fd8d833d780e333d399ec090b5570 WatchSource:0}: Error finding container 5841e4b58c69d320f00aa868f9d33e35e09fd8d833d780e333d399ec090b5570: Status 404 returned error can't find the container with id 5841e4b58c69d320f00aa868f9d33e35e09fd8d833d780e333d399ec090b5570
Sep 09 19:56:24.272973 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:24.272953    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8595546d49-z6rhf"]
Sep 09 19:56:24.275654 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:56:24.275625    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod041dfad6_0f95_4e92_9cb3_68bc5df83ff4.slice/crio-c5ed0980d681481b0629587d547daf17efc6427e1a80185a1e35d27f79ce9833 WatchSource:0}: Error finding container c5ed0980d681481b0629587d547daf17efc6427e1a80185a1e35d27f79ce9833: Status 404 returned error can't find the container with id c5ed0980d681481b0629587d547daf17efc6427e1a80185a1e35d27f79ce9833
Sep 09 19:56:25.152850 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:25.152806    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-qx2rd" event={"ID":"b6011a45-1105-4276-85a6-2145931807ba","Type":"ContainerStarted","Data":"5841e4b58c69d320f00aa868f9d33e35e09fd8d833d780e333d399ec090b5570"}
Sep 09 19:56:25.155383 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:25.155351    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8595546d49-z6rhf" event={"ID":"041dfad6-0f95-4e92-9cb3-68bc5df83ff4","Type":"ContainerStarted","Data":"c5ed0980d681481b0629587d547daf17efc6427e1a80185a1e35d27f79ce9833"}
Sep 09 19:56:27.098050 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:27.098021    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/27f5043a-c597-4d58-9a73-c0b1cd9071b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z7hmm\" (UID: \"27f5043a-c597-4d58-9a73-c0b1cd9071b3\") " pod="openshift-insights/insights-runtime-extractor-z7hmm"
Sep 09 19:56:27.100776 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:27.100750    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/27f5043a-c597-4d58-9a73-c0b1cd9071b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z7hmm\" (UID: \"27f5043a-c597-4d58-9a73-c0b1cd9071b3\") " pod="openshift-insights/insights-runtime-extractor-z7hmm"
Sep 09 19:56:27.372967 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:27.372883    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-57xxp\""
Sep 09 19:56:27.381034 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:27.381008    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-z7hmm"
Sep 09 19:56:28.529892 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:28.529764    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-z7hmm"]
Sep 09 19:56:28.533626 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:56:28.532714    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f5043a_c597_4d58_9a73_c0b1cd9071b3.slice/crio-fd48be79aa8385fb2022b869a89c2784fffb9a1e769ba1edbf310177b409e09f WatchSource:0}: Error finding container fd48be79aa8385fb2022b869a89c2784fffb9a1e769ba1edbf310177b409e09f: Status 404 returned error can't find the container with id fd48be79aa8385fb2022b869a89c2784fffb9a1e769ba1edbf310177b409e09f
Sep 09 19:56:29.173126 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:29.173091    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8595546d49-z6rhf" event={"ID":"041dfad6-0f95-4e92-9cb3-68bc5df83ff4","Type":"ContainerStarted","Data":"7115083b3fb45414573ca9760540a4863132582b40023b1dd8fa5e9da9390c6f"}
Sep 09 19:56:29.173331 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:29.173139    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8595546d49-z6rhf" event={"ID":"041dfad6-0f95-4e92-9cb3-68bc5df83ff4","Type":"ContainerStarted","Data":"c0e14cf6250d149649760e4c7e870a9eaca9f37615eb9103628c4a135d8c9238"}
Sep 09 19:56:29.173331 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:29.173185    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8595546d49-z6rhf" event={"ID":"041dfad6-0f95-4e92-9cb3-68bc5df83ff4","Type":"ContainerStarted","Data":"b7d1c2a0820be0550c82a00095eb5175d6ef8d477faf3bd03c08bb478dbb120a"}
Sep 09 19:56:29.174750 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:29.174721    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z7hmm" event={"ID":"27f5043a-c597-4d58-9a73-c0b1cd9071b3","Type":"ContainerStarted","Data":"83437dda2d3d7491ee55ab890f88147817bfdf5359d92d17c83444dd0bff2c15"}
Sep 09 19:56:29.174878 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:29.174783    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z7hmm" event={"ID":"27f5043a-c597-4d58-9a73-c0b1cd9071b3","Type":"ContainerStarted","Data":"fd48be79aa8385fb2022b869a89c2784fffb9a1e769ba1edbf310177b409e09f"}
Sep 09 19:56:29.176341 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:29.176309    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8zpvv" event={"ID":"72f24ca2-96db-441c-a051-9fbf6f361baa","Type":"ContainerStarted","Data":"a076d267af964fc133bfc74a436e479a6c78b1edcbb9dd1de4a362d3600a01fb"}
Sep 09 19:56:29.177857 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:29.177836    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-qx2rd" event={"ID":"b6011a45-1105-4276-85a6-2145931807ba","Type":"ContainerStarted","Data":"56b577caa600ed3387e1800956454d8ca4180229bdb2f11856d414ae86d63261"}
Sep 09 19:56:29.192630 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:29.192589    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8595546d49-z6rhf" podStartSLOduration=2.191137044 podStartE2EDuration="6.192577653s" totalImagesPullingTime="4.001440609s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:29.191955586 +0000 UTC m=+93.863974583" watchObservedRunningTime="2026-09-09 19:56:29.192577653 +0000 UTC m=+93.864596643"
Sep 09 19:56:29.207509 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:29.207468    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-qx2rd" podStartSLOduration=2.205707845 podStartE2EDuration="6.207456792s" totalImagesPullingTime="4.001748947s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:29.207024526 +0000 UTC m=+93.879043523" watchObservedRunningTime="2026-09-09 19:56:29.207456792 +0000 UTC m=+93.879475789"
Sep 09 19:56:29.225694 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:29.225655    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-8zpvv" podStartSLOduration=34.213412266 podStartE2EDuration="40.225643702s" totalImagesPullingTime="6.012231436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:55:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:29.224922556 +0000 UTC m=+93.896941553" watchObservedRunningTime="2026-09-09 19:56:29.225643702 +0000 UTC m=+93.897662699"
Sep 09 19:56:30.183026 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:30.182966    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z7hmm" event={"ID":"27f5043a-c597-4d58-9a73-c0b1cd9071b3","Type":"ContainerStarted","Data":"21b5524f0a503eca4e002c44e8eb049e5994a5ca2dddaf32e408577574ea2159"}
Sep 09 19:56:32.190086 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:32.190054    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z7hmm" event={"ID":"27f5043a-c597-4d58-9a73-c0b1cd9071b3","Type":"ContainerStarted","Data":"bc89b16d8b2c5fbf7163027413397ecb9677194fe2754b755ca403cc323f7336"}
Sep 09 19:56:32.209764 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:32.209718    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-z7hmm" podStartSLOduration=34.695163887 podStartE2EDuration="37.209706178s" totalImagesPullingTime="2.514542291s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:32.209512337 +0000 UTC m=+96.881531331" watchObservedRunningTime="2026-09-09 19:56:32.209706178 +0000 UTC m=+96.881725171"
Sep 09 19:56:33.079980 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:33.079951    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jcx8l"
Sep 09 19:56:33.645096 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:33.645064    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fb48971c-63be-4de1-b19e-0b8c1aadbc58-registry-tls\") pod \"image-registry-bcc64b9dc-p7sz5\" (UID: \"fb48971c-63be-4de1-b19e-0b8c1aadbc58\") " pod="openshift-image-registry/image-registry-bcc64b9dc-p7sz5"
Sep 09 19:56:33.645512 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:33.645107    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d3bbfe8-34df-41e1-b959-766ff5100634-metrics-tls\") pod \"dns-default-zr55l\" (UID: \"1d3bbfe8-34df-41e1-b959-766ff5100634\") " pod="openshift-dns/dns-default-zr55l"
Sep 09 19:56:33.647438 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:33.647415    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d3bbfe8-34df-41e1-b959-766ff5100634-metrics-tls\") pod \"dns-default-zr55l\" (UID: \"1d3bbfe8-34df-41e1-b959-766ff5100634\") " pod="openshift-dns/dns-default-zr55l"
Sep 09 19:56:33.647530 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:33.647504    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fb48971c-63be-4de1-b19e-0b8c1aadbc58-registry-tls\") pod \"image-registry-bcc64b9dc-p7sz5\" (UID: \"fb48971c-63be-4de1-b19e-0b8c1aadbc58\") " pod="openshift-image-registry/image-registry-bcc64b9dc-p7sz5"
Sep 09 19:56:33.936983 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:33.936926    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-j6mws\""
Sep 09 19:56:33.943844 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:33.943826    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bcc64b9dc-p7sz5"
Sep 09 19:56:33.948733 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:33.948713    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-shzs9\""
Sep 09 19:56:33.957129 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:33.957111    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zr55l"
Sep 09 19:56:34.220256 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:34.220126    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-bcc64b9dc-p7sz5"]
Sep 09 19:56:34.223041 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:56:34.223019    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb48971c_63be_4de1_b19e_0b8c1aadbc58.slice/crio-9eb9476f3a1bf7396abf81e6eefd16296a3455a946fec07a146cd2f4c005eac8 WatchSource:0}: Error finding container 9eb9476f3a1bf7396abf81e6eefd16296a3455a946fec07a146cd2f4c005eac8: Status 404 returned error can't find the container with id 9eb9476f3a1bf7396abf81e6eefd16296a3455a946fec07a146cd2f4c005eac8
Sep 09 19:56:34.238752 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:34.238734    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zr55l"]
Sep 09 19:56:34.241588 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:56:34.241569    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d3bbfe8_34df_41e1_b959_766ff5100634.slice/crio-9797bbf405aa7f13879712154325f82ff7cab45cfe7022c126142284b66a0b7e WatchSource:0}: Error finding container 9797bbf405aa7f13879712154325f82ff7cab45cfe7022c126142284b66a0b7e: Status 404 returned error can't find the container with id 9797bbf405aa7f13879712154325f82ff7cab45cfe7022c126142284b66a0b7e
Sep 09 19:56:35.201630 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:35.201596    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zr55l" event={"ID":"1d3bbfe8-34df-41e1-b959-766ff5100634","Type":"ContainerStarted","Data":"9797bbf405aa7f13879712154325f82ff7cab45cfe7022c126142284b66a0b7e"}
Sep 09 19:56:35.203462 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:35.203369    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bcc64b9dc-p7sz5" event={"ID":"fb48971c-63be-4de1-b19e-0b8c1aadbc58","Type":"ContainerStarted","Data":"f03fa6b561dfc291595e7606b966810c2d66fa218b7c623fd99b3f2f4de96fa5"}
Sep 09 19:56:35.203601 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:35.203533    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bcc64b9dc-p7sz5" event={"ID":"fb48971c-63be-4de1-b19e-0b8c1aadbc58","Type":"ContainerStarted","Data":"9eb9476f3a1bf7396abf81e6eefd16296a3455a946fec07a146cd2f4c005eac8"}
Sep 09 19:56:35.203601 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:35.203558    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-bcc64b9dc-p7sz5"
Sep 09 19:56:37.211949 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:37.211909    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zr55l" event={"ID":"1d3bbfe8-34df-41e1-b959-766ff5100634","Type":"ContainerStarted","Data":"a8f34f52a1b15225a05188a2178903ed95398bc1ee5da26c9111c08d34cd8185"}
Sep 09 19:56:37.212411 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:37.211961    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zr55l" event={"ID":"1d3bbfe8-34df-41e1-b959-766ff5100634","Type":"ContainerStarted","Data":"199f9212a28d09877381948d7047ec9fefe78580ae74496b53595cab8a6a26af"}
Sep 09 19:56:37.212411 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:37.212047    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zr55l"
Sep 09 19:56:37.229463 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:37.229423    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-bcc64b9dc-p7sz5" podStartSLOduration=101.229411401 podStartE2EDuration="1m41.229411401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:54:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:35.227402435 +0000 UTC m=+99.899421433" watchObservedRunningTime="2026-09-09 19:56:37.229411401 +0000 UTC m=+101.901430397"
Sep 09 19:56:37.229584 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:37.229517    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-zr55l" podStartSLOduration=66.07710187 podStartE2EDuration="1m8.229512776s" totalImagesPullingTime="2.152410906s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:55:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:37.228273602 +0000 UTC m=+101.900292612" watchObservedRunningTime="2026-09-09 19:56:37.229512776 +0000 UTC m=+101.901531775"
Sep 09 19:56:37.384939 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:37.384910    2523 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-lhjfm"]
Sep 09 19:56:37.388064 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:37.388050    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lhjfm"
Sep 09 19:56:37.390586 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:37.390565    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 19:56:37.390699 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:37.390652    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 19:56:37.392168 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:37.392139    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 19:56:37.392323 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:37.392307    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-nplxr\""
Sep 09 19:56:37.392492 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:37.392478    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 19:56:37.475051 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:37.474990    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3cce0c6c-933f-4559-89ae-6af9e6373cc9-node-exporter-accelerators-collector-config\") pod \"node-exporter-lhjfm\" (UID: \"3cce0c6c-933f-4559-89ae-6af9e6373cc9\") " pod="openshift-monitoring/node-exporter-lhjfm"
Sep 09 19:56:37.475051 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:37.475020    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3cce0c6c-933f-4559-89ae-6af9e6373cc9-node-exporter-tls\") pod \"node-exporter-lhjfm\" (UID: \"3cce0c6c-933f-4559-89ae-6af9e6373cc9\") " pod="openshift-monitoring/node-exporter-lhjfm"
Sep 09 19:56:37.475051 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:37.475040    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3cce0c6c-933f-4559-89ae-6af9e6373cc9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lhjfm\" (UID: \"3cce0c6c-933f-4559-89ae-6af9e6373cc9\") " pod="openshift-monitoring/node-exporter-lhjfm"
Sep 09 19:56:37.475286 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:37.475069    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm6m2\" (UniqueName: \"kubernetes.io/projected/3cce0c6c-933f-4559-89ae-6af9e6373cc9-kube-api-access-cm6m2\") pod \"node-exporter-lhjfm\" (UID: \"3cce0c6c-933f-4559-89ae-6af9e6373cc9\") " pod="openshift-monitoring/node-exporter-lhjfm"
Sep 09 19:56:37.475286 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:37.475117    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3cce0c6c-933f-4559-89ae-6af9e6373cc9-root\") pod \"node-exporter-lhjfm\" (UID: \"3cce0c6c-933f-4559-89ae-6af9e6373cc9\") " pod="openshift-monitoring/node-exporter-lhjfm"
Sep 09 19:56:37.475286 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:37.475137    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3cce0c6c-933f-4559-89ae-6af9e6373cc9-node-exporter-wtmp\") pod \"node-exporter-lhjfm\" (UID: \"3cce0c6c-933f-4559-89ae-6af9e6373cc9\") " pod="openshift-monitoring/node-exporter-lhjfm"
Sep 09 19:56:37.475286 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:37.475201    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3cce0c6c-933f-4559-89ae-6af9e6373cc9-sys\") pod \"node-exporter-lhjfm\" (UID: \"3cce0c6c-933f-4559-89ae-6af9e6373cc9\") " pod="openshift-monitoring/node-exporter-lhjfm"
Sep 09 19:56:37.475286 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:37.475219    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3cce0c6c-933f-4559-89ae-6af9e6373cc9-node-exporter-textfile\") pod \"node-exporter-lhjfm\" (UID: \"3cce0c6c-933f-4559-89ae-6af9e6373cc9\") " pod="openshift-monitoring/node-exporter-lhjfm"
Sep 09 19:56:37.475286 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:37.475274    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3cce0c6c-933f-4559-89ae-6af9e6373cc9-metrics-client-ca\") pod \"node-exporter-lhjfm\" (UID: \"3cce0c6c-933f-4559-89ae-6af9e6373cc9\") " pod="openshift-monitoring/node-exporter-lhjfm"
Sep 09 19:56:37.576539 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:37.576511    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm6m2\" (UniqueName: \"kubernetes.io/projected/3cce0c6c-933f-4559-89ae-6af9e6373cc9-kube-api-access-cm6m2\") pod \"node-exporter-lhjfm\" (UID: \"3cce0c6c-933f-4559-89ae-6af9e6373cc9\") " pod="openshift-monitoring/node-exporter-lhjfm"
Sep 09 19:56:37.576754 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:37.576731    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3cce0c6c-933f-4559-89ae-6af9e6373cc9-root\") pod \"node-exporter-lhjfm\" (UID: \"3cce0c6c-933f-4559-89ae-6af9e6373cc9\") " pod="openshift-monitoring/node-exporter-lhjfm"
Sep 09 19:56:37.576883 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:37.576765    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3cce0c6c-933f-4559-89ae-6af9e6373cc9-node-exporter-wtmp\") pod \"node-exporter-lhjfm\" (UID: \"3cce0c6c-933f-4559-89ae-6af9e6373cc9\") " pod="openshift-monitoring/node-exporter-lhjfm"
Sep 09 19:56:37.576883 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:37.576809    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3cce0c6c-933f-4559-89ae-6af9e6373cc9-sys\") pod \"node-exporter-lhjfm\" (UID: \"3cce0c6c-933f-4559-89ae-6af9e6373cc9\") " pod="openshift-monitoring/node-exporter-lhjfm"
Sep 09 19:56:37.576883 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:37.576833    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3cce0c6c-933f-4559-89ae-6af9e6373cc9-node-exporter-textfile\") pod \"node-exporter-lhjfm\" (UID: \"3cce0c6c-933f-4559-89ae-6af9e6373cc9\") " pod="openshift-monitoring/node-exporter-lhjfm"
Sep 09 19:56:37.576883 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:37.576860    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3cce0c6c-933f-4559-89ae-6af9e6373cc9-metrics-client-ca\") pod \"node-exporter-lhjfm\" (UID: \"3cce0c6c-933f-4559-89ae-6af9e6373cc9\") " pod="openshift-monitoring/node-exporter-lhjfm"
Sep 09 19:56:37.577109 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:37.576900    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3cce0c6c-933f-4559-89ae-6af9e6373cc9-node-exporter-accelerators-collector-config\") pod \"node-exporter-lhjfm\" (UID: \"3cce0c6c-933f-4559-89ae-6af9e6373cc9\") " pod="openshift-monitoring/node-exporter-lhjfm"
Sep 09 19:56:37.577109 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:37.576934    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3cce0c6c-933f-4559-89ae-6af9e6373cc9-node-exporter-tls\") pod \"node-exporter-lhjfm\" (UID: \"3cce0c6c-933f-4559-89ae-6af9e6373cc9\") " pod="openshift-monitoring/node-exporter-lhjfm"
Sep 09 19:56:37.577109 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:37.576960    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3cce0c6c-933f-4559-89ae-6af9e6373cc9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lhjfm\" (UID: \"3cce0c6c-933f-4559-89ae-6af9e6373cc9\") " pod="openshift-monitoring/node-exporter-lhjfm"
Sep 09 19:56:37.577109 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:37.576962    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3cce0c6c-933f-4559-89ae-6af9e6373cc9-node-exporter-wtmp\") pod \"node-exporter-lhjfm\" (UID: \"3cce0c6c-933f-4559-89ae-6af9e6373cc9\") " pod="openshift-monitoring/node-exporter-lhjfm"
Sep 09 19:56:37.577109 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:37.577026    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3cce0c6c-933f-4559-89ae-6af9e6373cc9-root\") pod \"node-exporter-lhjfm\" (UID: \"3cce0c6c-933f-4559-89ae-6af9e6373cc9\") " pod="openshift-monitoring/node-exporter-lhjfm"
Sep 09 19:56:37.577425 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:37.577381    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3cce0c6c-933f-4559-89ae-6af9e6373cc9-sys\") pod \"node-exporter-lhjfm\" (UID: \"3cce0c6c-933f-4559-89ae-6af9e6373cc9\") " pod="openshift-monitoring/node-exporter-lhjfm"
Sep 09 19:56:37.577425 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:56:37.577415    2523 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 09 19:56:37.577530 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:56:37.577469    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cce0c6c-933f-4559-89ae-6af9e6373cc9-node-exporter-tls podName:3cce0c6c-933f-4559-89ae-6af9e6373cc9 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:38.077452844 +0000 UTC m=+102.749471834 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/3cce0c6c-933f-4559-89ae-6af9e6373cc9-node-exporter-tls") pod "node-exporter-lhjfm" (UID: "3cce0c6c-933f-4559-89ae-6af9e6373cc9") : secret "node-exporter-tls" not found
Sep 09 19:56:37.577530 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:37.577510    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3cce0c6c-933f-4559-89ae-6af9e6373cc9-metrics-client-ca\") pod \"node-exporter-lhjfm\" (UID: \"3cce0c6c-933f-4559-89ae-6af9e6373cc9\") " pod="openshift-monitoring/node-exporter-lhjfm"
Sep 09 19:56:37.577640 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:37.577626    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3cce0c6c-933f-4559-89ae-6af9e6373cc9-node-exporter-textfile\") pod \"node-exporter-lhjfm\" (UID: \"3cce0c6c-933f-4559-89ae-6af9e6373cc9\") " pod="openshift-monitoring/node-exporter-lhjfm"
Sep 09 19:56:37.577838 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:37.577820    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3cce0c6c-933f-4559-89ae-6af9e6373cc9-node-exporter-accelerators-collector-config\") pod \"node-exporter-lhjfm\" (UID: \"3cce0c6c-933f-4559-89ae-6af9e6373cc9\") " pod="openshift-monitoring/node-exporter-lhjfm"
Sep 09 19:56:37.579845 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:37.579820    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3cce0c6c-933f-4559-89ae-6af9e6373cc9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lhjfm\" (UID: \"3cce0c6c-933f-4559-89ae-6af9e6373cc9\") " pod="openshift-monitoring/node-exporter-lhjfm"
Sep 09 19:56:37.584046 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:37.584017    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm6m2\" (UniqueName: \"kubernetes.io/projected/3cce0c6c-933f-4559-89ae-6af9e6373cc9-kube-api-access-cm6m2\") pod \"node-exporter-lhjfm\" (UID: \"3cce0c6c-933f-4559-89ae-6af9e6373cc9\") " pod="openshift-monitoring/node-exporter-lhjfm"
Sep 09 19:56:38.081093 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:38.081061    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3cce0c6c-933f-4559-89ae-6af9e6373cc9-node-exporter-tls\") pod \"node-exporter-lhjfm\" (UID: \"3cce0c6c-933f-4559-89ae-6af9e6373cc9\") " pod="openshift-monitoring/node-exporter-lhjfm"
Sep 09 19:56:38.083458 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:38.083435    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3cce0c6c-933f-4559-89ae-6af9e6373cc9-node-exporter-tls\") pod \"node-exporter-lhjfm\" (UID: \"3cce0c6c-933f-4559-89ae-6af9e6373cc9\") " pod="openshift-monitoring/node-exporter-lhjfm"
Sep 09 19:56:38.296206 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:38.296178    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lhjfm"
Sep 09 19:56:38.308758 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:56:38.308728    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cce0c6c_933f_4559_89ae_6af9e6373cc9.slice/crio-9c3df6ecd0ba9f97c8d19e5412984c2cc10982439f7561c353ba16d1866c91d7 WatchSource:0}: Error finding container 9c3df6ecd0ba9f97c8d19e5412984c2cc10982439f7561c353ba16d1866c91d7: Status 404 returned error can't find the container with id 9c3df6ecd0ba9f97c8d19e5412984c2cc10982439f7561c353ba16d1866c91d7
Sep 09 19:56:39.221370 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:39.221327    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lhjfm" event={"ID":"3cce0c6c-933f-4559-89ae-6af9e6373cc9","Type":"ContainerStarted","Data":"9c3df6ecd0ba9f97c8d19e5412984c2cc10982439f7561c353ba16d1866c91d7"}
Sep 09 19:56:40.225863 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:40.225827    2523 generic.go:356] "Generic (PLEG): container finished" podID="3cce0c6c-933f-4559-89ae-6af9e6373cc9" containerID="5e32060bd03167200d9b63d8e5cff30e0dec56536d6a833bcecf24e2ab93c8f3" exitCode=0
Sep 09 19:56:40.226239 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:40.225893    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lhjfm" event={"ID":"3cce0c6c-933f-4559-89ae-6af9e6373cc9","Type":"ContainerDied","Data":"5e32060bd03167200d9b63d8e5cff30e0dec56536d6a833bcecf24e2ab93c8f3"}
Sep 09 19:56:41.231123 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:41.231088    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lhjfm" event={"ID":"3cce0c6c-933f-4559-89ae-6af9e6373cc9","Type":"ContainerStarted","Data":"7897ee5ef9a3bab55d14b016ef98e0bda1a48a6c27dea5f96ab7466c32b623a1"}
Sep 09 19:56:41.231587 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:41.231144    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lhjfm" event={"ID":"3cce0c6c-933f-4559-89ae-6af9e6373cc9","Type":"ContainerStarted","Data":"edaafdfa20da8346a215ac705b63b41dc6551c86de35f8fb1c7fc214f584d6ee"}
Sep 09 19:56:41.258428 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:41.258386    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-lhjfm" podStartSLOduration=3.153796785 podStartE2EDuration="4.258372558s" totalImagesPullingTime="1.035952416s" totalInitContainerRuntime="68.623357ms" isStatefulPod=true podCreationTimestamp="2026-09-09 19:56:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:41.256598834 +0000 UTC m=+105.928617852" watchObservedRunningTime="2026-09-09 19:56:41.258372558 +0000 UTC m=+105.930391556"
Sep 09 19:56:47.217726 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:47.217692    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zr55l"
Sep 09 19:56:47.543380 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:47.543308    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-bcc64b9dc-p7sz5"]
Sep 09 19:56:57.549113 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:56:57.549084    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-bcc64b9dc-p7sz5"
Sep 09 19:57:05.311296 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:57:05.311261    2523 generic.go:356] "Generic (PLEG): container finished" podID="9e6f3e8a-9da3-48f8-a081-dfcfa2917b40" containerID="f6cb2b8d676e26fb78dea4d506eb8e57a41bc7ed44f983c2a4e9c4594a6a4821" exitCode=0
Sep 09 19:57:05.311740 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:57:05.311312    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-fn5ll" event={"ID":"9e6f3e8a-9da3-48f8-a081-dfcfa2917b40","Type":"ContainerDied","Data":"f6cb2b8d676e26fb78dea4d506eb8e57a41bc7ed44f983c2a4e9c4594a6a4821"}
Sep 09 19:57:05.311740 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:57:05.311679    2523 scope.go:117] "RemoveContainer" containerID="f6cb2b8d676e26fb78dea4d506eb8e57a41bc7ed44f983c2a4e9c4594a6a4821"
Sep 09 19:57:05.568138 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:57:05.568105    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8-metrics-certs\") pod \"network-metrics-daemon-tmvpv\" (UID: \"21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8\") " pod="openshift-multus/network-metrics-daemon-tmvpv"
Sep 09 19:57:05.570411 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:57:05.570391    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8-metrics-certs\") pod \"network-metrics-daemon-tmvpv\" (UID: \"21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8\") " pod="openshift-multus/network-metrics-daemon-tmvpv"
Sep 09 19:57:05.830609 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:57:05.830547    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7r2s2\""
Sep 09 19:57:05.838971 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:57:05.838954    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tmvpv"
Sep 09 19:57:06.107319 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:57:06.107295    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tmvpv"]
Sep 09 19:57:06.110072 ip-10-0-132-226 kubenswrapper[2523]: W0909 19:57:06.110037    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21f9ae32_c8d6_42ea_94cf_4cfe645ee5c8.slice/crio-9d24f21bb31f02efb8b67afb64140715eed53dec23c1e7834b810756b3074419 WatchSource:0}: Error finding container 9d24f21bb31f02efb8b67afb64140715eed53dec23c1e7834b810756b3074419: Status 404 returned error can't find the container with id 9d24f21bb31f02efb8b67afb64140715eed53dec23c1e7834b810756b3074419
Sep 09 19:57:06.316107 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:57:06.316078    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tmvpv" event={"ID":"21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8","Type":"ContainerStarted","Data":"9d24f21bb31f02efb8b67afb64140715eed53dec23c1e7834b810756b3074419"}
Sep 09 19:57:06.317924 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:57:06.317899    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-fn5ll" event={"ID":"9e6f3e8a-9da3-48f8-a081-dfcfa2917b40","Type":"ContainerStarted","Data":"ade53382cd60fe70bd58b342feb53d9b31668ee8dd7c17039095c1ab7553c0c1"}
Sep 09 19:57:08.325657 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:57:08.325622    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tmvpv" event={"ID":"21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8","Type":"ContainerStarted","Data":"2c8e3b64c6887cb9b62cc6e1b205383f63c0e9eb37b4b357317964b040264b4b"}
Sep 09 19:57:08.325982 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:57:08.325684    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tmvpv" event={"ID":"21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8","Type":"ContainerStarted","Data":"cf838c725c72da96525f1f8d4fce479e192e4fdbde64de0cd98191b6c0e2d4f9"}
Sep 09 19:57:08.342322 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:57:08.342283    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-tmvpv" podStartSLOduration=131.54632948 podStartE2EDuration="2m13.342268983s" totalImagesPullingTime="1.795939503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:54:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:57:08.341491154 +0000 UTC m=+133.013510152" watchObservedRunningTime="2026-09-09 19:57:08.342268983 +0000 UTC m=+133.014287980"
Sep 09 19:57:12.562378 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:57:12.562324    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-bcc64b9dc-p7sz5" podUID="fb48971c-63be-4de1-b19e-0b8c1aadbc58" containerName="registry" containerID="cri-o://f03fa6b561dfc291595e7606b966810c2d66fa218b7c623fd99b3f2f4de96fa5" gracePeriod=30
Sep 09 19:57:12.798850 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:57:12.798827    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bcc64b9dc-p7sz5"
Sep 09 19:57:12.808532 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:57:12.808512    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fb48971c-63be-4de1-b19e-0b8c1aadbc58-bound-sa-token\") pod \"fb48971c-63be-4de1-b19e-0b8c1aadbc58\" (UID: \"fb48971c-63be-4de1-b19e-0b8c1aadbc58\") "
Sep 09 19:57:12.808637 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:57:12.808543    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fb48971c-63be-4de1-b19e-0b8c1aadbc58-registry-certificates\") pod \"fb48971c-63be-4de1-b19e-0b8c1aadbc58\" (UID: \"fb48971c-63be-4de1-b19e-0b8c1aadbc58\") "
Sep 09 19:57:12.808637 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:57:12.808571    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fb48971c-63be-4de1-b19e-0b8c1aadbc58-installation-pull-secrets\") pod \"fb48971c-63be-4de1-b19e-0b8c1aadbc58\" (UID: \"fb48971c-63be-4de1-b19e-0b8c1aadbc58\") "
Sep 09 19:57:12.808637 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:57:12.808607    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fb48971c-63be-4de1-b19e-0b8c1aadbc58-registry-tls\") pod \"fb48971c-63be-4de1-b19e-0b8c1aadbc58\" (UID: \"fb48971c-63be-4de1-b19e-0b8c1aadbc58\") "
Sep 09 19:57:12.808796 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:57:12.808639    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jsvj2\" (UniqueName: \"kubernetes.io/projected/fb48971c-63be-4de1-b19e-0b8c1aadbc58-kube-api-access-jsvj2\") pod \"fb48971c-63be-4de1-b19e-0b8c1aadbc58\" (UID: \"fb48971c-63be-4de1-b19e-0b8c1aadbc58\") "
Sep 09 19:57:12.808796 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:57:12.808668    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fb48971c-63be-4de1-b19e-0b8c1aadbc58-image-registry-private-configuration\") pod \"fb48971c-63be-4de1-b19e-0b8c1aadbc58\" (UID: \"fb48971c-63be-4de1-b19e-0b8c1aadbc58\") "
Sep 09 19:57:12.808796 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:57:12.808710    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fb48971c-63be-4de1-b19e-0b8c1aadbc58-ca-trust-extracted\") pod \"fb48971c-63be-4de1-b19e-0b8c1aadbc58\" (UID: \"fb48971c-63be-4de1-b19e-0b8c1aadbc58\") "
Sep 09 19:57:12.808796 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:57:12.808735    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fb48971c-63be-4de1-b19e-0b8c1aadbc58-trusted-ca\") pod \"fb48971c-63be-4de1-b19e-0b8c1aadbc58\" (UID: \"fb48971c-63be-4de1-b19e-0b8c1aadbc58\") "
Sep 09 19:57:12.809000 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:57:12.808946    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb48971c-63be-4de1-b19e-0b8c1aadbc58-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "fb48971c-63be-4de1-b19e-0b8c1aadbc58" (UID: "fb48971c-63be-4de1-b19e-0b8c1aadbc58"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:57:12.809366 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:57:12.809325    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb48971c-63be-4de1-b19e-0b8c1aadbc58-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "fb48971c-63be-4de1-b19e-0b8c1aadbc58" (UID: "fb48971c-63be-4de1-b19e-0b8c1aadbc58"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:57:12.811056 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:57:12.811031    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb48971c-63be-4de1-b19e-0b8c1aadbc58-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "fb48971c-63be-4de1-b19e-0b8c1aadbc58" (UID: "fb48971c-63be-4de1-b19e-0b8c1aadbc58"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 19:57:12.811362 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:57:12.811306    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb48971c-63be-4de1-b19e-0b8c1aadbc58-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "fb48971c-63be-4de1-b19e-0b8c1aadbc58" (UID: "fb48971c-63be-4de1-b19e-0b8c1aadbc58"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:57:12.811459 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:57:12.811391    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb48971c-63be-4de1-b19e-0b8c1aadbc58-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "fb48971c-63be-4de1-b19e-0b8c1aadbc58" (UID: "fb48971c-63be-4de1-b19e-0b8c1aadbc58"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:57:12.811521 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:57:12.811491    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb48971c-63be-4de1-b19e-0b8c1aadbc58-kube-api-access-jsvj2" (OuterVolumeSpecName: "kube-api-access-jsvj2") pod "fb48971c-63be-4de1-b19e-0b8c1aadbc58" (UID: "fb48971c-63be-4de1-b19e-0b8c1aadbc58"). InnerVolumeSpecName "kube-api-access-jsvj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 19:57:12.811521 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:57:12.811502    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb48971c-63be-4de1-b19e-0b8c1aadbc58-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "fb48971c-63be-4de1-b19e-0b8c1aadbc58" (UID: "fb48971c-63be-4de1-b19e-0b8c1aadbc58"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 19:57:12.817345 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:57:12.817319    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb48971c-63be-4de1-b19e-0b8c1aadbc58-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "fb48971c-63be-4de1-b19e-0b8c1aadbc58" (UID: "fb48971c-63be-4de1-b19e-0b8c1aadbc58"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 19:57:12.909099 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:57:12.909079    2523 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fb48971c-63be-4de1-b19e-0b8c1aadbc58-registry-tls\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 19:57:12.909099 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:57:12.909098    2523 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsvj2\" (UniqueName: \"kubernetes.io/projected/fb48971c-63be-4de1-b19e-0b8c1aadbc58-kube-api-access-jsvj2\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 19:57:12.909258 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:57:12.909107    2523 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fb48971c-63be-4de1-b19e-0b8c1aadbc58-image-registry-private-configuration\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 19:57:12.909258 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:57:12.909116    2523 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fb48971c-63be-4de1-b19e-0b8c1aadbc58-ca-trust-extracted\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 19:57:12.909258 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:57:12.909125    2523 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fb48971c-63be-4de1-b19e-0b8c1aadbc58-trusted-ca\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 19:57:12.909258 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:57:12.909133    2523 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fb48971c-63be-4de1-b19e-0b8c1aadbc58-bound-sa-token\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 19:57:12.909258 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:57:12.909140    2523 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fb48971c-63be-4de1-b19e-0b8c1aadbc58-registry-certificates\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 19:57:12.909258 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:57:12.909165    2523 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fb48971c-63be-4de1-b19e-0b8c1aadbc58-installation-pull-secrets\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 19:57:13.342761 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:57:13.342730    2523 generic.go:356] "Generic (PLEG): container finished" podID="fb48971c-63be-4de1-b19e-0b8c1aadbc58" containerID="f03fa6b561dfc291595e7606b966810c2d66fa218b7c623fd99b3f2f4de96fa5" exitCode=0
Sep 09 19:57:13.342879 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:57:13.342775    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bcc64b9dc-p7sz5" event={"ID":"fb48971c-63be-4de1-b19e-0b8c1aadbc58","Type":"ContainerDied","Data":"f03fa6b561dfc291595e7606b966810c2d66fa218b7c623fd99b3f2f4de96fa5"}
Sep 09 19:57:13.342879 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:57:13.342778    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bcc64b9dc-p7sz5"
Sep 09 19:57:13.342879 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:57:13.342818    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bcc64b9dc-p7sz5" event={"ID":"fb48971c-63be-4de1-b19e-0b8c1aadbc58","Type":"ContainerDied","Data":"9eb9476f3a1bf7396abf81e6eefd16296a3455a946fec07a146cd2f4c005eac8"}
Sep 09 19:57:13.342879 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:57:13.342840    2523 scope.go:117] "RemoveContainer" containerID="f03fa6b561dfc291595e7606b966810c2d66fa218b7c623fd99b3f2f4de96fa5"
Sep 09 19:57:13.351118 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:57:13.351098    2523 scope.go:117] "RemoveContainer" containerID="f03fa6b561dfc291595e7606b966810c2d66fa218b7c623fd99b3f2f4de96fa5"
Sep 09 19:57:13.351418 ip-10-0-132-226 kubenswrapper[2523]: E0909 19:57:13.351396    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f03fa6b561dfc291595e7606b966810c2d66fa218b7c623fd99b3f2f4de96fa5\": container with ID starting with f03fa6b561dfc291595e7606b966810c2d66fa218b7c623fd99b3f2f4de96fa5 not found: ID does not exist" containerID="f03fa6b561dfc291595e7606b966810c2d66fa218b7c623fd99b3f2f4de96fa5"
Sep 09 19:57:13.351476 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:57:13.351425    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f03fa6b561dfc291595e7606b966810c2d66fa218b7c623fd99b3f2f4de96fa5"} err="failed to get container status \"f03fa6b561dfc291595e7606b966810c2d66fa218b7c623fd99b3f2f4de96fa5\": rpc error: code = NotFound desc = could not find container \"f03fa6b561dfc291595e7606b966810c2d66fa218b7c623fd99b3f2f4de96fa5\": container with ID starting with f03fa6b561dfc291595e7606b966810c2d66fa218b7c623fd99b3f2f4de96fa5 not found: ID does not exist"
Sep 09 19:57:13.363189 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:57:13.363140    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-bcc64b9dc-p7sz5"]
Sep 09 19:57:13.369993 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:57:13.369971    2523 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-bcc64b9dc-p7sz5"]
Sep 09 19:57:13.810066 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:57:13.810009    2523 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb48971c-63be-4de1-b19e-0b8c1aadbc58" path="/var/lib/kubelet/pods/fb48971c-63be-4de1-b19e-0b8c1aadbc58/volumes"
Sep 09 19:59:55.714746 ip-10-0-132-226 kubenswrapper[2523]: I0909 19:59:55.714719    2523 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 20:01:04.757121 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:04.757033    2523 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1amsbrv"]
Sep 09 20:01:04.757514 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:04.757294    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb48971c-63be-4de1-b19e-0b8c1aadbc58" containerName="registry"
Sep 09 20:01:04.757514 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:04.757305    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb48971c-63be-4de1-b19e-0b8c1aadbc58" containerName="registry"
Sep 09 20:01:04.757514 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:04.757352    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb48971c-63be-4de1-b19e-0b8c1aadbc58" containerName="registry"
Sep 09 20:01:04.759973 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:04.759955    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1amsbrv"
Sep 09 20:01:04.762960 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:04.762939    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 20:01:04.763920 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:04.763903    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 20:01:04.764009 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:04.763968    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-6rjmn\""
Sep 09 20:01:04.810630 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:04.810610    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1amsbrv"]
Sep 09 20:01:04.859686 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:04.859666    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00a64dd9-1e54-4700-a82f-5a6351ee67f6-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1amsbrv\" (UID: \"00a64dd9-1e54-4700-a82f-5a6351ee67f6\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1amsbrv"
Sep 09 20:01:04.859780 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:04.859695    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2gnq\" (UniqueName: \"kubernetes.io/projected/00a64dd9-1e54-4700-a82f-5a6351ee67f6-kube-api-access-m2gnq\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1amsbrv\" (UID: \"00a64dd9-1e54-4700-a82f-5a6351ee67f6\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1amsbrv"
Sep 09 20:01:04.859780 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:04.859714    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00a64dd9-1e54-4700-a82f-5a6351ee67f6-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1amsbrv\" (UID: \"00a64dd9-1e54-4700-a82f-5a6351ee67f6\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1amsbrv"
Sep 09 20:01:04.960551 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:04.960527    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00a64dd9-1e54-4700-a82f-5a6351ee67f6-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1amsbrv\" (UID: \"00a64dd9-1e54-4700-a82f-5a6351ee67f6\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1amsbrv"
Sep 09 20:01:04.960639 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:04.960554    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2gnq\" (UniqueName: \"kubernetes.io/projected/00a64dd9-1e54-4700-a82f-5a6351ee67f6-kube-api-access-m2gnq\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1amsbrv\" (UID: \"00a64dd9-1e54-4700-a82f-5a6351ee67f6\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1amsbrv"
Sep 09 20:01:04.960639 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:04.960578    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00a64dd9-1e54-4700-a82f-5a6351ee67f6-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1amsbrv\" (UID: \"00a64dd9-1e54-4700-a82f-5a6351ee67f6\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1amsbrv"
Sep 09 20:01:04.960941 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:04.960920    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00a64dd9-1e54-4700-a82f-5a6351ee67f6-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1amsbrv\" (UID: \"00a64dd9-1e54-4700-a82f-5a6351ee67f6\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1amsbrv"
Sep 09 20:01:04.960979 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:04.960931    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00a64dd9-1e54-4700-a82f-5a6351ee67f6-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1amsbrv\" (UID: \"00a64dd9-1e54-4700-a82f-5a6351ee67f6\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1amsbrv"
Sep 09 20:01:04.972485 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:04.972466    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2gnq\" (UniqueName: \"kubernetes.io/projected/00a64dd9-1e54-4700-a82f-5a6351ee67f6-kube-api-access-m2gnq\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1amsbrv\" (UID: \"00a64dd9-1e54-4700-a82f-5a6351ee67f6\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1amsbrv"
Sep 09 20:01:05.068069 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:05.068050    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1amsbrv"
Sep 09 20:01:05.341825 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:05.341802    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1amsbrv"]
Sep 09 20:01:05.344212 ip-10-0-132-226 kubenswrapper[2523]: W0909 20:01:05.344185    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00a64dd9_1e54_4700_a82f_5a6351ee67f6.slice/crio-831618595d7472057eed39a7885e83d3e045c0fd01932f122c6515c00657af11 WatchSource:0}: Error finding container 831618595d7472057eed39a7885e83d3e045c0fd01932f122c6515c00657af11: Status 404 returned error can't find the container with id 831618595d7472057eed39a7885e83d3e045c0fd01932f122c6515c00657af11
Sep 09 20:01:05.346057 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:05.346043    2523 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:01:06.084576 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:06.084539    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1amsbrv" event={"ID":"00a64dd9-1e54-4700-a82f-5a6351ee67f6","Type":"ContainerStarted","Data":"831618595d7472057eed39a7885e83d3e045c0fd01932f122c6515c00657af11"}
Sep 09 20:01:12.107154 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:12.107119    2523 generic.go:356] "Generic (PLEG): container finished" podID="00a64dd9-1e54-4700-a82f-5a6351ee67f6" containerID="0540602492ab9e3636b76f87d796c66f2f706d199470e22b656b4c881afd997d" exitCode=0
Sep 09 20:01:12.107470 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:12.107209    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1amsbrv" event={"ID":"00a64dd9-1e54-4700-a82f-5a6351ee67f6","Type":"ContainerDied","Data":"0540602492ab9e3636b76f87d796c66f2f706d199470e22b656b4c881afd997d"}
Sep 09 20:01:37.190100 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:37.190066    2523 generic.go:356] "Generic (PLEG): container finished" podID="00a64dd9-1e54-4700-a82f-5a6351ee67f6" containerID="45a872fbb7b37f5977bd0f1f467d8a188b9781ba322d9bd17c00f5a37029f2b8" exitCode=0
Sep 09 20:01:37.190449 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:37.190159    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1amsbrv" event={"ID":"00a64dd9-1e54-4700-a82f-5a6351ee67f6","Type":"ContainerDied","Data":"45a872fbb7b37f5977bd0f1f467d8a188b9781ba322d9bd17c00f5a37029f2b8"}
Sep 09 20:01:45.221477 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:45.221448    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1amsbrv" event={"ID":"00a64dd9-1e54-4700-a82f-5a6351ee67f6","Type":"ContainerStarted","Data":"6b117129406aac90f0b829acf8f66dba6c6c35acf379c7a39ed50e65aa771c09"}
Sep 09 20:01:45.240626 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:45.240587    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1amsbrv" podStartSLOduration=2.1313512709999998 podStartE2EDuration="41.240566566s" totalImagesPullingTime="39.109215295s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:01:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:01:45.239105804 +0000 UTC m=+409.911124800" watchObservedRunningTime="2026-09-09 20:01:45.240566566 +0000 UTC m=+409.912585563"
Sep 09 20:01:46.226757 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:46.226723    2523 generic.go:356] "Generic (PLEG): container finished" podID="00a64dd9-1e54-4700-a82f-5a6351ee67f6" containerID="6b117129406aac90f0b829acf8f66dba6c6c35acf379c7a39ed50e65aa771c09" exitCode=0
Sep 09 20:01:46.227114 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:46.226802    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1amsbrv" event={"ID":"00a64dd9-1e54-4700-a82f-5a6351ee67f6","Type":"ContainerDied","Data":"6b117129406aac90f0b829acf8f66dba6c6c35acf379c7a39ed50e65aa771c09"}
Sep 09 20:01:47.348834 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:47.348808    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1amsbrv"
Sep 09 20:01:47.532346 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:47.532277    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00a64dd9-1e54-4700-a82f-5a6351ee67f6-bundle\") pod \"00a64dd9-1e54-4700-a82f-5a6351ee67f6\" (UID: \"00a64dd9-1e54-4700-a82f-5a6351ee67f6\") "
Sep 09 20:01:47.532346 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:47.532342    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m2gnq\" (UniqueName: \"kubernetes.io/projected/00a64dd9-1e54-4700-a82f-5a6351ee67f6-kube-api-access-m2gnq\") pod \"00a64dd9-1e54-4700-a82f-5a6351ee67f6\" (UID: \"00a64dd9-1e54-4700-a82f-5a6351ee67f6\") "
Sep 09 20:01:47.532512 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:47.532367    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00a64dd9-1e54-4700-a82f-5a6351ee67f6-util\") pod \"00a64dd9-1e54-4700-a82f-5a6351ee67f6\" (UID: \"00a64dd9-1e54-4700-a82f-5a6351ee67f6\") "
Sep 09 20:01:47.532924 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:47.532898    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00a64dd9-1e54-4700-a82f-5a6351ee67f6-bundle" (OuterVolumeSpecName: "bundle") pod "00a64dd9-1e54-4700-a82f-5a6351ee67f6" (UID: "00a64dd9-1e54-4700-a82f-5a6351ee67f6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:01:47.534539 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:47.534510    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00a64dd9-1e54-4700-a82f-5a6351ee67f6-kube-api-access-m2gnq" (OuterVolumeSpecName: "kube-api-access-m2gnq") pod "00a64dd9-1e54-4700-a82f-5a6351ee67f6" (UID: "00a64dd9-1e54-4700-a82f-5a6351ee67f6"). InnerVolumeSpecName "kube-api-access-m2gnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:01:47.538926 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:47.538902    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00a64dd9-1e54-4700-a82f-5a6351ee67f6-util" (OuterVolumeSpecName: "util") pod "00a64dd9-1e54-4700-a82f-5a6351ee67f6" (UID: "00a64dd9-1e54-4700-a82f-5a6351ee67f6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:01:47.633415 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:47.633381    2523 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00a64dd9-1e54-4700-a82f-5a6351ee67f6-bundle\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:01:47.633415 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:47.633411    2523 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2gnq\" (UniqueName: \"kubernetes.io/projected/00a64dd9-1e54-4700-a82f-5a6351ee67f6-kube-api-access-m2gnq\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:01:47.633415 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:47.633420    2523 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00a64dd9-1e54-4700-a82f-5a6351ee67f6-util\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:01:48.234475 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:48.234456    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1amsbrv"
Sep 09 20:01:48.234594 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:48.234455    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1amsbrv" event={"ID":"00a64dd9-1e54-4700-a82f-5a6351ee67f6","Type":"ContainerDied","Data":"831618595d7472057eed39a7885e83d3e045c0fd01932f122c6515c00657af11"}
Sep 09 20:01:48.234594 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:48.234577    2523 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="831618595d7472057eed39a7885e83d3e045c0fd01932f122c6515c00657af11"
Sep 09 20:01:59.620188 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:59.620138    2523 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-t2thk"]
Sep 09 20:01:59.620535 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:59.620382    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00a64dd9-1e54-4700-a82f-5a6351ee67f6" containerName="extract"
Sep 09 20:01:59.620535 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:59.620392    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="00a64dd9-1e54-4700-a82f-5a6351ee67f6" containerName="extract"
Sep 09 20:01:59.620535 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:59.620403    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00a64dd9-1e54-4700-a82f-5a6351ee67f6" containerName="pull"
Sep 09 20:01:59.620535 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:59.620409    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="00a64dd9-1e54-4700-a82f-5a6351ee67f6" containerName="pull"
Sep 09 20:01:59.620535 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:59.620422    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00a64dd9-1e54-4700-a82f-5a6351ee67f6" containerName="util"
Sep 09 20:01:59.620535 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:59.620427    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="00a64dd9-1e54-4700-a82f-5a6351ee67f6" containerName="util"
Sep 09 20:01:59.620535 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:59.620468    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="00a64dd9-1e54-4700-a82f-5a6351ee67f6" containerName="extract"
Sep 09 20:01:59.622623 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:59.622607    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-t2thk"
Sep 09 20:01:59.625302 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:59.625279    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 09 20:01:59.625439 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:59.625321    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 09 20:01:59.625439 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:59.625398    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 09 20:01:59.625604 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:59.625588    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 09 20:01:59.626262 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:59.626247    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 09 20:01:59.626374 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:59.626268    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-w8wgg\""
Sep 09 20:01:59.634315 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:59.634294    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-t2thk"]
Sep 09 20:01:59.707162 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:59.707130    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/33223478-c613-4d26-b91b-9da8c4e4a6ef-certificates\") pod \"keda-operator-579c8958f7-t2thk\" (UID: \"33223478-c613-4d26-b91b-9da8c4e4a6ef\") " pod="openshift-keda/keda-operator-579c8958f7-t2thk"
Sep 09 20:01:59.707252 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:59.707198    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clp8v\" (UniqueName: \"kubernetes.io/projected/33223478-c613-4d26-b91b-9da8c4e4a6ef-kube-api-access-clp8v\") pod \"keda-operator-579c8958f7-t2thk\" (UID: \"33223478-c613-4d26-b91b-9da8c4e4a6ef\") " pod="openshift-keda/keda-operator-579c8958f7-t2thk"
Sep 09 20:01:59.707252 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:59.707221    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/33223478-c613-4d26-b91b-9da8c4e4a6ef-cabundle0\") pod \"keda-operator-579c8958f7-t2thk\" (UID: \"33223478-c613-4d26-b91b-9da8c4e4a6ef\") " pod="openshift-keda/keda-operator-579c8958f7-t2thk"
Sep 09 20:01:59.807950 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:59.807923    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clp8v\" (UniqueName: \"kubernetes.io/projected/33223478-c613-4d26-b91b-9da8c4e4a6ef-kube-api-access-clp8v\") pod \"keda-operator-579c8958f7-t2thk\" (UID: \"33223478-c613-4d26-b91b-9da8c4e4a6ef\") " pod="openshift-keda/keda-operator-579c8958f7-t2thk"
Sep 09 20:01:59.808066 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:59.807964    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/33223478-c613-4d26-b91b-9da8c4e4a6ef-cabundle0\") pod \"keda-operator-579c8958f7-t2thk\" (UID: \"33223478-c613-4d26-b91b-9da8c4e4a6ef\") " pod="openshift-keda/keda-operator-579c8958f7-t2thk"
Sep 09 20:01:59.808066 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:59.808032    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/33223478-c613-4d26-b91b-9da8c4e4a6ef-certificates\") pod \"keda-operator-579c8958f7-t2thk\" (UID: \"33223478-c613-4d26-b91b-9da8c4e4a6ef\") " pod="openshift-keda/keda-operator-579c8958f7-t2thk"
Sep 09 20:01:59.808185 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:01:59.808131    2523 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Sep 09 20:01:59.808185 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:01:59.808176    2523 secret.go:281] references non-existent secret key: ca.crt
Sep 09 20:01:59.808300 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:01:59.808188    2523 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 09 20:01:59.808300 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:01:59.808219    2523 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-t2thk: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Sep 09 20:01:59.808300 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:01:59.808280    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33223478-c613-4d26-b91b-9da8c4e4a6ef-certificates podName:33223478-c613-4d26-b91b-9da8c4e4a6ef nodeName:}" failed. No retries permitted until 2026-09-09 20:02:00.308261122 +0000 UTC m=+424.980280114 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/33223478-c613-4d26-b91b-9da8c4e4a6ef-certificates") pod "keda-operator-579c8958f7-t2thk" (UID: "33223478-c613-4d26-b91b-9da8c4e4a6ef") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Sep 09 20:01:59.808737 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:59.808711    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/33223478-c613-4d26-b91b-9da8c4e4a6ef-cabundle0\") pod \"keda-operator-579c8958f7-t2thk\" (UID: \"33223478-c613-4d26-b91b-9da8c4e4a6ef\") " pod="openshift-keda/keda-operator-579c8958f7-t2thk"
Sep 09 20:01:59.820530 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:01:59.820502    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clp8v\" (UniqueName: \"kubernetes.io/projected/33223478-c613-4d26-b91b-9da8c4e4a6ef-kube-api-access-clp8v\") pod \"keda-operator-579c8958f7-t2thk\" (UID: \"33223478-c613-4d26-b91b-9da8c4e4a6ef\") " pod="openshift-keda/keda-operator-579c8958f7-t2thk"
Sep 09 20:02:00.122470 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:02:00.122443    2523 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-x9qms"]
Sep 09 20:02:00.124769 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:02:00.124755    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-x9qms"
Sep 09 20:02:00.127594 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:02:00.127579    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 09 20:02:00.140832 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:02:00.140812    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-x9qms"]
Sep 09 20:02:00.210902 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:02:00.210876    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/58f63578-1d43-4257-ac0c-e0384659db59-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-x9qms\" (UID: \"58f63578-1d43-4257-ac0c-e0384659db59\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-x9qms"
Sep 09 20:02:00.211007 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:02:00.210907    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m8f2\" (UniqueName: \"kubernetes.io/projected/58f63578-1d43-4257-ac0c-e0384659db59-kube-api-access-9m8f2\") pod \"keda-metrics-apiserver-77857b7b98-x9qms\" (UID: \"58f63578-1d43-4257-ac0c-e0384659db59\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-x9qms"
Sep 09 20:02:00.211007 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:02:00.210934    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/58f63578-1d43-4257-ac0c-e0384659db59-certificates\") pod \"keda-metrics-apiserver-77857b7b98-x9qms\" (UID: \"58f63578-1d43-4257-ac0c-e0384659db59\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-x9qms"
Sep 09 20:02:00.311531 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:02:00.311512    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/33223478-c613-4d26-b91b-9da8c4e4a6ef-certificates\") pod \"keda-operator-579c8958f7-t2thk\" (UID: \"33223478-c613-4d26-b91b-9da8c4e4a6ef\") " pod="openshift-keda/keda-operator-579c8958f7-t2thk"
Sep 09 20:02:00.311646 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:02:00.311539    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/58f63578-1d43-4257-ac0c-e0384659db59-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-x9qms\" (UID: \"58f63578-1d43-4257-ac0c-e0384659db59\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-x9qms"
Sep 09 20:02:00.311646 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:02:00.311557    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m8f2\" (UniqueName: \"kubernetes.io/projected/58f63578-1d43-4257-ac0c-e0384659db59-kube-api-access-9m8f2\") pod \"keda-metrics-apiserver-77857b7b98-x9qms\" (UID: \"58f63578-1d43-4257-ac0c-e0384659db59\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-x9qms"
Sep 09 20:02:00.311646 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:02:00.311581    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/58f63578-1d43-4257-ac0c-e0384659db59-certificates\") pod \"keda-metrics-apiserver-77857b7b98-x9qms\" (UID: \"58f63578-1d43-4257-ac0c-e0384659db59\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-x9qms"
Sep 09 20:02:00.311899 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:02:00.311876    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/58f63578-1d43-4257-ac0c-e0384659db59-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-x9qms\" (UID: \"58f63578-1d43-4257-ac0c-e0384659db59\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-x9qms"
Sep 09 20:02:00.314000 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:02:00.313982    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/33223478-c613-4d26-b91b-9da8c4e4a6ef-certificates\") pod \"keda-operator-579c8958f7-t2thk\" (UID: \"33223478-c613-4d26-b91b-9da8c4e4a6ef\") " pod="openshift-keda/keda-operator-579c8958f7-t2thk"
Sep 09 20:02:00.314250 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:02:00.314233    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/58f63578-1d43-4257-ac0c-e0384659db59-certificates\") pod \"keda-metrics-apiserver-77857b7b98-x9qms\" (UID: \"58f63578-1d43-4257-ac0c-e0384659db59\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-x9qms"
Sep 09 20:02:00.320695 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:02:00.320676    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m8f2\" (UniqueName: \"kubernetes.io/projected/58f63578-1d43-4257-ac0c-e0384659db59-kube-api-access-9m8f2\") pod \"keda-metrics-apiserver-77857b7b98-x9qms\" (UID: \"58f63578-1d43-4257-ac0c-e0384659db59\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-x9qms"
Sep 09 20:02:00.434474 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:02:00.434408    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-x9qms"
Sep 09 20:02:00.532328 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:02:00.532297    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-t2thk"
Sep 09 20:02:00.703060 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:02:00.702992    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-x9qms"]
Sep 09 20:02:00.705832 ip-10-0-132-226 kubenswrapper[2523]: W0909 20:02:00.705800    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58f63578_1d43_4257_ac0c_e0384659db59.slice/crio-68cc2b90afb397c13a00f14632420ea536e5dd8c0aa4373ecb5e288ee303c89f WatchSource:0}: Error finding container 68cc2b90afb397c13a00f14632420ea536e5dd8c0aa4373ecb5e288ee303c89f: Status 404 returned error can't find the container with id 68cc2b90afb397c13a00f14632420ea536e5dd8c0aa4373ecb5e288ee303c89f
Sep 09 20:02:00.799944 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:02:00.799913    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-t2thk"]
Sep 09 20:02:00.802979 ip-10-0-132-226 kubenswrapper[2523]: W0909 20:02:00.802953    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33223478_c613_4d26_b91b_9da8c4e4a6ef.slice/crio-660cbd4e30720d92ac3c99889b9cb020dfca2bf0e4716dbbbabc0e01c7283eb0 WatchSource:0}: Error finding container 660cbd4e30720d92ac3c99889b9cb020dfca2bf0e4716dbbbabc0e01c7283eb0: Status 404 returned error can't find the container with id 660cbd4e30720d92ac3c99889b9cb020dfca2bf0e4716dbbbabc0e01c7283eb0
Sep 09 20:02:01.275336 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:02:01.275298    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-t2thk" event={"ID":"33223478-c613-4d26-b91b-9da8c4e4a6ef","Type":"ContainerStarted","Data":"660cbd4e30720d92ac3c99889b9cb020dfca2bf0e4716dbbbabc0e01c7283eb0"}
Sep 09 20:02:01.276320 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:02:01.276299    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-x9qms" event={"ID":"58f63578-1d43-4257-ac0c-e0384659db59","Type":"ContainerStarted","Data":"68cc2b90afb397c13a00f14632420ea536e5dd8c0aa4373ecb5e288ee303c89f"}
Sep 09 20:02:05.294545 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:02:05.294510    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-x9qms" event={"ID":"58f63578-1d43-4257-ac0c-e0384659db59","Type":"ContainerStarted","Data":"6dfa50a106aed37d2a621972242407c3a38add714bb51040972368267b8c619f"}
Sep 09 20:02:05.295085 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:02:05.294577    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-x9qms"
Sep 09 20:02:05.296308 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:02:05.296275    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-t2thk" event={"ID":"33223478-c613-4d26-b91b-9da8c4e4a6ef","Type":"ContainerStarted","Data":"1f8369e42676789a9f5302ac23b17ead8cc93a731ff241be5ed599c296b56edd"}
Sep 09 20:02:05.296433 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:02:05.296411    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-t2thk"
Sep 09 20:02:05.312743 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:02:05.312689    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-x9qms" podStartSLOduration=1.365710042 podStartE2EDuration="5.31267242s" totalImagesPullingTime="3.946962378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:02:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:02:05.310813546 +0000 UTC m=+429.982832543" watchObservedRunningTime="2026-09-09 20:02:05.31267242 +0000 UTC m=+429.984691418"
Sep 09 20:02:05.326969 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:02:05.326931    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-t2thk" podStartSLOduration=2.471742182 podStartE2EDuration="6.326919495s" totalImagesPullingTime="3.855177313s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:01:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:02:05.326564932 +0000 UTC m=+429.998583929" watchObservedRunningTime="2026-09-09 20:02:05.326919495 +0000 UTC m=+429.998938492"
Sep 09 20:02:16.304735 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:02:16.304707    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-x9qms"
Sep 09 20:02:26.302864 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:02:26.302836    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-t2thk"
Sep 09 20:03:24.383515 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:03:24.383485    2523 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-ddscs"]
Sep 09 20:03:24.385803 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:03:24.385787    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-ddscs"
Sep 09 20:03:24.394466 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:03:24.394444    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 09 20:03:24.394580 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:03:24.394468    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 09 20:03:24.394580 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:03:24.394442    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 09 20:03:24.394580 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:03:24.394497    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-7t4jk\""
Sep 09 20:03:24.416183 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:03:24.416134    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-ddscs"]
Sep 09 20:03:24.462521 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:03:24.462498    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fa3e7d87-045f-480a-9f42-28ffb444623e-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-ddscs\" (UID: \"fa3e7d87-045f-480a-9f42-28ffb444623e\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-ddscs"
Sep 09 20:03:24.462648 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:03:24.462538    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt4r8\" (UniqueName: \"kubernetes.io/projected/fa3e7d87-045f-480a-9f42-28ffb444623e-kube-api-access-zt4r8\") pod \"llmisvc-controller-manager-645f9c8fd6-ddscs\" (UID: \"fa3e7d87-045f-480a-9f42-28ffb444623e\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-ddscs"
Sep 09 20:03:24.563697 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:03:24.563662    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fa3e7d87-045f-480a-9f42-28ffb444623e-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-ddscs\" (UID: \"fa3e7d87-045f-480a-9f42-28ffb444623e\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-ddscs"
Sep 09 20:03:24.563697 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:03:24.563704    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt4r8\" (UniqueName: \"kubernetes.io/projected/fa3e7d87-045f-480a-9f42-28ffb444623e-kube-api-access-zt4r8\") pod \"llmisvc-controller-manager-645f9c8fd6-ddscs\" (UID: \"fa3e7d87-045f-480a-9f42-28ffb444623e\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-ddscs"
Sep 09 20:03:24.563844 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:03:24.563794    2523 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 09 20:03:24.563882 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:03:24.563860    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa3e7d87-045f-480a-9f42-28ffb444623e-cert podName:fa3e7d87-045f-480a-9f42-28ffb444623e nodeName:}" failed. No retries permitted until 2026-09-09 20:03:25.063844192 +0000 UTC m=+509.735863171 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/fa3e7d87-045f-480a-9f42-28ffb444623e-cert") pod "llmisvc-controller-manager-645f9c8fd6-ddscs" (UID: "fa3e7d87-045f-480a-9f42-28ffb444623e") : secret "llmisvc-webhook-server-cert" not found
Sep 09 20:03:24.573296 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:03:24.573267    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt4r8\" (UniqueName: \"kubernetes.io/projected/fa3e7d87-045f-480a-9f42-28ffb444623e-kube-api-access-zt4r8\") pod \"llmisvc-controller-manager-645f9c8fd6-ddscs\" (UID: \"fa3e7d87-045f-480a-9f42-28ffb444623e\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-ddscs"
Sep 09 20:03:25.067991 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:03:25.067956    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fa3e7d87-045f-480a-9f42-28ffb444623e-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-ddscs\" (UID: \"fa3e7d87-045f-480a-9f42-28ffb444623e\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-ddscs"
Sep 09 20:03:25.070437 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:03:25.070416    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fa3e7d87-045f-480a-9f42-28ffb444623e-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-ddscs\" (UID: \"fa3e7d87-045f-480a-9f42-28ffb444623e\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-ddscs"
Sep 09 20:03:25.294659 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:03:25.294632    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-ddscs"
Sep 09 20:03:25.568461 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:03:25.568431    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-ddscs"]
Sep 09 20:03:25.571307 ip-10-0-132-226 kubenswrapper[2523]: W0909 20:03:25.571277    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podfa3e7d87_045f_480a_9f42_28ffb444623e.slice/crio-b219b7892a53f7d009cbee0d4fb290a342eef624401d5c9dbb764a99e4d09a0a WatchSource:0}: Error finding container b219b7892a53f7d009cbee0d4fb290a342eef624401d5c9dbb764a99e4d09a0a: Status 404 returned error can't find the container with id b219b7892a53f7d009cbee0d4fb290a342eef624401d5c9dbb764a99e4d09a0a
Sep 09 20:03:26.576029 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:03:26.575996    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-ddscs" event={"ID":"fa3e7d87-045f-480a-9f42-28ffb444623e","Type":"ContainerStarted","Data":"b219b7892a53f7d009cbee0d4fb290a342eef624401d5c9dbb764a99e4d09a0a"}
Sep 09 20:03:28.585703 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:03:28.585670    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-ddscs" event={"ID":"fa3e7d87-045f-480a-9f42-28ffb444623e","Type":"ContainerStarted","Data":"96c9eb11bd70f94646328c594392ed5da25aa4b69f74bad5d80e1d0b83f7f8f5"}
Sep 09 20:03:28.586072 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:03:28.585772    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-ddscs"
Sep 09 20:03:28.603211 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:03:28.603164    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-645f9c8fd6-ddscs" podStartSLOduration=1.700037282 podStartE2EDuration="4.603134218s" totalImagesPullingTime="2.903096936s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:03:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:03:28.601993517 +0000 UTC m=+513.274012514" watchObservedRunningTime="2026-09-09 20:03:28.603134218 +0000 UTC m=+513.275153214"
Sep 09 20:03:59.591169 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:03:59.591075    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-ddscs"
Sep 09 20:06:42.609713 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:06:42.609675    2523 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-r95ll"]
Sep 09 20:06:42.611950 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:06:42.611930    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-r95ll"
Sep 09 20:06:42.614571 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:06:42.614541    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Sep 09 20:06:42.614571 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:06:42.614541    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving\""
Sep 09 20:06:42.614722 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:06:42.614580    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-cvxcb\""
Sep 09 20:06:42.621027 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:06:42.621008    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-r95ll"]
Sep 09 20:06:42.739863 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:06:42.739839    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klq9v\" (UniqueName: \"kubernetes.io/projected/e9013ad1-a536-407f-916a-41300babbaf6-kube-api-access-klq9v\") pod \"seaweedfs-tls-serving-67b7477c6d-r95ll\" (UID: \"e9013ad1-a536-407f-916a-41300babbaf6\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-r95ll"
Sep 09 20:06:42.739955 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:06:42.739874    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/e9013ad1-a536-407f-916a-41300babbaf6-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-r95ll\" (UID: \"e9013ad1-a536-407f-916a-41300babbaf6\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-r95ll"
Sep 09 20:06:42.739955 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:06:42.739926    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e9013ad1-a536-407f-916a-41300babbaf6-data\") pod \"seaweedfs-tls-serving-67b7477c6d-r95ll\" (UID: \"e9013ad1-a536-407f-916a-41300babbaf6\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-r95ll"
Sep 09 20:06:42.840764 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:06:42.840740    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/e9013ad1-a536-407f-916a-41300babbaf6-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-r95ll\" (UID: \"e9013ad1-a536-407f-916a-41300babbaf6\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-r95ll"
Sep 09 20:06:42.840869 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:06:42.840776    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e9013ad1-a536-407f-916a-41300babbaf6-data\") pod \"seaweedfs-tls-serving-67b7477c6d-r95ll\" (UID: \"e9013ad1-a536-407f-916a-41300babbaf6\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-r95ll"
Sep 09 20:06:42.840869 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:06:42.840806    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klq9v\" (UniqueName: \"kubernetes.io/projected/e9013ad1-a536-407f-916a-41300babbaf6-kube-api-access-klq9v\") pod \"seaweedfs-tls-serving-67b7477c6d-r95ll\" (UID: \"e9013ad1-a536-407f-916a-41300babbaf6\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-r95ll"
Sep 09 20:06:42.841141 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:06:42.841120    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e9013ad1-a536-407f-916a-41300babbaf6-data\") pod \"seaweedfs-tls-serving-67b7477c6d-r95ll\" (UID: \"e9013ad1-a536-407f-916a-41300babbaf6\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-r95ll"
Sep 09 20:06:42.843206 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:06:42.843189    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/e9013ad1-a536-407f-916a-41300babbaf6-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-r95ll\" (UID: \"e9013ad1-a536-407f-916a-41300babbaf6\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-r95ll"
Sep 09 20:06:42.850009 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:06:42.849981    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klq9v\" (UniqueName: \"kubernetes.io/projected/e9013ad1-a536-407f-916a-41300babbaf6-kube-api-access-klq9v\") pod \"seaweedfs-tls-serving-67b7477c6d-r95ll\" (UID: \"e9013ad1-a536-407f-916a-41300babbaf6\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-r95ll"
Sep 09 20:06:42.921506 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:06:42.921450    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-r95ll"
Sep 09 20:06:43.189426 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:06:43.189399    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-r95ll"]
Sep 09 20:06:43.191462 ip-10-0-132-226 kubenswrapper[2523]: W0909 20:06:43.191419    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9013ad1_a536_407f_916a_41300babbaf6.slice/crio-9cc3e58f17b5eed14ea2e28fa199206c02a3afbd86ca63424d76f604696ddf5d WatchSource:0}: Error finding container 9cc3e58f17b5eed14ea2e28fa199206c02a3afbd86ca63424d76f604696ddf5d: Status 404 returned error can't find the container with id 9cc3e58f17b5eed14ea2e28fa199206c02a3afbd86ca63424d76f604696ddf5d
Sep 09 20:06:43.192779 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:06:43.192761    2523 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:06:43.255577 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:06:43.255554    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-r95ll" event={"ID":"e9013ad1-a536-407f-916a-41300babbaf6","Type":"ContainerStarted","Data":"9cc3e58f17b5eed14ea2e28fa199206c02a3afbd86ca63424d76f604696ddf5d"}
Sep 09 20:06:46.273550 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:06:46.273474    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-r95ll" event={"ID":"e9013ad1-a536-407f-916a-41300babbaf6","Type":"ContainerStarted","Data":"a90a05cbd5578108929f232ce626f90f961b9459dcbe3169de15eb0fe0b381cc"}
Sep 09 20:06:46.293026 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:06:46.292981    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-serving-67b7477c6d-r95ll" podStartSLOduration=1.532914745 podStartE2EDuration="4.292967478s" totalImagesPullingTime="2.760052733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:06:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:06:46.292384788 +0000 UTC m=+710.964403780" watchObservedRunningTime="2026-09-09 20:06:46.292967478 +0000 UTC m=+710.964986474"
Sep 09 20:07:13.527430 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:07:13.527361    2523 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v"]
Sep 09 20:07:13.530528 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:07:13.530511    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v"
Sep 09 20:07:13.537439 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:07:13.537420    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 20:07:13.537557 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:07:13.537545    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Sep 09 20:07:13.537998 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:07:13.537976    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:07:13.538098 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:07:13.538052    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 20:07:13.538185 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:07:13.538171    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-ks7mb\""
Sep 09 20:07:13.538471 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:07:13.538454    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Sep 09 20:07:13.538542 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:07:13.538489    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 20:07:13.539634 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:07:13.539611    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6b1d7ad5-1658-4b84-9722-57b72b8d8d7d-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v\" (UID: \"6b1d7ad5-1658-4b84-9722-57b72b8d8d7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v"
Sep 09 20:07:13.539705 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:07:13.539671    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b1d7ad5-1658-4b84-9722-57b72b8d8d7d-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v\" (UID: \"6b1d7ad5-1658-4b84-9722-57b72b8d8d7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v"
Sep 09 20:07:13.539741 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:07:13.539701    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b1d7ad5-1658-4b84-9722-57b72b8d8d7d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v\" (UID: \"6b1d7ad5-1658-4b84-9722-57b72b8d8d7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v"
Sep 09 20:07:13.539741 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:07:13.539731    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7825\" (UniqueName: \"kubernetes.io/projected/6b1d7ad5-1658-4b84-9722-57b72b8d8d7d-kube-api-access-d7825\") pod \"isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v\" (UID: \"6b1d7ad5-1658-4b84-9722-57b72b8d8d7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v"
Sep 09 20:07:13.539804 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:07:13.539756    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b1d7ad5-1658-4b84-9722-57b72b8d8d7d-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v\" (UID: \"6b1d7ad5-1658-4b84-9722-57b72b8d8d7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v"
Sep 09 20:07:13.571202 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:07:13.571179    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v"]
Sep 09 20:07:13.641012 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:07:13.640989    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b1d7ad5-1658-4b84-9722-57b72b8d8d7d-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v\" (UID: \"6b1d7ad5-1658-4b84-9722-57b72b8d8d7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v"
Sep 09 20:07:13.641120 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:07:13.641027    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b1d7ad5-1658-4b84-9722-57b72b8d8d7d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v\" (UID: \"6b1d7ad5-1658-4b84-9722-57b72b8d8d7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v"
Sep 09 20:07:13.641120 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:07:13.641051    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7825\" (UniqueName: \"kubernetes.io/projected/6b1d7ad5-1658-4b84-9722-57b72b8d8d7d-kube-api-access-d7825\") pod \"isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v\" (UID: \"6b1d7ad5-1658-4b84-9722-57b72b8d8d7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v"
Sep 09 20:07:13.641120 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:07:13.641078    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b1d7ad5-1658-4b84-9722-57b72b8d8d7d-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v\" (UID: \"6b1d7ad5-1658-4b84-9722-57b72b8d8d7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v"
Sep 09 20:07:13.641120 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:07:13.641112    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6b1d7ad5-1658-4b84-9722-57b72b8d8d7d-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v\" (UID: \"6b1d7ad5-1658-4b84-9722-57b72b8d8d7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v"
Sep 09 20:07:13.641625 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:07:13.641601    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b1d7ad5-1658-4b84-9722-57b72b8d8d7d-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v\" (UID: \"6b1d7ad5-1658-4b84-9722-57b72b8d8d7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v"
Sep 09 20:07:13.641821 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:07:13.641805    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6b1d7ad5-1658-4b84-9722-57b72b8d8d7d-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v\" (UID: \"6b1d7ad5-1658-4b84-9722-57b72b8d8d7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v"
Sep 09 20:07:13.641869 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:07:13.641847    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b1d7ad5-1658-4b84-9722-57b72b8d8d7d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v\" (UID: \"6b1d7ad5-1658-4b84-9722-57b72b8d8d7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v"
Sep 09 20:07:13.643475 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:07:13.643455    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b1d7ad5-1658-4b84-9722-57b72b8d8d7d-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v\" (UID: \"6b1d7ad5-1658-4b84-9722-57b72b8d8d7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v"
Sep 09 20:07:13.655923 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:07:13.655901    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7825\" (UniqueName: \"kubernetes.io/projected/6b1d7ad5-1658-4b84-9722-57b72b8d8d7d-kube-api-access-d7825\") pod \"isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v\" (UID: \"6b1d7ad5-1658-4b84-9722-57b72b8d8d7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v"
Sep 09 20:07:13.840255 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:07:13.840227    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v"
Sep 09 20:07:14.114831 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:07:14.114809    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v"]
Sep 09 20:07:14.116741 ip-10-0-132-226 kubenswrapper[2523]: W0909 20:07:14.116714    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b1d7ad5_1658_4b84_9722_57b72b8d8d7d.slice/crio-5a7c3e6de8a8383c0eb884a345a37ebac0b70a9ef9362fd51dfa2abbefc89f8c WatchSource:0}: Error finding container 5a7c3e6de8a8383c0eb884a345a37ebac0b70a9ef9362fd51dfa2abbefc89f8c: Status 404 returned error can't find the container with id 5a7c3e6de8a8383c0eb884a345a37ebac0b70a9ef9362fd51dfa2abbefc89f8c
Sep 09 20:07:14.377499 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:07:14.377431    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v" event={"ID":"6b1d7ad5-1658-4b84-9722-57b72b8d8d7d","Type":"ContainerStarted","Data":"5a7c3e6de8a8383c0eb884a345a37ebac0b70a9ef9362fd51dfa2abbefc89f8c"}
Sep 09 20:07:18.395685 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:07:18.395644    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v" event={"ID":"6b1d7ad5-1658-4b84-9722-57b72b8d8d7d","Type":"ContainerStarted","Data":"f4e452ab5b5c5a107008696a23bc2163eabd7c2e3317d068e4393742c05d6b10"}
Sep 09 20:07:21.407455 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:07:21.407424    2523 generic.go:356] "Generic (PLEG): container finished" podID="6b1d7ad5-1658-4b84-9722-57b72b8d8d7d" containerID="f4e452ab5b5c5a107008696a23bc2163eabd7c2e3317d068e4393742c05d6b10" exitCode=0
Sep 09 20:07:21.407766 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:07:21.407461    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v" event={"ID":"6b1d7ad5-1658-4b84-9722-57b72b8d8d7d","Type":"ContainerDied","Data":"f4e452ab5b5c5a107008696a23bc2163eabd7c2e3317d068e4393742c05d6b10"}
Sep 09 20:07:34.469522 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:07:34.469492    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v" event={"ID":"6b1d7ad5-1658-4b84-9722-57b72b8d8d7d","Type":"ContainerStarted","Data":"41f5096699eda13e285ff79b3319bbd318e510858a6f9349b85b6bb7e72eb1da"}
Sep 09 20:07:38.488628 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:07:38.488595    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v" event={"ID":"6b1d7ad5-1658-4b84-9722-57b72b8d8d7d","Type":"ContainerStarted","Data":"f33d71a9cc7a347a0d8bb94d9f9749856c13a141a59b8b213768d9a64fee31cd"}
Sep 09 20:07:42.505889 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:07:42.505860    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v" event={"ID":"6b1d7ad5-1658-4b84-9722-57b72b8d8d7d","Type":"ContainerStarted","Data":"9b6a12ec705f817ae86ff332ae54552d95cefe75e28d51fe5566d45b05e1b18b"}
Sep 09 20:07:42.506405 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:07:42.506075    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v"
Sep 09 20:07:42.510786 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:07:42.510766    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v"
Sep 09 20:07:42.530164 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:07:42.530109    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v" podStartSLOduration=2.434274618 podStartE2EDuration="29.530097047s" totalImagesPullingTime="23.480586194s" totalInitContainerRuntime="3.615236235s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:07:13 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:07:42.527749718 +0000 UTC m=+767.199768729" watchObservedRunningTime="2026-09-09 20:07:42.530097047 +0000 UTC m=+767.202116084"
Sep 09 20:07:43.510594 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:07:43.510557    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v"
Sep 09 20:07:43.511003 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:07:43.510626    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v"
Sep 09 20:07:43.511003 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:07:43.510868    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v" podUID="6b1d7ad5-1658-4b84-9722-57b72b8d8d7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 09 20:07:43.511353 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:07:43.511319    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v" podUID="6b1d7ad5-1658-4b84-9722-57b72b8d8d7d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:07:44.514439 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:07:44.514393    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v" podUID="6b1d7ad5-1658-4b84-9722-57b72b8d8d7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 09 20:07:44.514879 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:07:44.514753    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v" podUID="6b1d7ad5-1658-4b84-9722-57b72b8d8d7d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:07:54.514897 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:07:54.514856    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v" podUID="6b1d7ad5-1658-4b84-9722-57b72b8d8d7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 09 20:07:54.515447 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:07:54.515328    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v" podUID="6b1d7ad5-1658-4b84-9722-57b72b8d8d7d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:08:04.515124 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:08:04.515082    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v" podUID="6b1d7ad5-1658-4b84-9722-57b72b8d8d7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 09 20:08:04.515599 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:08:04.515577    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v" podUID="6b1d7ad5-1658-4b84-9722-57b72b8d8d7d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:08:14.514884 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:08:14.514846    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v" podUID="6b1d7ad5-1658-4b84-9722-57b72b8d8d7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 09 20:08:14.515428 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:08:14.515404    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v" podUID="6b1d7ad5-1658-4b84-9722-57b72b8d8d7d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:08:24.514341 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:08:24.514293    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v" podUID="6b1d7ad5-1658-4b84-9722-57b72b8d8d7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 09 20:08:24.514779 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:08:24.514644    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v" podUID="6b1d7ad5-1658-4b84-9722-57b72b8d8d7d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:08:34.515176 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:08:34.515128    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v" podUID="6b1d7ad5-1658-4b84-9722-57b72b8d8d7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 09 20:08:34.515698 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:08:34.515611    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v" podUID="6b1d7ad5-1658-4b84-9722-57b72b8d8d7d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:08:44.515169 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:08:44.515121    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v"
Sep 09 20:08:44.515574 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:08:44.515291    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v"
Sep 09 20:08:58.387573 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:08:58.387536    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v"]
Sep 09 20:08:58.387971 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:08:58.387802    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v" podUID="6b1d7ad5-1658-4b84-9722-57b72b8d8d7d" containerName="kserve-container" containerID="cri-o://41f5096699eda13e285ff79b3319bbd318e510858a6f9349b85b6bb7e72eb1da" gracePeriod=30
Sep 09 20:08:58.387971 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:08:58.387823    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v" podUID="6b1d7ad5-1658-4b84-9722-57b72b8d8d7d" containerName="agent" containerID="cri-o://9b6a12ec705f817ae86ff332ae54552d95cefe75e28d51fe5566d45b05e1b18b" gracePeriod=30
Sep 09 20:08:58.387971 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:08:58.387826    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v" podUID="6b1d7ad5-1658-4b84-9722-57b72b8d8d7d" containerName="kube-rbac-proxy" containerID="cri-o://f33d71a9cc7a347a0d8bb94d9f9749856c13a141a59b8b213768d9a64fee31cd" gracePeriod=30
Sep 09 20:08:58.522548 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:08:58.522518    2523 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh"]
Sep 09 20:08:58.525676 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:08:58.525660    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh"
Sep 09 20:08:58.527957 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:08:58.527934    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Sep 09 20:08:58.528128 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:08:58.528108    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Sep 09 20:08:58.536462 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:08:58.536442    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh"]
Sep 09 20:08:58.569325 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:08:58.569299    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fedcf89e-e782-41b3-a761-53d04a626bde-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh\" (UID: \"fedcf89e-e782-41b3-a761-53d04a626bde\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh"
Sep 09 20:08:58.569417 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:08:58.569338    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fedcf89e-e782-41b3-a761-53d04a626bde-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh\" (UID: \"fedcf89e-e782-41b3-a761-53d04a626bde\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh"
Sep 09 20:08:58.569417 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:08:58.569368    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fedcf89e-e782-41b3-a761-53d04a626bde-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh\" (UID: \"fedcf89e-e782-41b3-a761-53d04a626bde\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh"
Sep 09 20:08:58.569502 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:08:58.569423    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fedcf89e-e782-41b3-a761-53d04a626bde-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh\" (UID: \"fedcf89e-e782-41b3-a761-53d04a626bde\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh"
Sep 09 20:08:58.569502 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:08:58.569468    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkpt5\" (UniqueName: \"kubernetes.io/projected/fedcf89e-e782-41b3-a761-53d04a626bde-kube-api-access-tkpt5\") pod \"isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh\" (UID: \"fedcf89e-e782-41b3-a761-53d04a626bde\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh"
Sep 09 20:08:58.669910 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:08:58.669844    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkpt5\" (UniqueName: \"kubernetes.io/projected/fedcf89e-e782-41b3-a761-53d04a626bde-kube-api-access-tkpt5\") pod \"isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh\" (UID: \"fedcf89e-e782-41b3-a761-53d04a626bde\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh"
Sep 09 20:08:58.670011 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:08:58.669905    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fedcf89e-e782-41b3-a761-53d04a626bde-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh\" (UID: \"fedcf89e-e782-41b3-a761-53d04a626bde\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh"
Sep 09 20:08:58.670011 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:08:58.669946    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fedcf89e-e782-41b3-a761-53d04a626bde-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh\" (UID: \"fedcf89e-e782-41b3-a761-53d04a626bde\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh"
Sep 09 20:08:58.670011 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:08:58.669976    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fedcf89e-e782-41b3-a761-53d04a626bde-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh\" (UID: \"fedcf89e-e782-41b3-a761-53d04a626bde\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh"
Sep 09 20:08:58.670011 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:08:58.670004    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fedcf89e-e782-41b3-a761-53d04a626bde-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh\" (UID: \"fedcf89e-e782-41b3-a761-53d04a626bde\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh"
Sep 09 20:08:58.670372 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:08:58.670351    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fedcf89e-e782-41b3-a761-53d04a626bde-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh\" (UID: \"fedcf89e-e782-41b3-a761-53d04a626bde\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh"
Sep 09 20:08:58.670688 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:08:58.670667    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fedcf89e-e782-41b3-a761-53d04a626bde-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh\" (UID: \"fedcf89e-e782-41b3-a761-53d04a626bde\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh"
Sep 09 20:08:58.670755 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:08:58.670671    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fedcf89e-e782-41b3-a761-53d04a626bde-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh\" (UID: \"fedcf89e-e782-41b3-a761-53d04a626bde\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh"
Sep 09 20:08:58.672489 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:08:58.672471    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fedcf89e-e782-41b3-a761-53d04a626bde-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh\" (UID: \"fedcf89e-e782-41b3-a761-53d04a626bde\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh"
Sep 09 20:08:58.679669 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:08:58.679646    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkpt5\" (UniqueName: \"kubernetes.io/projected/fedcf89e-e782-41b3-a761-53d04a626bde-kube-api-access-tkpt5\") pod \"isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh\" (UID: \"fedcf89e-e782-41b3-a761-53d04a626bde\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh"
Sep 09 20:08:58.793031 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:08:58.793007    2523 generic.go:356] "Generic (PLEG): container finished" podID="6b1d7ad5-1658-4b84-9722-57b72b8d8d7d" containerID="f33d71a9cc7a347a0d8bb94d9f9749856c13a141a59b8b213768d9a64fee31cd" exitCode=2
Sep 09 20:08:58.793120 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:08:58.793082    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v" event={"ID":"6b1d7ad5-1658-4b84-9722-57b72b8d8d7d","Type":"ContainerDied","Data":"f33d71a9cc7a347a0d8bb94d9f9749856c13a141a59b8b213768d9a64fee31cd"}
Sep 09 20:08:58.835607 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:08:58.835585    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh"
Sep 09 20:08:59.109300 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:08:59.109274    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh"]
Sep 09 20:08:59.111787 ip-10-0-132-226 kubenswrapper[2523]: W0909 20:08:59.111762    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfedcf89e_e782_41b3_a761_53d04a626bde.slice/crio-4c2f22e01b84ccc9a4cac831e90869800637a864fd154f43c7427b60497a2b18 WatchSource:0}: Error finding container 4c2f22e01b84ccc9a4cac831e90869800637a864fd154f43c7427b60497a2b18: Status 404 returned error can't find the container with id 4c2f22e01b84ccc9a4cac831e90869800637a864fd154f43c7427b60497a2b18
Sep 09 20:08:59.798820 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:08:59.798785    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh" event={"ID":"fedcf89e-e782-41b3-a761-53d04a626bde","Type":"ContainerStarted","Data":"fe10b4b532726e682c0c7776bbe3df24c4eba9cdb5df76861f6b4853ce0573cf"}
Sep 09 20:08:59.799196 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:08:59.798843    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh" event={"ID":"fedcf89e-e782-41b3-a761-53d04a626bde","Type":"ContainerStarted","Data":"4c2f22e01b84ccc9a4cac831e90869800637a864fd154f43c7427b60497a2b18"}
Sep 09 20:09:02.506502 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:02.506428    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v" podUID="6b1d7ad5-1658-4b84-9722-57b72b8d8d7d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
Sep 09 20:09:02.812365 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:02.812305    2523 generic.go:356] "Generic (PLEG): container finished" podID="6b1d7ad5-1658-4b84-9722-57b72b8d8d7d" containerID="41f5096699eda13e285ff79b3319bbd318e510858a6f9349b85b6bb7e72eb1da" exitCode=0
Sep 09 20:09:02.812479 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:02.812388    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v" event={"ID":"6b1d7ad5-1658-4b84-9722-57b72b8d8d7d","Type":"ContainerDied","Data":"41f5096699eda13e285ff79b3319bbd318e510858a6f9349b85b6bb7e72eb1da"}
Sep 09 20:09:02.813881 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:02.813861    2523 generic.go:356] "Generic (PLEG): container finished" podID="fedcf89e-e782-41b3-a761-53d04a626bde" containerID="fe10b4b532726e682c0c7776bbe3df24c4eba9cdb5df76861f6b4853ce0573cf" exitCode=0
Sep 09 20:09:02.813983 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:02.813901    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh" event={"ID":"fedcf89e-e782-41b3-a761-53d04a626bde","Type":"ContainerDied","Data":"fe10b4b532726e682c0c7776bbe3df24c4eba9cdb5df76861f6b4853ce0573cf"}
Sep 09 20:09:03.819601 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:03.819563    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh" event={"ID":"fedcf89e-e782-41b3-a761-53d04a626bde","Type":"ContainerStarted","Data":"ab3149e87747fa2a5e67a43ccfbb0355b5226e2bd978b20773354d7d54b81660"}
Sep 09 20:09:03.819963 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:03.819634    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh" event={"ID":"fedcf89e-e782-41b3-a761-53d04a626bde","Type":"ContainerStarted","Data":"8c43ea55692580bac4f5454ee4676089af0138213cb68beb77e5aed78514975e"}
Sep 09 20:09:03.819963 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:03.819660    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh" event={"ID":"fedcf89e-e782-41b3-a761-53d04a626bde","Type":"ContainerStarted","Data":"db4ef39233dcccbaa1fb113560672d0b444aa91b16148fec6da8dd4826934d56"}
Sep 09 20:09:03.819963 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:03.819933    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh"
Sep 09 20:09:03.820105 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:03.820083    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh"
Sep 09 20:09:03.821310 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:03.821285    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh" podUID="fedcf89e-e782-41b3-a761-53d04a626bde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:5000: connect: connection refused"
Sep 09 20:09:03.841601 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:03.841559    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh" podStartSLOduration=2.339191019 podStartE2EDuration="5.841547038s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.502356019s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:08:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:09:03.839979935 +0000 UTC m=+848.511998943" watchObservedRunningTime="2026-09-09 20:09:03.841547038 +0000 UTC m=+848.513566075"
Sep 09 20:09:04.515100 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:04.515056    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v" podUID="6b1d7ad5-1658-4b84-9722-57b72b8d8d7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 09 20:09:04.515430 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:04.515406    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v" podUID="6b1d7ad5-1658-4b84-9722-57b72b8d8d7d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:09:04.823904 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:04.823860    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh" podUID="fedcf89e-e782-41b3-a761-53d04a626bde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:5000: connect: connection refused"
Sep 09 20:09:04.824334 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:04.823874    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh"
Sep 09 20:09:04.825018 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:04.824984    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh" podUID="fedcf89e-e782-41b3-a761-53d04a626bde" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:09:05.827664 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:05.827624    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh" podUID="fedcf89e-e782-41b3-a761-53d04a626bde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:5000: connect: connection refused"
Sep 09 20:09:05.828338 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:05.828010    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh" podUID="fedcf89e-e782-41b3-a761-53d04a626bde" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:09:07.506409 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:07.506367    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v" podUID="6b1d7ad5-1658-4b84-9722-57b72b8d8d7d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
Sep 09 20:09:10.832473 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:10.832446    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh"
Sep 09 20:09:10.833111 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:10.833082    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh" podUID="fedcf89e-e782-41b3-a761-53d04a626bde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:5000: connect: connection refused"
Sep 09 20:09:10.833407 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:10.833382    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh" podUID="fedcf89e-e782-41b3-a761-53d04a626bde" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:09:12.507353 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:12.507306    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v" podUID="6b1d7ad5-1658-4b84-9722-57b72b8d8d7d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
Sep 09 20:09:12.507699 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:12.507455    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v"
Sep 09 20:09:14.514724 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:14.514686    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v" podUID="6b1d7ad5-1658-4b84-9722-57b72b8d8d7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 09 20:09:14.515452 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:14.515427    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v" podUID="6b1d7ad5-1658-4b84-9722-57b72b8d8d7d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:09:17.507263 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:17.507220    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v" podUID="6b1d7ad5-1658-4b84-9722-57b72b8d8d7d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
Sep 09 20:09:20.833009 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:20.832964    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh" podUID="fedcf89e-e782-41b3-a761-53d04a626bde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:5000: connect: connection refused"
Sep 09 20:09:20.833442 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:20.833371    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh" podUID="fedcf89e-e782-41b3-a761-53d04a626bde" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:09:22.506536 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:22.506498    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v" podUID="6b1d7ad5-1658-4b84-9722-57b72b8d8d7d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
Sep 09 20:09:24.515034 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:24.514990    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v" podUID="6b1d7ad5-1658-4b84-9722-57b72b8d8d7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 09 20:09:24.515475 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:24.515172    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v"
Sep 09 20:09:24.515475 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:24.515416    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v" podUID="6b1d7ad5-1658-4b84-9722-57b72b8d8d7d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:09:24.515561 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:24.515510    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v"
Sep 09 20:09:27.507096 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:27.507054    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v" podUID="6b1d7ad5-1658-4b84-9722-57b72b8d8d7d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
Sep 09 20:09:28.526354 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:28.526329    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v"
Sep 09 20:09:28.578630 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:28.578604    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d7825\" (UniqueName: \"kubernetes.io/projected/6b1d7ad5-1658-4b84-9722-57b72b8d8d7d-kube-api-access-d7825\") pod \"6b1d7ad5-1658-4b84-9722-57b72b8d8d7d\" (UID: \"6b1d7ad5-1658-4b84-9722-57b72b8d8d7d\") "
Sep 09 20:09:28.578754 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:28.578638    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b1d7ad5-1658-4b84-9722-57b72b8d8d7d-proxy-tls\") pod \"6b1d7ad5-1658-4b84-9722-57b72b8d8d7d\" (UID: \"6b1d7ad5-1658-4b84-9722-57b72b8d8d7d\") "
Sep 09 20:09:28.578798 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:28.578749    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b1d7ad5-1658-4b84-9722-57b72b8d8d7d-kube-rbac-proxy-sar-config\") pod \"6b1d7ad5-1658-4b84-9722-57b72b8d8d7d\" (UID: \"6b1d7ad5-1658-4b84-9722-57b72b8d8d7d\") "
Sep 09 20:09:28.578798 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:28.578783    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6b1d7ad5-1658-4b84-9722-57b72b8d8d7d-cabundle-cert\") pod \"6b1d7ad5-1658-4b84-9722-57b72b8d8d7d\" (UID: \"6b1d7ad5-1658-4b84-9722-57b72b8d8d7d\") "
Sep 09 20:09:28.578894 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:28.578813    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b1d7ad5-1658-4b84-9722-57b72b8d8d7d-kserve-provision-location\") pod \"6b1d7ad5-1658-4b84-9722-57b72b8d8d7d\" (UID: \"6b1d7ad5-1658-4b84-9722-57b72b8d8d7d\") "
Sep 09 20:09:28.579162 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:28.579120    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b1d7ad5-1658-4b84-9722-57b72b8d8d7d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6b1d7ad5-1658-4b84-9722-57b72b8d8d7d" (UID: "6b1d7ad5-1658-4b84-9722-57b72b8d8d7d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:09:28.579267 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:28.579185    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b1d7ad5-1658-4b84-9722-57b72b8d8d7d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6b1d7ad5-1658-4b84-9722-57b72b8d8d7d" (UID: "6b1d7ad5-1658-4b84-9722-57b72b8d8d7d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:09:28.579267 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:28.579217    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b1d7ad5-1658-4b84-9722-57b72b8d8d7d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6b1d7ad5-1658-4b84-9722-57b72b8d8d7d" (UID: "6b1d7ad5-1658-4b84-9722-57b72b8d8d7d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:09:28.580716 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:28.580693    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b1d7ad5-1658-4b84-9722-57b72b8d8d7d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6b1d7ad5-1658-4b84-9722-57b72b8d8d7d" (UID: "6b1d7ad5-1658-4b84-9722-57b72b8d8d7d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:09:28.580903 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:28.580885    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b1d7ad5-1658-4b84-9722-57b72b8d8d7d-kube-api-access-d7825" (OuterVolumeSpecName: "kube-api-access-d7825") pod "6b1d7ad5-1658-4b84-9722-57b72b8d8d7d" (UID: "6b1d7ad5-1658-4b84-9722-57b72b8d8d7d"). InnerVolumeSpecName "kube-api-access-d7825". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:09:28.679650 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:28.679584    2523 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7825\" (UniqueName: \"kubernetes.io/projected/6b1d7ad5-1658-4b84-9722-57b72b8d8d7d-kube-api-access-d7825\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:09:28.679650 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:28.679610    2523 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b1d7ad5-1658-4b84-9722-57b72b8d8d7d-proxy-tls\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:09:28.679650 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:28.679619    2523 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b1d7ad5-1658-4b84-9722-57b72b8d8d7d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:09:28.679650 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:28.679627    2523 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6b1d7ad5-1658-4b84-9722-57b72b8d8d7d-cabundle-cert\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:09:28.679650 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:28.679635    2523 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b1d7ad5-1658-4b84-9722-57b72b8d8d7d-kserve-provision-location\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:09:28.917116 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:28.917085    2523 generic.go:356] "Generic (PLEG): container finished" podID="6b1d7ad5-1658-4b84-9722-57b72b8d8d7d" containerID="9b6a12ec705f817ae86ff332ae54552d95cefe75e28d51fe5566d45b05e1b18b" exitCode=0
Sep 09 20:09:28.917301 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:28.917145    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v"
Sep 09 20:09:28.917301 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:28.917189    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v" event={"ID":"6b1d7ad5-1658-4b84-9722-57b72b8d8d7d","Type":"ContainerDied","Data":"9b6a12ec705f817ae86ff332ae54552d95cefe75e28d51fe5566d45b05e1b18b"}
Sep 09 20:09:28.917301 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:28.917270    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v" event={"ID":"6b1d7ad5-1658-4b84-9722-57b72b8d8d7d","Type":"ContainerDied","Data":"5a7c3e6de8a8383c0eb884a345a37ebac0b70a9ef9362fd51dfa2abbefc89f8c"}
Sep 09 20:09:28.917461 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:28.917313    2523 scope.go:117] "RemoveContainer" containerID="9b6a12ec705f817ae86ff332ae54552d95cefe75e28d51fe5566d45b05e1b18b"
Sep 09 20:09:28.925834 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:28.925818    2523 scope.go:117] "RemoveContainer" containerID="f33d71a9cc7a347a0d8bb94d9f9749856c13a141a59b8b213768d9a64fee31cd"
Sep 09 20:09:28.933404 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:28.933384    2523 scope.go:117] "RemoveContainer" containerID="41f5096699eda13e285ff79b3319bbd318e510858a6f9349b85b6bb7e72eb1da"
Sep 09 20:09:28.939384 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:28.939360    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v"]
Sep 09 20:09:28.940856 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:28.940839    2523 scope.go:117] "RemoveContainer" containerID="f4e452ab5b5c5a107008696a23bc2163eabd7c2e3317d068e4393742c05d6b10"
Sep 09 20:09:28.943481 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:28.943455    2523 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6cf87485dc-9wg2v"]
Sep 09 20:09:28.947766 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:28.947751    2523 scope.go:117] "RemoveContainer" containerID="9b6a12ec705f817ae86ff332ae54552d95cefe75e28d51fe5566d45b05e1b18b"
Sep 09 20:09:28.948062 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:09:28.948040    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b6a12ec705f817ae86ff332ae54552d95cefe75e28d51fe5566d45b05e1b18b\": container with ID starting with 9b6a12ec705f817ae86ff332ae54552d95cefe75e28d51fe5566d45b05e1b18b not found: ID does not exist" containerID="9b6a12ec705f817ae86ff332ae54552d95cefe75e28d51fe5566d45b05e1b18b"
Sep 09 20:09:28.948106 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:28.948072    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b6a12ec705f817ae86ff332ae54552d95cefe75e28d51fe5566d45b05e1b18b"} err="failed to get container status \"9b6a12ec705f817ae86ff332ae54552d95cefe75e28d51fe5566d45b05e1b18b\": rpc error: code = NotFound desc = could not find container \"9b6a12ec705f817ae86ff332ae54552d95cefe75e28d51fe5566d45b05e1b18b\": container with ID starting with 9b6a12ec705f817ae86ff332ae54552d95cefe75e28d51fe5566d45b05e1b18b not found: ID does not exist"
Sep 09 20:09:28.948106 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:28.948088    2523 scope.go:117] "RemoveContainer" containerID="f33d71a9cc7a347a0d8bb94d9f9749856c13a141a59b8b213768d9a64fee31cd"
Sep 09 20:09:28.948377 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:09:28.948354    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f33d71a9cc7a347a0d8bb94d9f9749856c13a141a59b8b213768d9a64fee31cd\": container with ID starting with f33d71a9cc7a347a0d8bb94d9f9749856c13a141a59b8b213768d9a64fee31cd not found: ID does not exist" containerID="f33d71a9cc7a347a0d8bb94d9f9749856c13a141a59b8b213768d9a64fee31cd"
Sep 09 20:09:28.948434 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:28.948385    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f33d71a9cc7a347a0d8bb94d9f9749856c13a141a59b8b213768d9a64fee31cd"} err="failed to get container status \"f33d71a9cc7a347a0d8bb94d9f9749856c13a141a59b8b213768d9a64fee31cd\": rpc error: code = NotFound desc = could not find container \"f33d71a9cc7a347a0d8bb94d9f9749856c13a141a59b8b213768d9a64fee31cd\": container with ID starting with f33d71a9cc7a347a0d8bb94d9f9749856c13a141a59b8b213768d9a64fee31cd not found: ID does not exist"
Sep 09 20:09:28.948434 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:28.948406    2523 scope.go:117] "RemoveContainer" containerID="41f5096699eda13e285ff79b3319bbd318e510858a6f9349b85b6bb7e72eb1da"
Sep 09 20:09:28.948690 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:09:28.948671    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41f5096699eda13e285ff79b3319bbd318e510858a6f9349b85b6bb7e72eb1da\": container with ID starting with 41f5096699eda13e285ff79b3319bbd318e510858a6f9349b85b6bb7e72eb1da not found: ID does not exist" containerID="41f5096699eda13e285ff79b3319bbd318e510858a6f9349b85b6bb7e72eb1da"
Sep 09 20:09:28.948764 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:28.948697    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41f5096699eda13e285ff79b3319bbd318e510858a6f9349b85b6bb7e72eb1da"} err="failed to get container status \"41f5096699eda13e285ff79b3319bbd318e510858a6f9349b85b6bb7e72eb1da\": rpc error: code = NotFound desc = could not find container \"41f5096699eda13e285ff79b3319bbd318e510858a6f9349b85b6bb7e72eb1da\": container with ID starting with 41f5096699eda13e285ff79b3319bbd318e510858a6f9349b85b6bb7e72eb1da not found: ID does not exist"
Sep 09 20:09:28.948764 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:28.948717    2523 scope.go:117] "RemoveContainer" containerID="f4e452ab5b5c5a107008696a23bc2163eabd7c2e3317d068e4393742c05d6b10"
Sep 09 20:09:28.949017 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:09:28.949000    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4e452ab5b5c5a107008696a23bc2163eabd7c2e3317d068e4393742c05d6b10\": container with ID starting with f4e452ab5b5c5a107008696a23bc2163eabd7c2e3317d068e4393742c05d6b10 not found: ID does not exist" containerID="f4e452ab5b5c5a107008696a23bc2163eabd7c2e3317d068e4393742c05d6b10"
Sep 09 20:09:28.949063 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:28.949022    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4e452ab5b5c5a107008696a23bc2163eabd7c2e3317d068e4393742c05d6b10"} err="failed to get container status \"f4e452ab5b5c5a107008696a23bc2163eabd7c2e3317d068e4393742c05d6b10\": rpc error: code = NotFound desc = could not find container \"f4e452ab5b5c5a107008696a23bc2163eabd7c2e3317d068e4393742c05d6b10\": container with ID starting with f4e452ab5b5c5a107008696a23bc2163eabd7c2e3317d068e4393742c05d6b10 not found: ID does not exist"
Sep 09 20:09:29.807569 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:29.807541    2523 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b1d7ad5-1658-4b84-9722-57b72b8d8d7d" path="/var/lib/kubelet/pods/6b1d7ad5-1658-4b84-9722-57b72b8d8d7d/volumes"
Sep 09 20:09:30.833920 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:30.833886    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh" podUID="fedcf89e-e782-41b3-a761-53d04a626bde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:5000: connect: connection refused"
Sep 09 20:09:30.834371 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:30.834347    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh" podUID="fedcf89e-e782-41b3-a761-53d04a626bde" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:09:40.833657 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:40.833619    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh" podUID="fedcf89e-e782-41b3-a761-53d04a626bde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:5000: connect: connection refused"
Sep 09 20:09:40.834193 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:40.834174    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh" podUID="fedcf89e-e782-41b3-a761-53d04a626bde" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:09:50.833839 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:50.833757    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh" podUID="fedcf89e-e782-41b3-a761-53d04a626bde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:5000: connect: connection refused"
Sep 09 20:09:50.834247 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:09:50.834134    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh" podUID="fedcf89e-e782-41b3-a761-53d04a626bde" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:10:00.833910 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:00.833865    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh" podUID="fedcf89e-e782-41b3-a761-53d04a626bde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:5000: connect: connection refused"
Sep 09 20:10:00.834400 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:00.834377    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh" podUID="fedcf89e-e782-41b3-a761-53d04a626bde" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:10:10.833968 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:10.833935    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh"
Sep 09 20:10:10.834463 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:10.834441    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh"
Sep 09 20:10:23.586212 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:23.586174    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh"]
Sep 09 20:10:23.586715 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:23.586431    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh" podUID="fedcf89e-e782-41b3-a761-53d04a626bde" containerName="kserve-container" containerID="cri-o://db4ef39233dcccbaa1fb113560672d0b444aa91b16148fec6da8dd4826934d56" gracePeriod=30
Sep 09 20:10:23.586715 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:23.586464    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh" podUID="fedcf89e-e782-41b3-a761-53d04a626bde" containerName="agent" containerID="cri-o://ab3149e87747fa2a5e67a43ccfbb0355b5226e2bd978b20773354d7d54b81660" gracePeriod=30
Sep 09 20:10:23.586715 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:23.586513    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh" podUID="fedcf89e-e782-41b3-a761-53d04a626bde" containerName="kube-rbac-proxy" containerID="cri-o://8c43ea55692580bac4f5454ee4676089af0138213cb68beb77e5aed78514975e" gracePeriod=30
Sep 09 20:10:24.116490 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:24.116463    2523 generic.go:356] "Generic (PLEG): container finished" podID="fedcf89e-e782-41b3-a761-53d04a626bde" containerID="8c43ea55692580bac4f5454ee4676089af0138213cb68beb77e5aed78514975e" exitCode=2
Sep 09 20:10:24.116648 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:24.116511    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh" event={"ID":"fedcf89e-e782-41b3-a761-53d04a626bde","Type":"ContainerDied","Data":"8c43ea55692580bac4f5454ee4676089af0138213cb68beb77e5aed78514975e"}
Sep 09 20:10:25.828255 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:25.828214    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh" podUID="fedcf89e-e782-41b3-a761-53d04a626bde" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
Sep 09 20:10:28.135169 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:28.135118    2523 generic.go:356] "Generic (PLEG): container finished" podID="fedcf89e-e782-41b3-a761-53d04a626bde" containerID="db4ef39233dcccbaa1fb113560672d0b444aa91b16148fec6da8dd4826934d56" exitCode=0
Sep 09 20:10:28.135524 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:28.135180    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh" event={"ID":"fedcf89e-e782-41b3-a761-53d04a626bde","Type":"ContainerDied","Data":"db4ef39233dcccbaa1fb113560672d0b444aa91b16148fec6da8dd4826934d56"}
Sep 09 20:10:30.828454 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:30.828418    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh" podUID="fedcf89e-e782-41b3-a761-53d04a626bde" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
Sep 09 20:10:30.834067 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:30.834039    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh" podUID="fedcf89e-e782-41b3-a761-53d04a626bde" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:10:30.834980 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:30.834956    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh" podUID="fedcf89e-e782-41b3-a761-53d04a626bde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:5000: connect: connection refused"
Sep 09 20:10:33.772472 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:33.772437    2523 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98"]
Sep 09 20:10:33.772816 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:33.772727    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b1d7ad5-1658-4b84-9722-57b72b8d8d7d" containerName="storage-initializer"
Sep 09 20:10:33.772816 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:33.772739    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b1d7ad5-1658-4b84-9722-57b72b8d8d7d" containerName="storage-initializer"
Sep 09 20:10:33.772816 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:33.772757    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b1d7ad5-1658-4b84-9722-57b72b8d8d7d" containerName="kserve-container"
Sep 09 20:10:33.772816 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:33.772763    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b1d7ad5-1658-4b84-9722-57b72b8d8d7d" containerName="kserve-container"
Sep 09 20:10:33.772816 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:33.772771    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b1d7ad5-1658-4b84-9722-57b72b8d8d7d" containerName="kube-rbac-proxy"
Sep 09 20:10:33.772816 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:33.772777    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b1d7ad5-1658-4b84-9722-57b72b8d8d7d" containerName="kube-rbac-proxy"
Sep 09 20:10:33.772816 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:33.772784    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b1d7ad5-1658-4b84-9722-57b72b8d8d7d" containerName="agent"
Sep 09 20:10:33.772816 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:33.772789    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b1d7ad5-1658-4b84-9722-57b72b8d8d7d" containerName="agent"
Sep 09 20:10:33.773056 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:33.772833    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b1d7ad5-1658-4b84-9722-57b72b8d8d7d" containerName="kube-rbac-proxy"
Sep 09 20:10:33.773056 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:33.772840    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b1d7ad5-1658-4b84-9722-57b72b8d8d7d" containerName="agent"
Sep 09 20:10:33.773056 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:33.772847    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b1d7ad5-1658-4b84-9722-57b72b8d8d7d" containerName="kserve-container"
Sep 09 20:10:33.776114 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:33.776097    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98"
Sep 09 20:10:33.779008 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:33.778988    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Sep 09 20:10:33.779312 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:33.779296    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Sep 09 20:10:33.790404 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:33.790385    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98"]
Sep 09 20:10:33.913190 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:33.913141    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c12080b6-da22-4d43-abcf-eb5ae28b652b-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-85d5cb84f6-x9r98\" (UID: \"c12080b6-da22-4d43-abcf-eb5ae28b652b\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98"
Sep 09 20:10:33.913295 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:33.913202    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c12080b6-da22-4d43-abcf-eb5ae28b652b-cabundle-cert\") pod \"isvc-logger-predictor-85d5cb84f6-x9r98\" (UID: \"c12080b6-da22-4d43-abcf-eb5ae28b652b\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98"
Sep 09 20:10:33.913295 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:33.913235    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rq4w\" (UniqueName: \"kubernetes.io/projected/c12080b6-da22-4d43-abcf-eb5ae28b652b-kube-api-access-9rq4w\") pod \"isvc-logger-predictor-85d5cb84f6-x9r98\" (UID: \"c12080b6-da22-4d43-abcf-eb5ae28b652b\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98"
Sep 09 20:10:33.913295 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:33.913254    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c12080b6-da22-4d43-abcf-eb5ae28b652b-proxy-tls\") pod \"isvc-logger-predictor-85d5cb84f6-x9r98\" (UID: \"c12080b6-da22-4d43-abcf-eb5ae28b652b\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98"
Sep 09 20:10:33.913295 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:33.913279    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c12080b6-da22-4d43-abcf-eb5ae28b652b-kserve-provision-location\") pod \"isvc-logger-predictor-85d5cb84f6-x9r98\" (UID: \"c12080b6-da22-4d43-abcf-eb5ae28b652b\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98"
Sep 09 20:10:34.013954 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:34.013932    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c12080b6-da22-4d43-abcf-eb5ae28b652b-cabundle-cert\") pod \"isvc-logger-predictor-85d5cb84f6-x9r98\" (UID: \"c12080b6-da22-4d43-abcf-eb5ae28b652b\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98"
Sep 09 20:10:34.014083 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:34.013965    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rq4w\" (UniqueName: \"kubernetes.io/projected/c12080b6-da22-4d43-abcf-eb5ae28b652b-kube-api-access-9rq4w\") pod \"isvc-logger-predictor-85d5cb84f6-x9r98\" (UID: \"c12080b6-da22-4d43-abcf-eb5ae28b652b\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98"
Sep 09 20:10:34.014083 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:34.013985    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c12080b6-da22-4d43-abcf-eb5ae28b652b-proxy-tls\") pod \"isvc-logger-predictor-85d5cb84f6-x9r98\" (UID: \"c12080b6-da22-4d43-abcf-eb5ae28b652b\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98"
Sep 09 20:10:34.014083 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:34.014003    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c12080b6-da22-4d43-abcf-eb5ae28b652b-kserve-provision-location\") pod \"isvc-logger-predictor-85d5cb84f6-x9r98\" (UID: \"c12080b6-da22-4d43-abcf-eb5ae28b652b\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98"
Sep 09 20:10:34.014083 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:34.014038    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c12080b6-da22-4d43-abcf-eb5ae28b652b-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-85d5cb84f6-x9r98\" (UID: \"c12080b6-da22-4d43-abcf-eb5ae28b652b\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98"
Sep 09 20:10:34.014443 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:34.014421    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c12080b6-da22-4d43-abcf-eb5ae28b652b-kserve-provision-location\") pod \"isvc-logger-predictor-85d5cb84f6-x9r98\" (UID: \"c12080b6-da22-4d43-abcf-eb5ae28b652b\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98"
Sep 09 20:10:34.014631 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:34.014614    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c12080b6-da22-4d43-abcf-eb5ae28b652b-cabundle-cert\") pod \"isvc-logger-predictor-85d5cb84f6-x9r98\" (UID: \"c12080b6-da22-4d43-abcf-eb5ae28b652b\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98"
Sep 09 20:10:34.014707 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:34.014690    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c12080b6-da22-4d43-abcf-eb5ae28b652b-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-85d5cb84f6-x9r98\" (UID: \"c12080b6-da22-4d43-abcf-eb5ae28b652b\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98"
Sep 09 20:10:34.016427 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:34.016410    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c12080b6-da22-4d43-abcf-eb5ae28b652b-proxy-tls\") pod \"isvc-logger-predictor-85d5cb84f6-x9r98\" (UID: \"c12080b6-da22-4d43-abcf-eb5ae28b652b\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98"
Sep 09 20:10:34.023008 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:34.022951    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rq4w\" (UniqueName: \"kubernetes.io/projected/c12080b6-da22-4d43-abcf-eb5ae28b652b-kube-api-access-9rq4w\") pod \"isvc-logger-predictor-85d5cb84f6-x9r98\" (UID: \"c12080b6-da22-4d43-abcf-eb5ae28b652b\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98"
Sep 09 20:10:34.085926 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:34.085896    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98"
Sep 09 20:10:34.370479 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:34.370437    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98"]
Sep 09 20:10:34.372492 ip-10-0-132-226 kubenswrapper[2523]: W0909 20:10:34.372461    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc12080b6_da22_4d43_abcf_eb5ae28b652b.slice/crio-22bfcdfa73e5640bd4bcacc544ab61a18840326e729e7e1847236710f15c7114 WatchSource:0}: Error finding container 22bfcdfa73e5640bd4bcacc544ab61a18840326e729e7e1847236710f15c7114: Status 404 returned error can't find the container with id 22bfcdfa73e5640bd4bcacc544ab61a18840326e729e7e1847236710f15c7114
Sep 09 20:10:35.164546 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:35.164512    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98" event={"ID":"c12080b6-da22-4d43-abcf-eb5ae28b652b","Type":"ContainerStarted","Data":"6c15105afe67d0cf891d3fae65dc7dd9b710c6276b1ab9d6f7857db8c1c53b36"}
Sep 09 20:10:35.164880 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:35.164580    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98" event={"ID":"c12080b6-da22-4d43-abcf-eb5ae28b652b","Type":"ContainerStarted","Data":"22bfcdfa73e5640bd4bcacc544ab61a18840326e729e7e1847236710f15c7114"}
Sep 09 20:10:35.828377 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:35.828341    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh" podUID="fedcf89e-e782-41b3-a761-53d04a626bde" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
Sep 09 20:10:35.828552 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:35.828432    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh"
Sep 09 20:10:39.181771 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:39.181736    2523 generic.go:356] "Generic (PLEG): container finished" podID="c12080b6-da22-4d43-abcf-eb5ae28b652b" containerID="6c15105afe67d0cf891d3fae65dc7dd9b710c6276b1ab9d6f7857db8c1c53b36" exitCode=0
Sep 09 20:10:39.182231 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:39.181809    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98" event={"ID":"c12080b6-da22-4d43-abcf-eb5ae28b652b","Type":"ContainerDied","Data":"6c15105afe67d0cf891d3fae65dc7dd9b710c6276b1ab9d6f7857db8c1c53b36"}
Sep 09 20:10:40.188136 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:40.188102    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98" event={"ID":"c12080b6-da22-4d43-abcf-eb5ae28b652b","Type":"ContainerStarted","Data":"453c01475dd497f9b4ebc9aded25e1d5706f16734b16da134bd3d085da6804d3"}
Sep 09 20:10:40.188597 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:40.188197    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98" event={"ID":"c12080b6-da22-4d43-abcf-eb5ae28b652b","Type":"ContainerStarted","Data":"5efe1a857554f5dd51b28408e4c2d80aa06347b8df85f28ad4e9358093c3f0e6"}
Sep 09 20:10:40.188597 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:40.188241    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98" event={"ID":"c12080b6-da22-4d43-abcf-eb5ae28b652b","Type":"ContainerStarted","Data":"092ea9e1bcbb3823e077776c606f56e3ddcc5f32e3e48341ae37378d297afdf3"}
Sep 09 20:10:40.188597 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:40.188479    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98"
Sep 09 20:10:40.211503 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:40.211456    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98" podStartSLOduration=3.457894512 podStartE2EDuration="7.211441941s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.753547429s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:10:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:10:40.209769063 +0000 UTC m=+944.881788063" watchObservedRunningTime="2026-09-09 20:10:40.211441941 +0000 UTC m=+944.883460932"
Sep 09 20:10:40.828693 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:40.828658    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh" podUID="fedcf89e-e782-41b3-a761-53d04a626bde" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
Sep 09 20:10:40.833327 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:40.833302    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh" podUID="fedcf89e-e782-41b3-a761-53d04a626bde" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:10:40.835509 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:40.835487    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh" podUID="fedcf89e-e782-41b3-a761-53d04a626bde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:5000: connect: connection refused"
Sep 09 20:10:41.192132 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:41.192057    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98"
Sep 09 20:10:41.192132 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:41.192132    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98"
Sep 09 20:10:41.193511 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:41.193482    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98" podUID="c12080b6-da22-4d43-abcf-eb5ae28b652b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 09 20:10:41.194211 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:41.194189    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98" podUID="c12080b6-da22-4d43-abcf-eb5ae28b652b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:10:42.195211 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:42.195170    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98" podUID="c12080b6-da22-4d43-abcf-eb5ae28b652b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 09 20:10:42.195726 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:42.195706    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98" podUID="c12080b6-da22-4d43-abcf-eb5ae28b652b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:10:45.827947 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:45.827909    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh" podUID="fedcf89e-e782-41b3-a761-53d04a626bde" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
Sep 09 20:10:47.199622 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:47.199593    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98"
Sep 09 20:10:47.200336 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:47.200297    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98" podUID="c12080b6-da22-4d43-abcf-eb5ae28b652b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 09 20:10:47.200578 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:47.200552    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98" podUID="c12080b6-da22-4d43-abcf-eb5ae28b652b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:10:50.827952 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:50.827912    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh" podUID="fedcf89e-e782-41b3-a761-53d04a626bde" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
Sep 09 20:10:50.833576 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:50.833552    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh" podUID="fedcf89e-e782-41b3-a761-53d04a626bde" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:10:50.833668 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:50.833656    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh"
Sep 09 20:10:50.834784 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:50.834760    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh" podUID="fedcf89e-e782-41b3-a761-53d04a626bde" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:5000: connect: connection refused"
Sep 09 20:10:50.834891 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:50.834853    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh"
Sep 09 20:10:53.734803 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:53.734779    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh"
Sep 09 20:10:53.845005 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:53.844946    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fedcf89e-e782-41b3-a761-53d04a626bde-cabundle-cert\") pod \"fedcf89e-e782-41b3-a761-53d04a626bde\" (UID: \"fedcf89e-e782-41b3-a761-53d04a626bde\") "
Sep 09 20:10:53.845005 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:53.844976    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tkpt5\" (UniqueName: \"kubernetes.io/projected/fedcf89e-e782-41b3-a761-53d04a626bde-kube-api-access-tkpt5\") pod \"fedcf89e-e782-41b3-a761-53d04a626bde\" (UID: \"fedcf89e-e782-41b3-a761-53d04a626bde\") "
Sep 09 20:10:53.845237 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:53.845013    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fedcf89e-e782-41b3-a761-53d04a626bde-kserve-provision-location\") pod \"fedcf89e-e782-41b3-a761-53d04a626bde\" (UID: \"fedcf89e-e782-41b3-a761-53d04a626bde\") "
Sep 09 20:10:53.845237 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:53.845055    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fedcf89e-e782-41b3-a761-53d04a626bde-kube-rbac-proxy-sar-config\") pod \"fedcf89e-e782-41b3-a761-53d04a626bde\" (UID: \"fedcf89e-e782-41b3-a761-53d04a626bde\") "
Sep 09 20:10:53.845237 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:53.845087    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fedcf89e-e782-41b3-a761-53d04a626bde-proxy-tls\") pod \"fedcf89e-e782-41b3-a761-53d04a626bde\" (UID: \"fedcf89e-e782-41b3-a761-53d04a626bde\") "
Sep 09 20:10:53.845466 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:53.845429    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fedcf89e-e782-41b3-a761-53d04a626bde-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fedcf89e-e782-41b3-a761-53d04a626bde" (UID: "fedcf89e-e782-41b3-a761-53d04a626bde"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:10:53.845466 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:53.845443    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fedcf89e-e782-41b3-a761-53d04a626bde-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fedcf89e-e782-41b3-a761-53d04a626bde" (UID: "fedcf89e-e782-41b3-a761-53d04a626bde"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:10:53.845645 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:53.845450    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fedcf89e-e782-41b3-a761-53d04a626bde-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fedcf89e-e782-41b3-a761-53d04a626bde" (UID: "fedcf89e-e782-41b3-a761-53d04a626bde"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:10:53.847349 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:53.847316    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fedcf89e-e782-41b3-a761-53d04a626bde-kube-api-access-tkpt5" (OuterVolumeSpecName: "kube-api-access-tkpt5") pod "fedcf89e-e782-41b3-a761-53d04a626bde" (UID: "fedcf89e-e782-41b3-a761-53d04a626bde"). InnerVolumeSpecName "kube-api-access-tkpt5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:10:53.847444 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:53.847379    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fedcf89e-e782-41b3-a761-53d04a626bde-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fedcf89e-e782-41b3-a761-53d04a626bde" (UID: "fedcf89e-e782-41b3-a761-53d04a626bde"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:10:53.946402 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:53.946376    2523 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fedcf89e-e782-41b3-a761-53d04a626bde-cabundle-cert\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:10:53.946402 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:53.946397    2523 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkpt5\" (UniqueName: \"kubernetes.io/projected/fedcf89e-e782-41b3-a761-53d04a626bde-kube-api-access-tkpt5\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:10:53.946402 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:53.946407    2523 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fedcf89e-e782-41b3-a761-53d04a626bde-kserve-provision-location\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:10:53.946578 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:53.946416    2523 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fedcf89e-e782-41b3-a761-53d04a626bde-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:10:53.946578 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:53.946425    2523 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fedcf89e-e782-41b3-a761-53d04a626bde-proxy-tls\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:10:54.237899 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:54.237819    2523 generic.go:356] "Generic (PLEG): container finished" podID="fedcf89e-e782-41b3-a761-53d04a626bde" containerID="ab3149e87747fa2a5e67a43ccfbb0355b5226e2bd978b20773354d7d54b81660" exitCode=0
Sep 09 20:10:54.237899 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:54.237891    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh"
Sep 09 20:10:54.238070 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:54.237905    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh" event={"ID":"fedcf89e-e782-41b3-a761-53d04a626bde","Type":"ContainerDied","Data":"ab3149e87747fa2a5e67a43ccfbb0355b5226e2bd978b20773354d7d54b81660"}
Sep 09 20:10:54.238070 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:54.237983    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh" event={"ID":"fedcf89e-e782-41b3-a761-53d04a626bde","Type":"ContainerDied","Data":"4c2f22e01b84ccc9a4cac831e90869800637a864fd154f43c7427b60497a2b18"}
Sep 09 20:10:54.238070 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:54.238030    2523 scope.go:117] "RemoveContainer" containerID="ab3149e87747fa2a5e67a43ccfbb0355b5226e2bd978b20773354d7d54b81660"
Sep 09 20:10:54.249170 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:54.249130    2523 scope.go:117] "RemoveContainer" containerID="8c43ea55692580bac4f5454ee4676089af0138213cb68beb77e5aed78514975e"
Sep 09 20:10:54.257082 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:54.257065    2523 scope.go:117] "RemoveContainer" containerID="db4ef39233dcccbaa1fb113560672d0b444aa91b16148fec6da8dd4826934d56"
Sep 09 20:10:54.262905 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:54.262884    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh"]
Sep 09 20:10:54.263872 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:54.263856    2523 scope.go:117] "RemoveContainer" containerID="fe10b4b532726e682c0c7776bbe3df24c4eba9cdb5df76861f6b4853ce0573cf"
Sep 09 20:10:54.268021 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:54.267998    2523 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-689fcfbf74-jnlbh"]
Sep 09 20:10:54.271588 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:54.271572    2523 scope.go:117] "RemoveContainer" containerID="ab3149e87747fa2a5e67a43ccfbb0355b5226e2bd978b20773354d7d54b81660"
Sep 09 20:10:54.271840 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:10:54.271823    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab3149e87747fa2a5e67a43ccfbb0355b5226e2bd978b20773354d7d54b81660\": container with ID starting with ab3149e87747fa2a5e67a43ccfbb0355b5226e2bd978b20773354d7d54b81660 not found: ID does not exist" containerID="ab3149e87747fa2a5e67a43ccfbb0355b5226e2bd978b20773354d7d54b81660"
Sep 09 20:10:54.271891 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:54.271845    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab3149e87747fa2a5e67a43ccfbb0355b5226e2bd978b20773354d7d54b81660"} err="failed to get container status \"ab3149e87747fa2a5e67a43ccfbb0355b5226e2bd978b20773354d7d54b81660\": rpc error: code = NotFound desc = could not find container \"ab3149e87747fa2a5e67a43ccfbb0355b5226e2bd978b20773354d7d54b81660\": container with ID starting with ab3149e87747fa2a5e67a43ccfbb0355b5226e2bd978b20773354d7d54b81660 not found: ID does not exist"
Sep 09 20:10:54.271891 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:54.271864    2523 scope.go:117] "RemoveContainer" containerID="8c43ea55692580bac4f5454ee4676089af0138213cb68beb77e5aed78514975e"
Sep 09 20:10:54.272139 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:10:54.272119    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c43ea55692580bac4f5454ee4676089af0138213cb68beb77e5aed78514975e\": container with ID starting with 8c43ea55692580bac4f5454ee4676089af0138213cb68beb77e5aed78514975e not found: ID does not exist" containerID="8c43ea55692580bac4f5454ee4676089af0138213cb68beb77e5aed78514975e"
Sep 09 20:10:54.272207 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:54.272160    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c43ea55692580bac4f5454ee4676089af0138213cb68beb77e5aed78514975e"} err="failed to get container status \"8c43ea55692580bac4f5454ee4676089af0138213cb68beb77e5aed78514975e\": rpc error: code = NotFound desc = could not find container \"8c43ea55692580bac4f5454ee4676089af0138213cb68beb77e5aed78514975e\": container with ID starting with 8c43ea55692580bac4f5454ee4676089af0138213cb68beb77e5aed78514975e not found: ID does not exist"
Sep 09 20:10:54.272207 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:54.272179    2523 scope.go:117] "RemoveContainer" containerID="db4ef39233dcccbaa1fb113560672d0b444aa91b16148fec6da8dd4826934d56"
Sep 09 20:10:54.272412 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:10:54.272396    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db4ef39233dcccbaa1fb113560672d0b444aa91b16148fec6da8dd4826934d56\": container with ID starting with db4ef39233dcccbaa1fb113560672d0b444aa91b16148fec6da8dd4826934d56 not found: ID does not exist" containerID="db4ef39233dcccbaa1fb113560672d0b444aa91b16148fec6da8dd4826934d56"
Sep 09 20:10:54.272451 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:54.272417    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db4ef39233dcccbaa1fb113560672d0b444aa91b16148fec6da8dd4826934d56"} err="failed to get container status \"db4ef39233dcccbaa1fb113560672d0b444aa91b16148fec6da8dd4826934d56\": rpc error: code = NotFound desc = could not find container \"db4ef39233dcccbaa1fb113560672d0b444aa91b16148fec6da8dd4826934d56\": container with ID starting with db4ef39233dcccbaa1fb113560672d0b444aa91b16148fec6da8dd4826934d56 not found: ID does not exist"
Sep 09 20:10:54.272451 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:54.272431    2523 scope.go:117] "RemoveContainer" containerID="fe10b4b532726e682c0c7776bbe3df24c4eba9cdb5df76861f6b4853ce0573cf"
Sep 09 20:10:54.272659 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:10:54.272645    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe10b4b532726e682c0c7776bbe3df24c4eba9cdb5df76861f6b4853ce0573cf\": container with ID starting with fe10b4b532726e682c0c7776bbe3df24c4eba9cdb5df76861f6b4853ce0573cf not found: ID does not exist" containerID="fe10b4b532726e682c0c7776bbe3df24c4eba9cdb5df76861f6b4853ce0573cf"
Sep 09 20:10:54.272699 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:54.272661    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe10b4b532726e682c0c7776bbe3df24c4eba9cdb5df76861f6b4853ce0573cf"} err="failed to get container status \"fe10b4b532726e682c0c7776bbe3df24c4eba9cdb5df76861f6b4853ce0573cf\": rpc error: code = NotFound desc = could not find container \"fe10b4b532726e682c0c7776bbe3df24c4eba9cdb5df76861f6b4853ce0573cf\": container with ID starting with fe10b4b532726e682c0c7776bbe3df24c4eba9cdb5df76861f6b4853ce0573cf not found: ID does not exist"
Sep 09 20:10:55.806869 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:55.806844    2523 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fedcf89e-e782-41b3-a761-53d04a626bde" path="/var/lib/kubelet/pods/fedcf89e-e782-41b3-a761-53d04a626bde/volumes"
Sep 09 20:10:57.200305 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:57.200258    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98" podUID="c12080b6-da22-4d43-abcf-eb5ae28b652b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 09 20:10:57.200798 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:10:57.200772    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98" podUID="c12080b6-da22-4d43-abcf-eb5ae28b652b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:11:07.200373 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:11:07.200333    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98" podUID="c12080b6-da22-4d43-abcf-eb5ae28b652b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 09 20:11:07.200894 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:11:07.200867    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98" podUID="c12080b6-da22-4d43-abcf-eb5ae28b652b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:11:17.200580 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:11:17.200537    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98" podUID="c12080b6-da22-4d43-abcf-eb5ae28b652b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 09 20:11:17.201073 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:11:17.200956    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98" podUID="c12080b6-da22-4d43-abcf-eb5ae28b652b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:11:27.200868 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:11:27.200827    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98" podUID="c12080b6-da22-4d43-abcf-eb5ae28b652b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 09 20:11:27.201445 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:11:27.201422    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98" podUID="c12080b6-da22-4d43-abcf-eb5ae28b652b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:11:37.200373 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:11:37.200321    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98" podUID="c12080b6-da22-4d43-abcf-eb5ae28b652b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 09 20:11:37.200812 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:11:37.200790    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98" podUID="c12080b6-da22-4d43-abcf-eb5ae28b652b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:11:47.201532 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:11:47.201486    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98"
Sep 09 20:11:47.202013 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:11:47.201993    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98"
Sep 09 20:11:58.914176 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:11:58.914125    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98"]
Sep 09 20:11:58.914578 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:11:58.914525    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98" podUID="c12080b6-da22-4d43-abcf-eb5ae28b652b" containerName="agent" containerID="cri-o://453c01475dd497f9b4ebc9aded25e1d5706f16734b16da134bd3d085da6804d3" gracePeriod=30
Sep 09 20:11:58.914650 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:11:58.914569    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98" podUID="c12080b6-da22-4d43-abcf-eb5ae28b652b" containerName="kserve-container" containerID="cri-o://092ea9e1bcbb3823e077776c606f56e3ddcc5f32e3e48341ae37378d297afdf3" gracePeriod=30
Sep 09 20:11:58.914700 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:11:58.914535    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98" podUID="c12080b6-da22-4d43-abcf-eb5ae28b652b" containerName="kube-rbac-proxy" containerID="cri-o://5efe1a857554f5dd51b28408e4c2d80aa06347b8df85f28ad4e9358093c3f0e6" gracePeriod=30
Sep 09 20:11:59.058865 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:11:59.058835    2523 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dczct"]
Sep 09 20:11:59.059124 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:11:59.059112    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fedcf89e-e782-41b3-a761-53d04a626bde" containerName="kube-rbac-proxy"
Sep 09 20:11:59.059189 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:11:59.059126    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="fedcf89e-e782-41b3-a761-53d04a626bde" containerName="kube-rbac-proxy"
Sep 09 20:11:59.059189 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:11:59.059142    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fedcf89e-e782-41b3-a761-53d04a626bde" containerName="storage-initializer"
Sep 09 20:11:59.059189 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:11:59.059161    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="fedcf89e-e782-41b3-a761-53d04a626bde" containerName="storage-initializer"
Sep 09 20:11:59.059189 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:11:59.059170    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fedcf89e-e782-41b3-a761-53d04a626bde" containerName="agent"
Sep 09 20:11:59.059189 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:11:59.059176    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="fedcf89e-e782-41b3-a761-53d04a626bde" containerName="agent"
Sep 09 20:11:59.059189 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:11:59.059183    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fedcf89e-e782-41b3-a761-53d04a626bde" containerName="kserve-container"
Sep 09 20:11:59.059189 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:11:59.059188    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="fedcf89e-e782-41b3-a761-53d04a626bde" containerName="kserve-container"
Sep 09 20:11:59.059394 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:11:59.059231    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="fedcf89e-e782-41b3-a761-53d04a626bde" containerName="agent"
Sep 09 20:11:59.059394 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:11:59.059239    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="fedcf89e-e782-41b3-a761-53d04a626bde" containerName="kserve-container"
Sep 09 20:11:59.059394 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:11:59.059245    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="fedcf89e-e782-41b3-a761-53d04a626bde" containerName="kube-rbac-proxy"
Sep 09 20:11:59.062034 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:11:59.062019    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dczct"
Sep 09 20:11:59.064416 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:11:59.064397    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Sep 09 20:11:59.064545 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:11:59.064528    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Sep 09 20:11:59.074061 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:11:59.074040    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dczct"]
Sep 09 20:11:59.190137 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:11:59.190065    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7cd34d1f-f5db-4993-aec0-8d4e237931a7-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-dczct\" (UID: \"7cd34d1f-f5db-4993-aec0-8d4e237931a7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dczct"
Sep 09 20:11:59.190137 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:11:59.190102    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cd34d1f-f5db-4993-aec0-8d4e237931a7-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-dczct\" (UID: \"7cd34d1f-f5db-4993-aec0-8d4e237931a7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dczct"
Sep 09 20:11:59.190307 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:11:59.190208    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7cd34d1f-f5db-4993-aec0-8d4e237931a7-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-dczct\" (UID: \"7cd34d1f-f5db-4993-aec0-8d4e237931a7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dczct"
Sep 09 20:11:59.190307 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:11:59.190242    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7cd34d1f-f5db-4993-aec0-8d4e237931a7-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-dczct\" (UID: \"7cd34d1f-f5db-4993-aec0-8d4e237931a7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dczct"
Sep 09 20:11:59.190307 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:11:59.190265    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qmq6\" (UniqueName: \"kubernetes.io/projected/7cd34d1f-f5db-4993-aec0-8d4e237931a7-kube-api-access-6qmq6\") pod \"isvc-autogluon-v1-predictor-574b5b7495-dczct\" (UID: \"7cd34d1f-f5db-4993-aec0-8d4e237931a7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dczct"
Sep 09 20:11:59.291087 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:11:59.291055    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7cd34d1f-f5db-4993-aec0-8d4e237931a7-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-dczct\" (UID: \"7cd34d1f-f5db-4993-aec0-8d4e237931a7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dczct"
Sep 09 20:11:59.291225 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:11:59.291095    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cd34d1f-f5db-4993-aec0-8d4e237931a7-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-dczct\" (UID: \"7cd34d1f-f5db-4993-aec0-8d4e237931a7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dczct"
Sep 09 20:11:59.291225 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:11:59.291130    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7cd34d1f-f5db-4993-aec0-8d4e237931a7-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-dczct\" (UID: \"7cd34d1f-f5db-4993-aec0-8d4e237931a7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dczct"
Sep 09 20:11:59.291225 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:11:59.291179    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7cd34d1f-f5db-4993-aec0-8d4e237931a7-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-dczct\" (UID: \"7cd34d1f-f5db-4993-aec0-8d4e237931a7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dczct"
Sep 09 20:11:59.291225 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:11:59.291201    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qmq6\" (UniqueName: \"kubernetes.io/projected/7cd34d1f-f5db-4993-aec0-8d4e237931a7-kube-api-access-6qmq6\") pod \"isvc-autogluon-v1-predictor-574b5b7495-dczct\" (UID: \"7cd34d1f-f5db-4993-aec0-8d4e237931a7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dczct"
Sep 09 20:11:59.291587 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:11:59.291568    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7cd34d1f-f5db-4993-aec0-8d4e237931a7-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-dczct\" (UID: \"7cd34d1f-f5db-4993-aec0-8d4e237931a7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dczct"
Sep 09 20:11:59.291872 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:11:59.291852    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7cd34d1f-f5db-4993-aec0-8d4e237931a7-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-dczct\" (UID: \"7cd34d1f-f5db-4993-aec0-8d4e237931a7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dczct"
Sep 09 20:11:59.291917 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:11:59.291895    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7cd34d1f-f5db-4993-aec0-8d4e237931a7-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-dczct\" (UID: \"7cd34d1f-f5db-4993-aec0-8d4e237931a7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dczct"
Sep 09 20:11:59.293654 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:11:59.293630    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cd34d1f-f5db-4993-aec0-8d4e237931a7-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-dczct\" (UID: \"7cd34d1f-f5db-4993-aec0-8d4e237931a7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dczct"
Sep 09 20:11:59.300243 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:11:59.300215    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qmq6\" (UniqueName: \"kubernetes.io/projected/7cd34d1f-f5db-4993-aec0-8d4e237931a7-kube-api-access-6qmq6\") pod \"isvc-autogluon-v1-predictor-574b5b7495-dczct\" (UID: \"7cd34d1f-f5db-4993-aec0-8d4e237931a7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dczct"
Sep 09 20:11:59.371198 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:11:59.371170    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dczct"
Sep 09 20:11:59.475442 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:11:59.475362    2523 generic.go:356] "Generic (PLEG): container finished" podID="c12080b6-da22-4d43-abcf-eb5ae28b652b" containerID="5efe1a857554f5dd51b28408e4c2d80aa06347b8df85f28ad4e9358093c3f0e6" exitCode=2
Sep 09 20:11:59.475442 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:11:59.475410    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98" event={"ID":"c12080b6-da22-4d43-abcf-eb5ae28b652b","Type":"ContainerDied","Data":"5efe1a857554f5dd51b28408e4c2d80aa06347b8df85f28ad4e9358093c3f0e6"}
Sep 09 20:11:59.642240 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:11:59.642215    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dczct"]
Sep 09 20:11:59.644399 ip-10-0-132-226 kubenswrapper[2523]: W0909 20:11:59.644362    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cd34d1f_f5db_4993_aec0_8d4e237931a7.slice/crio-74e7369d170c8d86a0545d7d61ed7b63f0ac51c21c68d13b854b9a49bf39bc2b WatchSource:0}: Error finding container 74e7369d170c8d86a0545d7d61ed7b63f0ac51c21c68d13b854b9a49bf39bc2b: Status 404 returned error can't find the container with id 74e7369d170c8d86a0545d7d61ed7b63f0ac51c21c68d13b854b9a49bf39bc2b
Sep 09 20:12:00.480423 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:12:00.480384    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dczct" event={"ID":"7cd34d1f-f5db-4993-aec0-8d4e237931a7","Type":"ContainerStarted","Data":"2d6e7a2e822f1d7048adee4b9be58774e606ced74862d9ba428f4e9e070032c8"}
Sep 09 20:12:00.480801 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:12:00.480451    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dczct" event={"ID":"7cd34d1f-f5db-4993-aec0-8d4e237931a7","Type":"ContainerStarted","Data":"74e7369d170c8d86a0545d7d61ed7b63f0ac51c21c68d13b854b9a49bf39bc2b"}
Sep 09 20:12:02.195715 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:12:02.195677    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98" podUID="c12080b6-da22-4d43-abcf-eb5ae28b652b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Sep 09 20:12:03.493591 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:12:03.493559    2523 generic.go:356] "Generic (PLEG): container finished" podID="c12080b6-da22-4d43-abcf-eb5ae28b652b" containerID="092ea9e1bcbb3823e077776c606f56e3ddcc5f32e3e48341ae37378d297afdf3" exitCode=0
Sep 09 20:12:03.493927 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:12:03.493636    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98" event={"ID":"c12080b6-da22-4d43-abcf-eb5ae28b652b","Type":"ContainerDied","Data":"092ea9e1bcbb3823e077776c606f56e3ddcc5f32e3e48341ae37378d297afdf3"}
Sep 09 20:12:07.195608 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:12:07.195571    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98" podUID="c12080b6-da22-4d43-abcf-eb5ae28b652b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Sep 09 20:12:07.201340 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:12:07.201311    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98" podUID="c12080b6-da22-4d43-abcf-eb5ae28b652b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:12:07.202552 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:12:07.202530    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98" podUID="c12080b6-da22-4d43-abcf-eb5ae28b652b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 09 20:12:12.195748 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:12:12.195706    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98" podUID="c12080b6-da22-4d43-abcf-eb5ae28b652b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Sep 09 20:12:12.196099 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:12:12.195831    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98"
Sep 09 20:12:17.195586 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:12:17.195547    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98" podUID="c12080b6-da22-4d43-abcf-eb5ae28b652b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Sep 09 20:12:17.201241 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:12:17.201210    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98" podUID="c12080b6-da22-4d43-abcf-eb5ae28b652b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:12:17.202111 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:12:17.202089    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98" podUID="c12080b6-da22-4d43-abcf-eb5ae28b652b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 09 20:12:22.195433 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:12:22.195385    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98" podUID="c12080b6-da22-4d43-abcf-eb5ae28b652b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Sep 09 20:12:27.196293 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:12:27.196250    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98" podUID="c12080b6-da22-4d43-abcf-eb5ae28b652b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Sep 09 20:12:27.200938 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:12:27.200909    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98" podUID="c12080b6-da22-4d43-abcf-eb5ae28b652b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:12:27.201063 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:12:27.201047    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98"
Sep 09 20:12:27.202163 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:12:27.202127    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98" podUID="c12080b6-da22-4d43-abcf-eb5ae28b652b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 09 20:12:27.202268 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:12:27.202244    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98"
Sep 09 20:12:29.055481 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:12:29.055457    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98"
Sep 09 20:12:29.182217 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:12:29.182134    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rq4w\" (UniqueName: \"kubernetes.io/projected/c12080b6-da22-4d43-abcf-eb5ae28b652b-kube-api-access-9rq4w\") pod \"c12080b6-da22-4d43-abcf-eb5ae28b652b\" (UID: \"c12080b6-da22-4d43-abcf-eb5ae28b652b\") "
Sep 09 20:12:29.182217 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:12:29.182201    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c12080b6-da22-4d43-abcf-eb5ae28b652b-kube-rbac-proxy-sar-config\") pod \"c12080b6-da22-4d43-abcf-eb5ae28b652b\" (UID: \"c12080b6-da22-4d43-abcf-eb5ae28b652b\") "
Sep 09 20:12:29.182380 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:12:29.182310    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c12080b6-da22-4d43-abcf-eb5ae28b652b-cabundle-cert\") pod \"c12080b6-da22-4d43-abcf-eb5ae28b652b\" (UID: \"c12080b6-da22-4d43-abcf-eb5ae28b652b\") "
Sep 09 20:12:29.182380 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:12:29.182370    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c12080b6-da22-4d43-abcf-eb5ae28b652b-kserve-provision-location\") pod \"c12080b6-da22-4d43-abcf-eb5ae28b652b\" (UID: \"c12080b6-da22-4d43-abcf-eb5ae28b652b\") "
Sep 09 20:12:29.182484 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:12:29.182409    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c12080b6-da22-4d43-abcf-eb5ae28b652b-proxy-tls\") pod \"c12080b6-da22-4d43-abcf-eb5ae28b652b\" (UID: \"c12080b6-da22-4d43-abcf-eb5ae28b652b\") "
Sep 09 20:12:29.182593 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:12:29.182480    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c12080b6-da22-4d43-abcf-eb5ae28b652b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c12080b6-da22-4d43-abcf-eb5ae28b652b" (UID: "c12080b6-da22-4d43-abcf-eb5ae28b652b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:12:29.182673 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:12:29.182649    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c12080b6-da22-4d43-abcf-eb5ae28b652b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c12080b6-da22-4d43-abcf-eb5ae28b652b" (UID: "c12080b6-da22-4d43-abcf-eb5ae28b652b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:12:29.182718 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:12:29.182657    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c12080b6-da22-4d43-abcf-eb5ae28b652b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c12080b6-da22-4d43-abcf-eb5ae28b652b" (UID: "c12080b6-da22-4d43-abcf-eb5ae28b652b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:12:29.184229 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:12:29.184207    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c12080b6-da22-4d43-abcf-eb5ae28b652b-kube-api-access-9rq4w" (OuterVolumeSpecName: "kube-api-access-9rq4w") pod "c12080b6-da22-4d43-abcf-eb5ae28b652b" (UID: "c12080b6-da22-4d43-abcf-eb5ae28b652b"). InnerVolumeSpecName "kube-api-access-9rq4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:12:29.184417 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:12:29.184397    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c12080b6-da22-4d43-abcf-eb5ae28b652b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c12080b6-da22-4d43-abcf-eb5ae28b652b" (UID: "c12080b6-da22-4d43-abcf-eb5ae28b652b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:12:29.283742 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:12:29.283715    2523 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c12080b6-da22-4d43-abcf-eb5ae28b652b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:12:29.283742 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:12:29.283737    2523 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c12080b6-da22-4d43-abcf-eb5ae28b652b-cabundle-cert\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:12:29.283908 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:12:29.283748    2523 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c12080b6-da22-4d43-abcf-eb5ae28b652b-kserve-provision-location\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:12:29.283908 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:12:29.283757    2523 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c12080b6-da22-4d43-abcf-eb5ae28b652b-proxy-tls\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:12:29.283908 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:12:29.283765    2523 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rq4w\" (UniqueName: \"kubernetes.io/projected/c12080b6-da22-4d43-abcf-eb5ae28b652b-kube-api-access-9rq4w\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:12:29.588675 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:12:29.588645    2523 generic.go:356] "Generic (PLEG): container finished" podID="7cd34d1f-f5db-4993-aec0-8d4e237931a7" containerID="2d6e7a2e822f1d7048adee4b9be58774e606ced74862d9ba428f4e9e070032c8" exitCode=0
Sep 09 20:12:29.588818 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:12:29.588726    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dczct" event={"ID":"7cd34d1f-f5db-4993-aec0-8d4e237931a7","Type":"ContainerDied","Data":"2d6e7a2e822f1d7048adee4b9be58774e606ced74862d9ba428f4e9e070032c8"}
Sep 09 20:12:29.590221 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:12:29.590204    2523 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:12:29.591088 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:12:29.591064    2523 generic.go:356] "Generic (PLEG): container finished" podID="c12080b6-da22-4d43-abcf-eb5ae28b652b" containerID="453c01475dd497f9b4ebc9aded25e1d5706f16734b16da134bd3d085da6804d3" exitCode=0
Sep 09 20:12:29.591203 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:12:29.591114    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98" event={"ID":"c12080b6-da22-4d43-abcf-eb5ae28b652b","Type":"ContainerDied","Data":"453c01475dd497f9b4ebc9aded25e1d5706f16734b16da134bd3d085da6804d3"}
Sep 09 20:12:29.591203 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:12:29.591140    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98"
Sep 09 20:12:29.591203 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:12:29.591184    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98" event={"ID":"c12080b6-da22-4d43-abcf-eb5ae28b652b","Type":"ContainerDied","Data":"22bfcdfa73e5640bd4bcacc544ab61a18840326e729e7e1847236710f15c7114"}
Sep 09 20:12:29.591356 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:12:29.591215    2523 scope.go:117] "RemoveContainer" containerID="453c01475dd497f9b4ebc9aded25e1d5706f16734b16da134bd3d085da6804d3"
Sep 09 20:12:29.599956 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:12:29.599934    2523 scope.go:117] "RemoveContainer" containerID="5efe1a857554f5dd51b28408e4c2d80aa06347b8df85f28ad4e9358093c3f0e6"
Sep 09 20:12:29.608999 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:12:29.608983    2523 scope.go:117] "RemoveContainer" containerID="092ea9e1bcbb3823e077776c606f56e3ddcc5f32e3e48341ae37378d297afdf3"
Sep 09 20:12:29.616133 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:12:29.616111    2523 scope.go:117] "RemoveContainer" containerID="6c15105afe67d0cf891d3fae65dc7dd9b710c6276b1ab9d6f7857db8c1c53b36"
Sep 09 20:12:29.621223 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:12:29.621205    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98"]
Sep 09 20:12:29.623298 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:12:29.623281    2523 scope.go:117] "RemoveContainer" containerID="453c01475dd497f9b4ebc9aded25e1d5706f16734b16da134bd3d085da6804d3"
Sep 09 20:12:29.623669 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:12:29.623643    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"453c01475dd497f9b4ebc9aded25e1d5706f16734b16da134bd3d085da6804d3\": container with ID starting with 453c01475dd497f9b4ebc9aded25e1d5706f16734b16da134bd3d085da6804d3 not found: ID does not exist" containerID="453c01475dd497f9b4ebc9aded25e1d5706f16734b16da134bd3d085da6804d3"
Sep 09 20:12:29.623669 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:12:29.623681    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"453c01475dd497f9b4ebc9aded25e1d5706f16734b16da134bd3d085da6804d3"} err="failed to get container status \"453c01475dd497f9b4ebc9aded25e1d5706f16734b16da134bd3d085da6804d3\": rpc error: code = NotFound desc = could not find container \"453c01475dd497f9b4ebc9aded25e1d5706f16734b16da134bd3d085da6804d3\": container with ID starting with 453c01475dd497f9b4ebc9aded25e1d5706f16734b16da134bd3d085da6804d3 not found: ID does not exist"
Sep 09 20:12:29.623669 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:12:29.623705    2523 scope.go:117] "RemoveContainer" containerID="5efe1a857554f5dd51b28408e4c2d80aa06347b8df85f28ad4e9358093c3f0e6"
Sep 09 20:12:29.624055 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:12:29.624007    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5efe1a857554f5dd51b28408e4c2d80aa06347b8df85f28ad4e9358093c3f0e6\": container with ID starting with 5efe1a857554f5dd51b28408e4c2d80aa06347b8df85f28ad4e9358093c3f0e6 not found: ID does not exist" containerID="5efe1a857554f5dd51b28408e4c2d80aa06347b8df85f28ad4e9358093c3f0e6"
Sep 09 20:12:29.624055 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:12:29.624041    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5efe1a857554f5dd51b28408e4c2d80aa06347b8df85f28ad4e9358093c3f0e6"} err="failed to get container status \"5efe1a857554f5dd51b28408e4c2d80aa06347b8df85f28ad4e9358093c3f0e6\": rpc error: code = NotFound desc = could not find container \"5efe1a857554f5dd51b28408e4c2d80aa06347b8df85f28ad4e9358093c3f0e6\": container with ID starting with 5efe1a857554f5dd51b28408e4c2d80aa06347b8df85f28ad4e9358093c3f0e6 not found: ID does not exist"
Sep 09 20:12:29.624276 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:12:29.624061    2523 scope.go:117] "RemoveContainer" containerID="092ea9e1bcbb3823e077776c606f56e3ddcc5f32e3e48341ae37378d297afdf3"
Sep 09 20:12:29.624423 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:12:29.624399    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"092ea9e1bcbb3823e077776c606f56e3ddcc5f32e3e48341ae37378d297afdf3\": container with ID starting with 092ea9e1bcbb3823e077776c606f56e3ddcc5f32e3e48341ae37378d297afdf3 not found: ID does not exist" containerID="092ea9e1bcbb3823e077776c606f56e3ddcc5f32e3e48341ae37378d297afdf3"
Sep 09 20:12:29.624463 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:12:29.624432    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"092ea9e1bcbb3823e077776c606f56e3ddcc5f32e3e48341ae37378d297afdf3"} err="failed to get container status \"092ea9e1bcbb3823e077776c606f56e3ddcc5f32e3e48341ae37378d297afdf3\": rpc error: code = NotFound desc = could not find container \"092ea9e1bcbb3823e077776c606f56e3ddcc5f32e3e48341ae37378d297afdf3\": container with ID starting with 092ea9e1bcbb3823e077776c606f56e3ddcc5f32e3e48341ae37378d297afdf3 not found: ID does not exist"
Sep 09 20:12:29.624463 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:12:29.624452    2523 scope.go:117] "RemoveContainer" containerID="6c15105afe67d0cf891d3fae65dc7dd9b710c6276b1ab9d6f7857db8c1c53b36"
Sep 09 20:12:29.624741 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:12:29.624715    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c15105afe67d0cf891d3fae65dc7dd9b710c6276b1ab9d6f7857db8c1c53b36\": container with ID starting with 6c15105afe67d0cf891d3fae65dc7dd9b710c6276b1ab9d6f7857db8c1c53b36 not found: ID does not exist" containerID="6c15105afe67d0cf891d3fae65dc7dd9b710c6276b1ab9d6f7857db8c1c53b36"
Sep 09 20:12:29.624741 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:12:29.624736    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c15105afe67d0cf891d3fae65dc7dd9b710c6276b1ab9d6f7857db8c1c53b36"} err="failed to get container status \"6c15105afe67d0cf891d3fae65dc7dd9b710c6276b1ab9d6f7857db8c1c53b36\": rpc error: code = NotFound desc = could not find container \"6c15105afe67d0cf891d3fae65dc7dd9b710c6276b1ab9d6f7857db8c1c53b36\": container with ID starting with 6c15105afe67d0cf891d3fae65dc7dd9b710c6276b1ab9d6f7857db8c1c53b36 not found: ID does not exist"
Sep 09 20:12:29.625382 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:12:29.625365    2523 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-85d5cb84f6-x9r98"]
Sep 09 20:12:29.807032 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:12:29.807006    2523 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c12080b6-da22-4d43-abcf-eb5ae28b652b" path="/var/lib/kubelet/pods/c12080b6-da22-4d43-abcf-eb5ae28b652b/volumes"
Sep 09 20:14:35.115779 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:14:35.115751    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dczct" event={"ID":"7cd34d1f-f5db-4993-aec0-8d4e237931a7","Type":"ContainerStarted","Data":"2c2fce9f8542206a3d10b8816e3b5f8211425ccf9e62475402e8d036d897dc9b"}
Sep 09 20:14:36.122041 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:14:36.121999    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dczct" event={"ID":"7cd34d1f-f5db-4993-aec0-8d4e237931a7","Type":"ContainerStarted","Data":"1523a736a08597ee1ed576a1a653140fda7e32b2923da6b8326a9934a21f4032"}
Sep 09 20:14:36.122502 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:14:36.122067    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dczct"
Sep 09 20:14:36.122502 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:14:36.122122    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dczct"
Sep 09 20:14:36.123803 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:14:36.123777    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dczct" podUID="7cd34d1f-f5db-4993-aec0-8d4e237931a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 09 20:14:36.143898 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:14:36.143857    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dczct" podStartSLOduration=2.32136963 podStartE2EDuration="2m37.143845249s" totalImagesPullingTime="2m5.366703544s" totalInitContainerRuntime="29.455772075s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:11:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:14:36.141502344 +0000 UTC m=+1180.813521343" watchObservedRunningTime="2026-09-09 20:14:36.143845249 +0000 UTC m=+1180.815864245"
Sep 09 20:14:37.126508 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:14:37.126465    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dczct" podUID="7cd34d1f-f5db-4993-aec0-8d4e237931a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 09 20:14:42.131433 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:14:42.131406    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dczct"
Sep 09 20:14:42.132024 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:14:42.131990    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dczct" podUID="7cd34d1f-f5db-4993-aec0-8d4e237931a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 09 20:14:52.132898 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:14:52.132865    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dczct"
Sep 09 20:15:05.208923 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:05.208839    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dczct"]
Sep 09 20:15:05.209553 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:05.209212    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dczct" podUID="7cd34d1f-f5db-4993-aec0-8d4e237931a7" containerName="kube-rbac-proxy" containerID="cri-o://1523a736a08597ee1ed576a1a653140fda7e32b2923da6b8326a9934a21f4032" gracePeriod=30
Sep 09 20:15:05.209553 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:05.209212    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dczct" podUID="7cd34d1f-f5db-4993-aec0-8d4e237931a7" containerName="kserve-container" containerID="cri-o://2c2fce9f8542206a3d10b8816e3b5f8211425ccf9e62475402e8d036d897dc9b" gracePeriod=30
Sep 09 20:15:06.237652 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:06.237618    2523 generic.go:356] "Generic (PLEG): container finished" podID="7cd34d1f-f5db-4993-aec0-8d4e237931a7" containerID="1523a736a08597ee1ed576a1a653140fda7e32b2923da6b8326a9934a21f4032" exitCode=2
Sep 09 20:15:06.238047 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:06.237695    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dczct" event={"ID":"7cd34d1f-f5db-4993-aec0-8d4e237931a7","Type":"ContainerDied","Data":"1523a736a08597ee1ed576a1a653140fda7e32b2923da6b8326a9934a21f4032"}
Sep 09 20:15:06.491272 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:06.491222    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dczct"
Sep 09 20:15:06.560357 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:06.560329    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7cd34d1f-f5db-4993-aec0-8d4e237931a7-kserve-provision-location\") pod \"7cd34d1f-f5db-4993-aec0-8d4e237931a7\" (UID: \"7cd34d1f-f5db-4993-aec0-8d4e237931a7\") "
Sep 09 20:15:06.560472 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:06.560385    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cd34d1f-f5db-4993-aec0-8d4e237931a7-proxy-tls\") pod \"7cd34d1f-f5db-4993-aec0-8d4e237931a7\" (UID: \"7cd34d1f-f5db-4993-aec0-8d4e237931a7\") "
Sep 09 20:15:06.560472 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:06.560437    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7cd34d1f-f5db-4993-aec0-8d4e237931a7-cabundle-cert\") pod \"7cd34d1f-f5db-4993-aec0-8d4e237931a7\" (UID: \"7cd34d1f-f5db-4993-aec0-8d4e237931a7\") "
Sep 09 20:15:06.560578 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:06.560561    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7cd34d1f-f5db-4993-aec0-8d4e237931a7-kube-rbac-proxy-sar-config\") pod \"7cd34d1f-f5db-4993-aec0-8d4e237931a7\" (UID: \"7cd34d1f-f5db-4993-aec0-8d4e237931a7\") "
Sep 09 20:15:06.560624 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:06.560589    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6qmq6\" (UniqueName: \"kubernetes.io/projected/7cd34d1f-f5db-4993-aec0-8d4e237931a7-kube-api-access-6qmq6\") pod \"7cd34d1f-f5db-4993-aec0-8d4e237931a7\" (UID: \"7cd34d1f-f5db-4993-aec0-8d4e237931a7\") "
Sep 09 20:15:06.560784 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:06.560763    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cd34d1f-f5db-4993-aec0-8d4e237931a7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7cd34d1f-f5db-4993-aec0-8d4e237931a7" (UID: "7cd34d1f-f5db-4993-aec0-8d4e237931a7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:15:06.560885 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:06.560862    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cd34d1f-f5db-4993-aec0-8d4e237931a7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7cd34d1f-f5db-4993-aec0-8d4e237931a7" (UID: "7cd34d1f-f5db-4993-aec0-8d4e237931a7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:15:06.562626 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:06.562597    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cd34d1f-f5db-4993-aec0-8d4e237931a7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7cd34d1f-f5db-4993-aec0-8d4e237931a7" (UID: "7cd34d1f-f5db-4993-aec0-8d4e237931a7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:15:06.562740 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:06.562718    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cd34d1f-f5db-4993-aec0-8d4e237931a7-kube-api-access-6qmq6" (OuterVolumeSpecName: "kube-api-access-6qmq6") pod "7cd34d1f-f5db-4993-aec0-8d4e237931a7" (UID: "7cd34d1f-f5db-4993-aec0-8d4e237931a7"). InnerVolumeSpecName "kube-api-access-6qmq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:15:06.564985 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:06.564963    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cd34d1f-f5db-4993-aec0-8d4e237931a7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7cd34d1f-f5db-4993-aec0-8d4e237931a7" (UID: "7cd34d1f-f5db-4993-aec0-8d4e237931a7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:15:06.661833 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:06.661809    2523 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cd34d1f-f5db-4993-aec0-8d4e237931a7-proxy-tls\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:15:06.661833 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:06.661829    2523 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7cd34d1f-f5db-4993-aec0-8d4e237931a7-cabundle-cert\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:15:06.661945 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:06.661838    2523 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7cd34d1f-f5db-4993-aec0-8d4e237931a7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:15:06.661945 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:06.661848    2523 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qmq6\" (UniqueName: \"kubernetes.io/projected/7cd34d1f-f5db-4993-aec0-8d4e237931a7-kube-api-access-6qmq6\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:15:06.661945 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:06.661857    2523 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7cd34d1f-f5db-4993-aec0-8d4e237931a7-kserve-provision-location\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:15:07.242656 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:07.242632    2523 generic.go:356] "Generic (PLEG): container finished" podID="7cd34d1f-f5db-4993-aec0-8d4e237931a7" containerID="2c2fce9f8542206a3d10b8816e3b5f8211425ccf9e62475402e8d036d897dc9b" exitCode=0
Sep 09 20:15:07.242946 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:07.242692    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dczct"
Sep 09 20:15:07.242946 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:07.242722    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dczct" event={"ID":"7cd34d1f-f5db-4993-aec0-8d4e237931a7","Type":"ContainerDied","Data":"2c2fce9f8542206a3d10b8816e3b5f8211425ccf9e62475402e8d036d897dc9b"}
Sep 09 20:15:07.242946 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:07.242779    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dczct" event={"ID":"7cd34d1f-f5db-4993-aec0-8d4e237931a7","Type":"ContainerDied","Data":"74e7369d170c8d86a0545d7d61ed7b63f0ac51c21c68d13b854b9a49bf39bc2b"}
Sep 09 20:15:07.242946 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:07.242807    2523 scope.go:117] "RemoveContainer" containerID="1523a736a08597ee1ed576a1a653140fda7e32b2923da6b8326a9934a21f4032"
Sep 09 20:15:07.251058 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:07.251042    2523 scope.go:117] "RemoveContainer" containerID="2c2fce9f8542206a3d10b8816e3b5f8211425ccf9e62475402e8d036d897dc9b"
Sep 09 20:15:07.258280 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:07.258262    2523 scope.go:117] "RemoveContainer" containerID="2d6e7a2e822f1d7048adee4b9be58774e606ced74862d9ba428f4e9e070032c8"
Sep 09 20:15:07.264610 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:07.264591    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dczct"]
Sep 09 20:15:07.265376 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:07.265360    2523 scope.go:117] "RemoveContainer" containerID="1523a736a08597ee1ed576a1a653140fda7e32b2923da6b8326a9934a21f4032"
Sep 09 20:15:07.265646 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:15:07.265627    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1523a736a08597ee1ed576a1a653140fda7e32b2923da6b8326a9934a21f4032\": container with ID starting with 1523a736a08597ee1ed576a1a653140fda7e32b2923da6b8326a9934a21f4032 not found: ID does not exist" containerID="1523a736a08597ee1ed576a1a653140fda7e32b2923da6b8326a9934a21f4032"
Sep 09 20:15:07.265693 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:07.265653    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1523a736a08597ee1ed576a1a653140fda7e32b2923da6b8326a9934a21f4032"} err="failed to get container status \"1523a736a08597ee1ed576a1a653140fda7e32b2923da6b8326a9934a21f4032\": rpc error: code = NotFound desc = could not find container \"1523a736a08597ee1ed576a1a653140fda7e32b2923da6b8326a9934a21f4032\": container with ID starting with 1523a736a08597ee1ed576a1a653140fda7e32b2923da6b8326a9934a21f4032 not found: ID does not exist"
Sep 09 20:15:07.265693 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:07.265670    2523 scope.go:117] "RemoveContainer" containerID="2c2fce9f8542206a3d10b8816e3b5f8211425ccf9e62475402e8d036d897dc9b"
Sep 09 20:15:07.265906 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:15:07.265889    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c2fce9f8542206a3d10b8816e3b5f8211425ccf9e62475402e8d036d897dc9b\": container with ID starting with 2c2fce9f8542206a3d10b8816e3b5f8211425ccf9e62475402e8d036d897dc9b not found: ID does not exist" containerID="2c2fce9f8542206a3d10b8816e3b5f8211425ccf9e62475402e8d036d897dc9b"
Sep 09 20:15:07.265965 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:07.265915    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c2fce9f8542206a3d10b8816e3b5f8211425ccf9e62475402e8d036d897dc9b"} err="failed to get container status \"2c2fce9f8542206a3d10b8816e3b5f8211425ccf9e62475402e8d036d897dc9b\": rpc error: code = NotFound desc = could not find container \"2c2fce9f8542206a3d10b8816e3b5f8211425ccf9e62475402e8d036d897dc9b\": container with ID starting with 2c2fce9f8542206a3d10b8816e3b5f8211425ccf9e62475402e8d036d897dc9b not found: ID does not exist"
Sep 09 20:15:07.265965 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:07.265937    2523 scope.go:117] "RemoveContainer" containerID="2d6e7a2e822f1d7048adee4b9be58774e606ced74862d9ba428f4e9e070032c8"
Sep 09 20:15:07.266285 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:15:07.266244    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d6e7a2e822f1d7048adee4b9be58774e606ced74862d9ba428f4e9e070032c8\": container with ID starting with 2d6e7a2e822f1d7048adee4b9be58774e606ced74862d9ba428f4e9e070032c8 not found: ID does not exist" containerID="2d6e7a2e822f1d7048adee4b9be58774e606ced74862d9ba428f4e9e070032c8"
Sep 09 20:15:07.266429 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:07.266322    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d6e7a2e822f1d7048adee4b9be58774e606ced74862d9ba428f4e9e070032c8"} err="failed to get container status \"2d6e7a2e822f1d7048adee4b9be58774e606ced74862d9ba428f4e9e070032c8\": rpc error: code = NotFound desc = could not find container \"2d6e7a2e822f1d7048adee4b9be58774e606ced74862d9ba428f4e9e070032c8\": container with ID starting with 2d6e7a2e822f1d7048adee4b9be58774e606ced74862d9ba428f4e9e070032c8 not found: ID does not exist"
Sep 09 20:15:07.267973 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:07.267953    2523 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-dczct"]
Sep 09 20:15:07.806707 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:07.806678    2523 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cd34d1f-f5db-4993-aec0-8d4e237931a7" path="/var/lib/kubelet/pods/7cd34d1f-f5db-4993-aec0-8d4e237931a7/volumes"
Sep 09 20:15:10.385837 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:10.385804    2523 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-k2dvq"]
Sep 09 20:15:10.386287 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:10.386051    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cd34d1f-f5db-4993-aec0-8d4e237931a7" containerName="kserve-container"
Sep 09 20:15:10.386287 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:10.386062    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cd34d1f-f5db-4993-aec0-8d4e237931a7" containerName="kserve-container"
Sep 09 20:15:10.386287 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:10.386071    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c12080b6-da22-4d43-abcf-eb5ae28b652b" containerName="storage-initializer"
Sep 09 20:15:10.386287 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:10.386076    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="c12080b6-da22-4d43-abcf-eb5ae28b652b" containerName="storage-initializer"
Sep 09 20:15:10.386287 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:10.386086    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c12080b6-da22-4d43-abcf-eb5ae28b652b" containerName="kserve-container"
Sep 09 20:15:10.386287 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:10.386092    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="c12080b6-da22-4d43-abcf-eb5ae28b652b" containerName="kserve-container"
Sep 09 20:15:10.386287 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:10.386103    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cd34d1f-f5db-4993-aec0-8d4e237931a7" containerName="kube-rbac-proxy"
Sep 09 20:15:10.386287 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:10.386108    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cd34d1f-f5db-4993-aec0-8d4e237931a7" containerName="kube-rbac-proxy"
Sep 09 20:15:10.386287 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:10.386115    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c12080b6-da22-4d43-abcf-eb5ae28b652b" containerName="kube-rbac-proxy"
Sep 09 20:15:10.386287 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:10.386120    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="c12080b6-da22-4d43-abcf-eb5ae28b652b" containerName="kube-rbac-proxy"
Sep 09 20:15:10.386287 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:10.386125    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c12080b6-da22-4d43-abcf-eb5ae28b652b" containerName="agent"
Sep 09 20:15:10.386287 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:10.386129    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="c12080b6-da22-4d43-abcf-eb5ae28b652b" containerName="agent"
Sep 09 20:15:10.386287 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:10.386134    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cd34d1f-f5db-4993-aec0-8d4e237931a7" containerName="storage-initializer"
Sep 09 20:15:10.386287 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:10.386139    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cd34d1f-f5db-4993-aec0-8d4e237931a7" containerName="storage-initializer"
Sep 09 20:15:10.386287 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:10.386194    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cd34d1f-f5db-4993-aec0-8d4e237931a7" containerName="kube-rbac-proxy"
Sep 09 20:15:10.386287 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:10.386202    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="c12080b6-da22-4d43-abcf-eb5ae28b652b" containerName="agent"
Sep 09 20:15:10.386287 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:10.386207    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="c12080b6-da22-4d43-abcf-eb5ae28b652b" containerName="kserve-container"
Sep 09 20:15:10.386287 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:10.386213    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="c12080b6-da22-4d43-abcf-eb5ae28b652b" containerName="kube-rbac-proxy"
Sep 09 20:15:10.386287 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:10.386218    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cd34d1f-f5db-4993-aec0-8d4e237931a7" containerName="kserve-container"
Sep 09 20:15:10.418028 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:10.418005    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-k2dvq"]
Sep 09 20:15:10.418168 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:10.418099    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-k2dvq"
Sep 09 20:15:10.420615 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:10.420590    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 20:15:10.420755 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:10.420737    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-ks7mb\""
Sep 09 20:15:10.420825 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:10.420811    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Sep 09 20:15:10.421581 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:10.421532    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 20:15:10.421655 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:10.421635    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Sep 09 20:15:10.421733 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:10.421673    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:15:10.421733 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:10.421568    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 20:15:10.488010 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:10.487987    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/597e4424-e45a-46c3-b79a-9940405f6161-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-k2dvq\" (UID: \"597e4424-e45a-46c3-b79a-9940405f6161\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-k2dvq"
Sep 09 20:15:10.488099 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:10.488045    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/597e4424-e45a-46c3-b79a-9940405f6161-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-k2dvq\" (UID: \"597e4424-e45a-46c3-b79a-9940405f6161\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-k2dvq"
Sep 09 20:15:10.488143 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:10.488102    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-689q6\" (UniqueName: \"kubernetes.io/projected/597e4424-e45a-46c3-b79a-9940405f6161-kube-api-access-689q6\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-k2dvq\" (UID: \"597e4424-e45a-46c3-b79a-9940405f6161\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-k2dvq"
Sep 09 20:15:10.488143 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:10.488124    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/597e4424-e45a-46c3-b79a-9940405f6161-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-k2dvq\" (UID: \"597e4424-e45a-46c3-b79a-9940405f6161\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-k2dvq"
Sep 09 20:15:10.488245 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:10.488180    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/597e4424-e45a-46c3-b79a-9940405f6161-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-k2dvq\" (UID: \"597e4424-e45a-46c3-b79a-9940405f6161\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-k2dvq"
Sep 09 20:15:10.589069 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:10.589043    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-689q6\" (UniqueName: \"kubernetes.io/projected/597e4424-e45a-46c3-b79a-9940405f6161-kube-api-access-689q6\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-k2dvq\" (UID: \"597e4424-e45a-46c3-b79a-9940405f6161\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-k2dvq"
Sep 09 20:15:10.589199 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:10.589075    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/597e4424-e45a-46c3-b79a-9940405f6161-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-k2dvq\" (UID: \"597e4424-e45a-46c3-b79a-9940405f6161\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-k2dvq"
Sep 09 20:15:10.589199 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:10.589096    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/597e4424-e45a-46c3-b79a-9940405f6161-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-k2dvq\" (UID: \"597e4424-e45a-46c3-b79a-9940405f6161\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-k2dvq"
Sep 09 20:15:10.589199 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:10.589113    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/597e4424-e45a-46c3-b79a-9940405f6161-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-k2dvq\" (UID: \"597e4424-e45a-46c3-b79a-9940405f6161\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-k2dvq"
Sep 09 20:15:10.589199 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:10.589140    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/597e4424-e45a-46c3-b79a-9940405f6161-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-k2dvq\" (UID: \"597e4424-e45a-46c3-b79a-9940405f6161\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-k2dvq"
Sep 09 20:15:10.589489 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:10.589472    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/597e4424-e45a-46c3-b79a-9940405f6161-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-k2dvq\" (UID: \"597e4424-e45a-46c3-b79a-9940405f6161\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-k2dvq"
Sep 09 20:15:10.589784 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:10.589765    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/597e4424-e45a-46c3-b79a-9940405f6161-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-k2dvq\" (UID: \"597e4424-e45a-46c3-b79a-9940405f6161\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-k2dvq"
Sep 09 20:15:10.589846 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:10.589782    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/597e4424-e45a-46c3-b79a-9940405f6161-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-k2dvq\" (UID: \"597e4424-e45a-46c3-b79a-9940405f6161\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-k2dvq"
Sep 09 20:15:10.591579 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:10.591560    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/597e4424-e45a-46c3-b79a-9940405f6161-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-k2dvq\" (UID: \"597e4424-e45a-46c3-b79a-9940405f6161\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-k2dvq"
Sep 09 20:15:10.597605 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:10.597586    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-689q6\" (UniqueName: \"kubernetes.io/projected/597e4424-e45a-46c3-b79a-9940405f6161-kube-api-access-689q6\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-k2dvq\" (UID: \"597e4424-e45a-46c3-b79a-9940405f6161\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-k2dvq"
Sep 09 20:15:10.727357 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:10.727289    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-k2dvq"
Sep 09 20:15:11.023198 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:11.023175    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-k2dvq"]
Sep 09 20:15:11.025379 ip-10-0-132-226 kubenswrapper[2523]: W0909 20:15:11.025350    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod597e4424_e45a_46c3_b79a_9940405f6161.slice/crio-1b0af389b30ead3d0e04501a841c6454f29a93a595d6eaeffed4f9c6b45c61e7 WatchSource:0}: Error finding container 1b0af389b30ead3d0e04501a841c6454f29a93a595d6eaeffed4f9c6b45c61e7: Status 404 returned error can't find the container with id 1b0af389b30ead3d0e04501a841c6454f29a93a595d6eaeffed4f9c6b45c61e7
Sep 09 20:15:11.261450 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:11.261413    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-k2dvq" event={"ID":"597e4424-e45a-46c3-b79a-9940405f6161","Type":"ContainerStarted","Data":"b6277a2397ec21964aa83622d0eeb1cd1f480355cdbe654dc364ec28a3ca2da9"}
Sep 09 20:15:11.261605 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:11.261480    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-k2dvq" event={"ID":"597e4424-e45a-46c3-b79a-9940405f6161","Type":"ContainerStarted","Data":"1b0af389b30ead3d0e04501a841c6454f29a93a595d6eaeffed4f9c6b45c61e7"}
Sep 09 20:15:31.336984 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:31.336943    2523 generic.go:356] "Generic (PLEG): container finished" podID="597e4424-e45a-46c3-b79a-9940405f6161" containerID="b6277a2397ec21964aa83622d0eeb1cd1f480355cdbe654dc364ec28a3ca2da9" exitCode=0
Sep 09 20:15:31.337395 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:31.337019    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-k2dvq" event={"ID":"597e4424-e45a-46c3-b79a-9940405f6161","Type":"ContainerDied","Data":"b6277a2397ec21964aa83622d0eeb1cd1f480355cdbe654dc364ec28a3ca2da9"}
Sep 09 20:15:32.343899 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:32.343858    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-k2dvq" event={"ID":"597e4424-e45a-46c3-b79a-9940405f6161","Type":"ContainerStarted","Data":"1d78936a0da4931565beaa421cc9a00f3e40534fd36e0ae9a0f38b9dd52bbbf0"}
Sep 09 20:15:32.344388 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:32.343927    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-k2dvq" event={"ID":"597e4424-e45a-46c3-b79a-9940405f6161","Type":"ContainerStarted","Data":"30f71e9344ef24032b9e070b2207e3981063d5923cbc7867d8e8a3e049478581"}
Sep 09 20:15:32.344388 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:32.344111    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-k2dvq"
Sep 09 20:15:32.364640 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:32.364583    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-k2dvq" podStartSLOduration=2.365875126 podStartE2EDuration="22.364566036s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.99869091s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:15:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:15:32.361899637 +0000 UTC m=+1237.033918645" watchObservedRunningTime="2026-09-09 20:15:32.364566036 +0000 UTC m=+1237.036585034"
Sep 09 20:15:33.348309 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:33.348274    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-k2dvq"
Sep 09 20:15:39.358318 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:15:39.358281    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-k2dvq"
Sep 09 20:17:02.811347 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:02.807118    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-k2dvq"
Sep 09 20:17:13.845308 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:13.845269    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-k2dvq" podUID="597e4424-e45a-46c3-b79a-9940405f6161" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.25:8080/v2/models/isvc-autogluon-v2/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 20:17:16.594412 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:16.594377    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-k2dvq"]
Sep 09 20:17:16.594819 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:16.594673    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-k2dvq" podUID="597e4424-e45a-46c3-b79a-9940405f6161" containerName="kserve-container" containerID="cri-o://30f71e9344ef24032b9e070b2207e3981063d5923cbc7867d8e8a3e049478581" gracePeriod=30
Sep 09 20:17:16.594819 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:16.594723    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-k2dvq" podUID="597e4424-e45a-46c3-b79a-9940405f6161" containerName="kube-rbac-proxy" containerID="cri-o://1d78936a0da4931565beaa421cc9a00f3e40534fd36e0ae9a0f38b9dd52bbbf0" gracePeriod=30
Sep 09 20:17:16.722610 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:16.722583    2523 generic.go:356] "Generic (PLEG): container finished" podID="597e4424-e45a-46c3-b79a-9940405f6161" containerID="1d78936a0da4931565beaa421cc9a00f3e40534fd36e0ae9a0f38b9dd52bbbf0" exitCode=2
Sep 09 20:17:16.722705 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:16.722655    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-k2dvq" event={"ID":"597e4424-e45a-46c3-b79a-9940405f6161","Type":"ContainerDied","Data":"1d78936a0da4931565beaa421cc9a00f3e40534fd36e0ae9a0f38b9dd52bbbf0"}
Sep 09 20:17:17.728869 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:17.728828    2523 generic.go:356] "Generic (PLEG): container finished" podID="597e4424-e45a-46c3-b79a-9940405f6161" containerID="30f71e9344ef24032b9e070b2207e3981063d5923cbc7867d8e8a3e049478581" exitCode=0
Sep 09 20:17:17.729203 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:17.728888    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-k2dvq" event={"ID":"597e4424-e45a-46c3-b79a-9940405f6161","Type":"ContainerDied","Data":"30f71e9344ef24032b9e070b2207e3981063d5923cbc7867d8e8a3e049478581"}
Sep 09 20:17:17.837376 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:17.837352    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-k2dvq"
Sep 09 20:17:17.956474 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:17.956418    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/597e4424-e45a-46c3-b79a-9940405f6161-kube-rbac-proxy-sar-config\") pod \"597e4424-e45a-46c3-b79a-9940405f6161\" (UID: \"597e4424-e45a-46c3-b79a-9940405f6161\") "
Sep 09 20:17:17.956474 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:17.956461    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/597e4424-e45a-46c3-b79a-9940405f6161-proxy-tls\") pod \"597e4424-e45a-46c3-b79a-9940405f6161\" (UID: \"597e4424-e45a-46c3-b79a-9940405f6161\") "
Sep 09 20:17:17.956615 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:17.956490    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-689q6\" (UniqueName: \"kubernetes.io/projected/597e4424-e45a-46c3-b79a-9940405f6161-kube-api-access-689q6\") pod \"597e4424-e45a-46c3-b79a-9940405f6161\" (UID: \"597e4424-e45a-46c3-b79a-9940405f6161\") "
Sep 09 20:17:17.956615 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:17.956537    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/597e4424-e45a-46c3-b79a-9940405f6161-cabundle-cert\") pod \"597e4424-e45a-46c3-b79a-9940405f6161\" (UID: \"597e4424-e45a-46c3-b79a-9940405f6161\") "
Sep 09 20:17:17.956615 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:17.956603    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/597e4424-e45a-46c3-b79a-9940405f6161-kserve-provision-location\") pod \"597e4424-e45a-46c3-b79a-9940405f6161\" (UID: \"597e4424-e45a-46c3-b79a-9940405f6161\") "
Sep 09 20:17:17.956903 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:17.956878    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/597e4424-e45a-46c3-b79a-9940405f6161-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "597e4424-e45a-46c3-b79a-9940405f6161" (UID: "597e4424-e45a-46c3-b79a-9940405f6161"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:17:17.957021 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:17.956978    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/597e4424-e45a-46c3-b79a-9940405f6161-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "597e4424-e45a-46c3-b79a-9940405f6161" (UID: "597e4424-e45a-46c3-b79a-9940405f6161"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:17:17.958673 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:17.958654    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/597e4424-e45a-46c3-b79a-9940405f6161-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "597e4424-e45a-46c3-b79a-9940405f6161" (UID: "597e4424-e45a-46c3-b79a-9940405f6161"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:17:17.958770 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:17.958756    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/597e4424-e45a-46c3-b79a-9940405f6161-kube-api-access-689q6" (OuterVolumeSpecName: "kube-api-access-689q6") pod "597e4424-e45a-46c3-b79a-9940405f6161" (UID: "597e4424-e45a-46c3-b79a-9940405f6161"). InnerVolumeSpecName "kube-api-access-689q6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:17:17.961202 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:17.961177    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/597e4424-e45a-46c3-b79a-9940405f6161-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "597e4424-e45a-46c3-b79a-9940405f6161" (UID: "597e4424-e45a-46c3-b79a-9940405f6161"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:17:18.057604 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:18.057580    2523 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/597e4424-e45a-46c3-b79a-9940405f6161-proxy-tls\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:17:18.057604 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:18.057600    2523 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-689q6\" (UniqueName: \"kubernetes.io/projected/597e4424-e45a-46c3-b79a-9940405f6161-kube-api-access-689q6\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:17:18.057719 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:18.057609    2523 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/597e4424-e45a-46c3-b79a-9940405f6161-cabundle-cert\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:17:18.057719 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:18.057617    2523 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/597e4424-e45a-46c3-b79a-9940405f6161-kserve-provision-location\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:17:18.057719 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:18.057624    2523 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/597e4424-e45a-46c3-b79a-9940405f6161-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:17:18.734451 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:18.734425    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-k2dvq" event={"ID":"597e4424-e45a-46c3-b79a-9940405f6161","Type":"ContainerDied","Data":"1b0af389b30ead3d0e04501a841c6454f29a93a595d6eaeffed4f9c6b45c61e7"}
Sep 09 20:17:18.734835 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:18.734487    2523 scope.go:117] "RemoveContainer" containerID="1d78936a0da4931565beaa421cc9a00f3e40534fd36e0ae9a0f38b9dd52bbbf0"
Sep 09 20:17:18.734835 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:18.734500    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-k2dvq"
Sep 09 20:17:18.742686 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:18.742669    2523 scope.go:117] "RemoveContainer" containerID="30f71e9344ef24032b9e070b2207e3981063d5923cbc7867d8e8a3e049478581"
Sep 09 20:17:18.749821 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:18.749805    2523 scope.go:117] "RemoveContainer" containerID="b6277a2397ec21964aa83622d0eeb1cd1f480355cdbe654dc364ec28a3ca2da9"
Sep 09 20:17:18.761802 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:18.761782    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-k2dvq"]
Sep 09 20:17:18.769807 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:18.769787    2523 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-k2dvq"]
Sep 09 20:17:19.807031 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:19.807000    2523 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="597e4424-e45a-46c3-b79a-9940405f6161" path="/var/lib/kubelet/pods/597e4424-e45a-46c3-b79a-9940405f6161/volumes"
Sep 09 20:17:21.767032 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:21.766992    2523 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t"]
Sep 09 20:17:21.767621 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:21.767600    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="597e4424-e45a-46c3-b79a-9940405f6161" containerName="kserve-container"
Sep 09 20:17:21.767686 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:21.767624    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="597e4424-e45a-46c3-b79a-9940405f6161" containerName="kserve-container"
Sep 09 20:17:21.767686 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:21.767638    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="597e4424-e45a-46c3-b79a-9940405f6161" containerName="storage-initializer"
Sep 09 20:17:21.767686 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:21.767647    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="597e4424-e45a-46c3-b79a-9940405f6161" containerName="storage-initializer"
Sep 09 20:17:21.767686 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:21.767667    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="597e4424-e45a-46c3-b79a-9940405f6161" containerName="kube-rbac-proxy"
Sep 09 20:17:21.767686 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:21.767678    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="597e4424-e45a-46c3-b79a-9940405f6161" containerName="kube-rbac-proxy"
Sep 09 20:17:21.767905 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:21.767809    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="597e4424-e45a-46c3-b79a-9940405f6161" containerName="kserve-container"
Sep 09 20:17:21.767905 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:21.767821    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="597e4424-e45a-46c3-b79a-9940405f6161" containerName="kube-rbac-proxy"
Sep 09 20:17:21.772787 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:21.772765    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t"
Sep 09 20:17:21.775358 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:21.775334    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Sep 09 20:17:21.775474 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:21.775439    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:17:21.775574 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:21.775559    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 20:17:21.775625 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:21.775564    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Sep 09 20:17:21.776411 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:21.776394    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-ks7mb\""
Sep 09 20:17:21.776411 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:21.776409    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 20:17:21.776554 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:21.776448    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 20:17:21.781732 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:21.781710    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t"]
Sep 09 20:17:21.879489 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:21.879461    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/07673bbf-0ab7-45cb-ba27-ba6cc385cc10-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t\" (UID: \"07673bbf-0ab7-45cb-ba27-ba6cc385cc10\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t"
Sep 09 20:17:21.879598 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:21.879498    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/07673bbf-0ab7-45cb-ba27-ba6cc385cc10-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t\" (UID: \"07673bbf-0ab7-45cb-ba27-ba6cc385cc10\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t"
Sep 09 20:17:21.879598 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:21.879528    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/07673bbf-0ab7-45cb-ba27-ba6cc385cc10-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t\" (UID: \"07673bbf-0ab7-45cb-ba27-ba6cc385cc10\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t"
Sep 09 20:17:21.879598 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:21.879583    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfrc8\" (UniqueName: \"kubernetes.io/projected/07673bbf-0ab7-45cb-ba27-ba6cc385cc10-kube-api-access-lfrc8\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t\" (UID: \"07673bbf-0ab7-45cb-ba27-ba6cc385cc10\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t"
Sep 09 20:17:21.879720 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:21.879615    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07673bbf-0ab7-45cb-ba27-ba6cc385cc10-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t\" (UID: \"07673bbf-0ab7-45cb-ba27-ba6cc385cc10\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t"
Sep 09 20:17:21.980725 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:21.980696    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/07673bbf-0ab7-45cb-ba27-ba6cc385cc10-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t\" (UID: \"07673bbf-0ab7-45cb-ba27-ba6cc385cc10\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t"
Sep 09 20:17:21.980814 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:21.980738    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfrc8\" (UniqueName: \"kubernetes.io/projected/07673bbf-0ab7-45cb-ba27-ba6cc385cc10-kube-api-access-lfrc8\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t\" (UID: \"07673bbf-0ab7-45cb-ba27-ba6cc385cc10\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t"
Sep 09 20:17:21.980814 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:21.980768    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07673bbf-0ab7-45cb-ba27-ba6cc385cc10-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t\" (UID: \"07673bbf-0ab7-45cb-ba27-ba6cc385cc10\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t"
Sep 09 20:17:21.980934 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:21.980835    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/07673bbf-0ab7-45cb-ba27-ba6cc385cc10-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t\" (UID: \"07673bbf-0ab7-45cb-ba27-ba6cc385cc10\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t"
Sep 09 20:17:21.980934 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:21.980863    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/07673bbf-0ab7-45cb-ba27-ba6cc385cc10-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t\" (UID: \"07673bbf-0ab7-45cb-ba27-ba6cc385cc10\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t"
Sep 09 20:17:21.980934 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:17:21.980927    2523 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-serving-cert: secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found
Sep 09 20:17:21.981082 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:17:21.980993    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07673bbf-0ab7-45cb-ba27-ba6cc385cc10-proxy-tls podName:07673bbf-0ab7-45cb-ba27-ba6cc385cc10 nodeName:}" failed. No retries permitted until 2026-09-09 20:17:22.480976358 +0000 UTC m=+1347.152995518 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/07673bbf-0ab7-45cb-ba27-ba6cc385cc10-proxy-tls") pod "isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t" (UID: "07673bbf-0ab7-45cb-ba27-ba6cc385cc10") : secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found
Sep 09 20:17:21.981261 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:21.981243    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/07673bbf-0ab7-45cb-ba27-ba6cc385cc10-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t\" (UID: \"07673bbf-0ab7-45cb-ba27-ba6cc385cc10\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t"
Sep 09 20:17:21.981459 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:21.981441    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/07673bbf-0ab7-45cb-ba27-ba6cc385cc10-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t\" (UID: \"07673bbf-0ab7-45cb-ba27-ba6cc385cc10\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t"
Sep 09 20:17:21.981495 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:21.981455    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/07673bbf-0ab7-45cb-ba27-ba6cc385cc10-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t\" (UID: \"07673bbf-0ab7-45cb-ba27-ba6cc385cc10\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t"
Sep 09 20:17:22.005494 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:22.005471    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfrc8\" (UniqueName: \"kubernetes.io/projected/07673bbf-0ab7-45cb-ba27-ba6cc385cc10-kube-api-access-lfrc8\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t\" (UID: \"07673bbf-0ab7-45cb-ba27-ba6cc385cc10\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t"
Sep 09 20:17:22.483578 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:22.483543    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07673bbf-0ab7-45cb-ba27-ba6cc385cc10-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t\" (UID: \"07673bbf-0ab7-45cb-ba27-ba6cc385cc10\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t"
Sep 09 20:17:22.485927 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:22.485904    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07673bbf-0ab7-45cb-ba27-ba6cc385cc10-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t\" (UID: \"07673bbf-0ab7-45cb-ba27-ba6cc385cc10\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t"
Sep 09 20:17:22.683141 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:22.683106    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t"
Sep 09 20:17:22.969755 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:22.969731    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t"]
Sep 09 20:17:22.970672 ip-10-0-132-226 kubenswrapper[2523]: W0909 20:17:22.970649    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07673bbf_0ab7_45cb_ba27_ba6cc385cc10.slice/crio-e451bf7de2f50cc43b9bf2f3cd56e31f6fe9f305c375a46124cec30577b797d8 WatchSource:0}: Error finding container e451bf7de2f50cc43b9bf2f3cd56e31f6fe9f305c375a46124cec30577b797d8: Status 404 returned error can't find the container with id e451bf7de2f50cc43b9bf2f3cd56e31f6fe9f305c375a46124cec30577b797d8
Sep 09 20:17:23.755598 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:23.755562    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t" event={"ID":"07673bbf-0ab7-45cb-ba27-ba6cc385cc10","Type":"ContainerStarted","Data":"f1d1211def067bb8181cb734a69b65a84530ed664268419b80b7bbe4c91c2db4"}
Sep 09 20:17:23.755752 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:23.755637    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t" event={"ID":"07673bbf-0ab7-45cb-ba27-ba6cc385cc10","Type":"ContainerStarted","Data":"e451bf7de2f50cc43b9bf2f3cd56e31f6fe9f305c375a46124cec30577b797d8"}
Sep 09 20:17:53.864079 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:53.864049    2523 generic.go:356] "Generic (PLEG): container finished" podID="07673bbf-0ab7-45cb-ba27-ba6cc385cc10" containerID="f1d1211def067bb8181cb734a69b65a84530ed664268419b80b7bbe4c91c2db4" exitCode=0
Sep 09 20:17:53.864454 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:53.864118    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t" event={"ID":"07673bbf-0ab7-45cb-ba27-ba6cc385cc10","Type":"ContainerDied","Data":"f1d1211def067bb8181cb734a69b65a84530ed664268419b80b7bbe4c91c2db4"}
Sep 09 20:17:54.189954 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:54.189871    2523 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-f7v6p"]
Sep 09 20:17:54.192081 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:54.192066    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-f7v6p"
Sep 09 20:17:54.194323 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:54.194302    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
Sep 09 20:17:54.194496 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:54.194479    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
Sep 09 20:17:54.203601 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:54.203580    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-f7v6p"]
Sep 09 20:17:54.306772 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:54.306746    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f670a4b2-cef5-4d1f-8295-5ed166e49fd0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-c6b88b959-f7v6p\" (UID: \"f670a4b2-cef5-4d1f-8295-5ed166e49fd0\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-f7v6p"
Sep 09 20:17:54.306945 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:54.306778    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f670a4b2-cef5-4d1f-8295-5ed166e49fd0-proxy-tls\") pod \"isvc-sklearn-predictor-c6b88b959-f7v6p\" (UID: \"f670a4b2-cef5-4d1f-8295-5ed166e49fd0\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-f7v6p"
Sep 09 20:17:54.306945 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:54.306796    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f670a4b2-cef5-4d1f-8295-5ed166e49fd0-kserve-provision-location\") pod \"isvc-sklearn-predictor-c6b88b959-f7v6p\" (UID: \"f670a4b2-cef5-4d1f-8295-5ed166e49fd0\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-f7v6p"
Sep 09 20:17:54.306945 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:54.306854    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f670a4b2-cef5-4d1f-8295-5ed166e49fd0-cabundle-cert\") pod \"isvc-sklearn-predictor-c6b88b959-f7v6p\" (UID: \"f670a4b2-cef5-4d1f-8295-5ed166e49fd0\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-f7v6p"
Sep 09 20:17:54.306945 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:54.306913    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgq9k\" (UniqueName: \"kubernetes.io/projected/f670a4b2-cef5-4d1f-8295-5ed166e49fd0-kube-api-access-pgq9k\") pod \"isvc-sklearn-predictor-c6b88b959-f7v6p\" (UID: \"f670a4b2-cef5-4d1f-8295-5ed166e49fd0\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-f7v6p"
Sep 09 20:17:54.407826 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:54.407798    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f670a4b2-cef5-4d1f-8295-5ed166e49fd0-cabundle-cert\") pod \"isvc-sklearn-predictor-c6b88b959-f7v6p\" (UID: \"f670a4b2-cef5-4d1f-8295-5ed166e49fd0\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-f7v6p"
Sep 09 20:17:54.407927 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:54.407841    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgq9k\" (UniqueName: \"kubernetes.io/projected/f670a4b2-cef5-4d1f-8295-5ed166e49fd0-kube-api-access-pgq9k\") pod \"isvc-sklearn-predictor-c6b88b959-f7v6p\" (UID: \"f670a4b2-cef5-4d1f-8295-5ed166e49fd0\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-f7v6p"
Sep 09 20:17:54.407927 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:54.407874    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f670a4b2-cef5-4d1f-8295-5ed166e49fd0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-c6b88b959-f7v6p\" (UID: \"f670a4b2-cef5-4d1f-8295-5ed166e49fd0\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-f7v6p"
Sep 09 20:17:54.407927 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:54.407892    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f670a4b2-cef5-4d1f-8295-5ed166e49fd0-proxy-tls\") pod \"isvc-sklearn-predictor-c6b88b959-f7v6p\" (UID: \"f670a4b2-cef5-4d1f-8295-5ed166e49fd0\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-f7v6p"
Sep 09 20:17:54.408086 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:17:54.407966    2523 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-predictor-serving-cert: secret "isvc-sklearn-predictor-serving-cert" not found
Sep 09 20:17:54.408086 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:54.407989    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f670a4b2-cef5-4d1f-8295-5ed166e49fd0-kserve-provision-location\") pod \"isvc-sklearn-predictor-c6b88b959-f7v6p\" (UID: \"f670a4b2-cef5-4d1f-8295-5ed166e49fd0\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-f7v6p"
Sep 09 20:17:54.408086 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:17:54.408024    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f670a4b2-cef5-4d1f-8295-5ed166e49fd0-proxy-tls podName:f670a4b2-cef5-4d1f-8295-5ed166e49fd0 nodeName:}" failed. No retries permitted until 2026-09-09 20:17:54.908004348 +0000 UTC m=+1379.580023342 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f670a4b2-cef5-4d1f-8295-5ed166e49fd0-proxy-tls") pod "isvc-sklearn-predictor-c6b88b959-f7v6p" (UID: "f670a4b2-cef5-4d1f-8295-5ed166e49fd0") : secret "isvc-sklearn-predictor-serving-cert" not found
Sep 09 20:17:54.408381 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:54.408359    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f670a4b2-cef5-4d1f-8295-5ed166e49fd0-kserve-provision-location\") pod \"isvc-sklearn-predictor-c6b88b959-f7v6p\" (UID: \"f670a4b2-cef5-4d1f-8295-5ed166e49fd0\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-f7v6p"
Sep 09 20:17:54.408600 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:54.408579    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f670a4b2-cef5-4d1f-8295-5ed166e49fd0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-c6b88b959-f7v6p\" (UID: \"f670a4b2-cef5-4d1f-8295-5ed166e49fd0\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-f7v6p"
Sep 09 20:17:54.408666 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:54.408624    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f670a4b2-cef5-4d1f-8295-5ed166e49fd0-cabundle-cert\") pod \"isvc-sklearn-predictor-c6b88b959-f7v6p\" (UID: \"f670a4b2-cef5-4d1f-8295-5ed166e49fd0\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-f7v6p"
Sep 09 20:17:54.416980 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:54.416955    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgq9k\" (UniqueName: \"kubernetes.io/projected/f670a4b2-cef5-4d1f-8295-5ed166e49fd0-kube-api-access-pgq9k\") pod \"isvc-sklearn-predictor-c6b88b959-f7v6p\" (UID: \"f670a4b2-cef5-4d1f-8295-5ed166e49fd0\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-f7v6p"
Sep 09 20:17:54.869990 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:54.869959    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t" event={"ID":"07673bbf-0ab7-45cb-ba27-ba6cc385cc10","Type":"ContainerStarted","Data":"a832cb7c312bd41912751a8af9f11191824476540a7a8b2bba18f11f7595ee6c"}
Sep 09 20:17:54.870353 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:54.870017    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t" event={"ID":"07673bbf-0ab7-45cb-ba27-ba6cc385cc10","Type":"ContainerStarted","Data":"cd97815d9516b53a2a27f3d51b3a240e1d9d61f7684de647f479602fb9814e85"}
Sep 09 20:17:54.870353 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:54.870295    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t"
Sep 09 20:17:54.896248 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:54.896202    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t" podStartSLOduration=3.45912164 podStartE2EDuration="33.896188019s" totalImagesPullingTime="0s" totalInitContainerRuntime="30.437066379s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:17:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:17:54.895207066 +0000 UTC m=+1379.567226062" watchObservedRunningTime="2026-09-09 20:17:54.896188019 +0000 UTC m=+1379.568207055"
Sep 09 20:17:54.913032 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:54.913004    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f670a4b2-cef5-4d1f-8295-5ed166e49fd0-proxy-tls\") pod \"isvc-sklearn-predictor-c6b88b959-f7v6p\" (UID: \"f670a4b2-cef5-4d1f-8295-5ed166e49fd0\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-f7v6p"
Sep 09 20:17:54.915486 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:54.915466    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f670a4b2-cef5-4d1f-8295-5ed166e49fd0-proxy-tls\") pod \"isvc-sklearn-predictor-c6b88b959-f7v6p\" (UID: \"f670a4b2-cef5-4d1f-8295-5ed166e49fd0\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-f7v6p"
Sep 09 20:17:55.102052 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:55.102020    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-f7v6p"
Sep 09 20:17:55.382790 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:55.382767    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-f7v6p"]
Sep 09 20:17:55.384500 ip-10-0-132-226 kubenswrapper[2523]: W0909 20:17:55.384468    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf670a4b2_cef5_4d1f_8295_5ed166e49fd0.slice/crio-ac4d11e18ba9e075eb54bb727af9e2c2cffae84e81045ef247ef29220a28e1fa WatchSource:0}: Error finding container ac4d11e18ba9e075eb54bb727af9e2c2cffae84e81045ef247ef29220a28e1fa: Status 404 returned error can't find the container with id ac4d11e18ba9e075eb54bb727af9e2c2cffae84e81045ef247ef29220a28e1fa
Sep 09 20:17:55.875973 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:55.875937    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-f7v6p" event={"ID":"f670a4b2-cef5-4d1f-8295-5ed166e49fd0","Type":"ContainerStarted","Data":"16ff286957d3358e721e746161c7e17e6a01ca0b33dbb6a7038b2d3a1e98a220"}
Sep 09 20:17:55.876417 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:55.876001    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-f7v6p" event={"ID":"f670a4b2-cef5-4d1f-8295-5ed166e49fd0","Type":"ContainerStarted","Data":"ac4d11e18ba9e075eb54bb727af9e2c2cffae84e81045ef247ef29220a28e1fa"}
Sep 09 20:17:55.876417 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:55.876331    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t"
Sep 09 20:17:59.894289 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:59.894255    2523 generic.go:356] "Generic (PLEG): container finished" podID="f670a4b2-cef5-4d1f-8295-5ed166e49fd0" containerID="16ff286957d3358e721e746161c7e17e6a01ca0b33dbb6a7038b2d3a1e98a220" exitCode=0
Sep 09 20:17:59.894802 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:17:59.894342    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-f7v6p" event={"ID":"f670a4b2-cef5-4d1f-8295-5ed166e49fd0","Type":"ContainerDied","Data":"16ff286957d3358e721e746161c7e17e6a01ca0b33dbb6a7038b2d3a1e98a220"}
Sep 09 20:18:00.900768 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:18:00.900734    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-f7v6p" event={"ID":"f670a4b2-cef5-4d1f-8295-5ed166e49fd0","Type":"ContainerStarted","Data":"70f856e30368761a6e46f52d5205e4b56b2d66c3c10ccc154cf3500e21ce9d2d"}
Sep 09 20:18:00.901183 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:18:00.900794    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-f7v6p" event={"ID":"f670a4b2-cef5-4d1f-8295-5ed166e49fd0","Type":"ContainerStarted","Data":"3d2429073de7bc47787c02ae7c523f5d1c7a9e9cb48028cb05484644ff54bc5c"}
Sep 09 20:18:00.901183 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:18:00.900981    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-f7v6p"
Sep 09 20:18:00.926822 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:18:00.926770    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-f7v6p" podStartSLOduration=2.991880417 podStartE2EDuration="6.926755999s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.934875582s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:17:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:18:00.925594794 +0000 UTC m=+1385.597613788" watchObservedRunningTime="2026-09-09 20:18:00.926755999 +0000 UTC m=+1385.598774995"
Sep 09 20:18:01.885806 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:18:01.885775    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t"
Sep 09 20:18:01.904473 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:18:01.904446    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-f7v6p"
Sep 09 20:18:01.905649 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:18:01.905622    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-f7v6p" podUID="f670a4b2-cef5-4d1f-8295-5ed166e49fd0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 09 20:18:02.909133 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:18:02.909095    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-f7v6p" podUID="f670a4b2-cef5-4d1f-8295-5ed166e49fd0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 09 20:18:07.913640 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:18:07.913613    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-f7v6p"
Sep 09 20:18:07.914084 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:18:07.914053    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-f7v6p" podUID="f670a4b2-cef5-4d1f-8295-5ed166e49fd0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 09 20:18:17.914208 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:18:17.914140    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-f7v6p" podUID="f670a4b2-cef5-4d1f-8295-5ed166e49fd0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 09 20:18:27.914965 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:18:27.914924    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-f7v6p" podUID="f670a4b2-cef5-4d1f-8295-5ed166e49fd0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 09 20:18:37.914218 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:18:37.914179    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-f7v6p" podUID="f670a4b2-cef5-4d1f-8295-5ed166e49fd0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 09 20:18:47.914940 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:18:47.914859    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-f7v6p" podUID="f670a4b2-cef5-4d1f-8295-5ed166e49fd0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 09 20:18:57.914367 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:18:57.914328    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-f7v6p" podUID="f670a4b2-cef5-4d1f-8295-5ed166e49fd0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 09 20:19:07.914781 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:07.914751    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-f7v6p"
Sep 09 20:19:14.308004 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:14.307972    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-f7v6p"]
Sep 09 20:19:14.308384 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:14.308237    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-f7v6p" podUID="f670a4b2-cef5-4d1f-8295-5ed166e49fd0" containerName="kserve-container" containerID="cri-o://3d2429073de7bc47787c02ae7c523f5d1c7a9e9cb48028cb05484644ff54bc5c" gracePeriod=30
Sep 09 20:19:14.308384 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:14.308278    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-f7v6p" podUID="f670a4b2-cef5-4d1f-8295-5ed166e49fd0" containerName="kube-rbac-proxy" containerID="cri-o://70f856e30368761a6e46f52d5205e4b56b2d66c3c10ccc154cf3500e21ce9d2d" gracePeriod=30
Sep 09 20:19:15.174076 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:15.174044    2523 generic.go:356] "Generic (PLEG): container finished" podID="f670a4b2-cef5-4d1f-8295-5ed166e49fd0" containerID="70f856e30368761a6e46f52d5205e4b56b2d66c3c10ccc154cf3500e21ce9d2d" exitCode=2
Sep 09 20:19:15.174256 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:15.174115    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-f7v6p" event={"ID":"f670a4b2-cef5-4d1f-8295-5ed166e49fd0","Type":"ContainerDied","Data":"70f856e30368761a6e46f52d5205e4b56b2d66c3c10ccc154cf3500e21ce9d2d"}
Sep 09 20:19:17.909966 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:17.909930    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-f7v6p" podUID="f670a4b2-cef5-4d1f-8295-5ed166e49fd0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Sep 09 20:19:17.914262 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:17.914235    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-f7v6p" podUID="f670a4b2-cef5-4d1f-8295-5ed166e49fd0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 09 20:19:18.049365 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:18.049339    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-f7v6p"
Sep 09 20:19:18.176475 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:18.176415    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f670a4b2-cef5-4d1f-8295-5ed166e49fd0-kserve-provision-location\") pod \"f670a4b2-cef5-4d1f-8295-5ed166e49fd0\" (UID: \"f670a4b2-cef5-4d1f-8295-5ed166e49fd0\") "
Sep 09 20:19:18.176475 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:18.176456    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgq9k\" (UniqueName: \"kubernetes.io/projected/f670a4b2-cef5-4d1f-8295-5ed166e49fd0-kube-api-access-pgq9k\") pod \"f670a4b2-cef5-4d1f-8295-5ed166e49fd0\" (UID: \"f670a4b2-cef5-4d1f-8295-5ed166e49fd0\") "
Sep 09 20:19:18.176642 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:18.176476    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f670a4b2-cef5-4d1f-8295-5ed166e49fd0-kube-rbac-proxy-sar-config\") pod \"f670a4b2-cef5-4d1f-8295-5ed166e49fd0\" (UID: \"f670a4b2-cef5-4d1f-8295-5ed166e49fd0\") "
Sep 09 20:19:18.176642 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:18.176518    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f670a4b2-cef5-4d1f-8295-5ed166e49fd0-cabundle-cert\") pod \"f670a4b2-cef5-4d1f-8295-5ed166e49fd0\" (UID: \"f670a4b2-cef5-4d1f-8295-5ed166e49fd0\") "
Sep 09 20:19:18.176642 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:18.176549    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f670a4b2-cef5-4d1f-8295-5ed166e49fd0-proxy-tls\") pod \"f670a4b2-cef5-4d1f-8295-5ed166e49fd0\" (UID: \"f670a4b2-cef5-4d1f-8295-5ed166e49fd0\") "
Sep 09 20:19:18.176842 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:18.176814    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f670a4b2-cef5-4d1f-8295-5ed166e49fd0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f670a4b2-cef5-4d1f-8295-5ed166e49fd0" (UID: "f670a4b2-cef5-4d1f-8295-5ed166e49fd0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:19:18.176917 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:18.176882    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f670a4b2-cef5-4d1f-8295-5ed166e49fd0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f670a4b2-cef5-4d1f-8295-5ed166e49fd0" (UID: "f670a4b2-cef5-4d1f-8295-5ed166e49fd0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:19:18.176968 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:18.176924    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f670a4b2-cef5-4d1f-8295-5ed166e49fd0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f670a4b2-cef5-4d1f-8295-5ed166e49fd0" (UID: "f670a4b2-cef5-4d1f-8295-5ed166e49fd0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:19:18.178580 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:18.178560    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f670a4b2-cef5-4d1f-8295-5ed166e49fd0-kube-api-access-pgq9k" (OuterVolumeSpecName: "kube-api-access-pgq9k") pod "f670a4b2-cef5-4d1f-8295-5ed166e49fd0" (UID: "f670a4b2-cef5-4d1f-8295-5ed166e49fd0"). InnerVolumeSpecName "kube-api-access-pgq9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:19:18.178736 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:18.178716    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f670a4b2-cef5-4d1f-8295-5ed166e49fd0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f670a4b2-cef5-4d1f-8295-5ed166e49fd0" (UID: "f670a4b2-cef5-4d1f-8295-5ed166e49fd0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:19:18.188290 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:18.188268    2523 generic.go:356] "Generic (PLEG): container finished" podID="f670a4b2-cef5-4d1f-8295-5ed166e49fd0" containerID="3d2429073de7bc47787c02ae7c523f5d1c7a9e9cb48028cb05484644ff54bc5c" exitCode=0
Sep 09 20:19:18.188381 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:18.188318    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-f7v6p" event={"ID":"f670a4b2-cef5-4d1f-8295-5ed166e49fd0","Type":"ContainerDied","Data":"3d2429073de7bc47787c02ae7c523f5d1c7a9e9cb48028cb05484644ff54bc5c"}
Sep 09 20:19:18.188381 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:18.188321    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-f7v6p"
Sep 09 20:19:18.188381 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:18.188362    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-f7v6p" event={"ID":"f670a4b2-cef5-4d1f-8295-5ed166e49fd0","Type":"ContainerDied","Data":"ac4d11e18ba9e075eb54bb727af9e2c2cffae84e81045ef247ef29220a28e1fa"}
Sep 09 20:19:18.188489 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:18.188387    2523 scope.go:117] "RemoveContainer" containerID="70f856e30368761a6e46f52d5205e4b56b2d66c3c10ccc154cf3500e21ce9d2d"
Sep 09 20:19:18.196196 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:18.196143    2523 scope.go:117] "RemoveContainer" containerID="3d2429073de7bc47787c02ae7c523f5d1c7a9e9cb48028cb05484644ff54bc5c"
Sep 09 20:19:18.203198 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:18.203179    2523 scope.go:117] "RemoveContainer" containerID="16ff286957d3358e721e746161c7e17e6a01ca0b33dbb6a7038b2d3a1e98a220"
Sep 09 20:19:18.209980 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:18.209964    2523 scope.go:117] "RemoveContainer" containerID="70f856e30368761a6e46f52d5205e4b56b2d66c3c10ccc154cf3500e21ce9d2d"
Sep 09 20:19:18.210358 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:19:18.210338    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70f856e30368761a6e46f52d5205e4b56b2d66c3c10ccc154cf3500e21ce9d2d\": container with ID starting with 70f856e30368761a6e46f52d5205e4b56b2d66c3c10ccc154cf3500e21ce9d2d not found: ID does not exist" containerID="70f856e30368761a6e46f52d5205e4b56b2d66c3c10ccc154cf3500e21ce9d2d"
Sep 09 20:19:18.210421 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:18.210365    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70f856e30368761a6e46f52d5205e4b56b2d66c3c10ccc154cf3500e21ce9d2d"} err="failed to get container status \"70f856e30368761a6e46f52d5205e4b56b2d66c3c10ccc154cf3500e21ce9d2d\": rpc error: code = NotFound desc = could not find container \"70f856e30368761a6e46f52d5205e4b56b2d66c3c10ccc154cf3500e21ce9d2d\": container with ID starting with 70f856e30368761a6e46f52d5205e4b56b2d66c3c10ccc154cf3500e21ce9d2d not found: ID does not exist"
Sep 09 20:19:18.210421 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:18.210382    2523 scope.go:117] "RemoveContainer" containerID="3d2429073de7bc47787c02ae7c523f5d1c7a9e9cb48028cb05484644ff54bc5c"
Sep 09 20:19:18.210606 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:19:18.210592    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d2429073de7bc47787c02ae7c523f5d1c7a9e9cb48028cb05484644ff54bc5c\": container with ID starting with 3d2429073de7bc47787c02ae7c523f5d1c7a9e9cb48028cb05484644ff54bc5c not found: ID does not exist" containerID="3d2429073de7bc47787c02ae7c523f5d1c7a9e9cb48028cb05484644ff54bc5c"
Sep 09 20:19:18.210647 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:18.210609    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d2429073de7bc47787c02ae7c523f5d1c7a9e9cb48028cb05484644ff54bc5c"} err="failed to get container status \"3d2429073de7bc47787c02ae7c523f5d1c7a9e9cb48028cb05484644ff54bc5c\": rpc error: code = NotFound desc = could not find container \"3d2429073de7bc47787c02ae7c523f5d1c7a9e9cb48028cb05484644ff54bc5c\": container with ID starting with 3d2429073de7bc47787c02ae7c523f5d1c7a9e9cb48028cb05484644ff54bc5c not found: ID does not exist"
Sep 09 20:19:18.210647 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:18.210622    2523 scope.go:117] "RemoveContainer" containerID="16ff286957d3358e721e746161c7e17e6a01ca0b33dbb6a7038b2d3a1e98a220"
Sep 09 20:19:18.210840 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:19:18.210824    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16ff286957d3358e721e746161c7e17e6a01ca0b33dbb6a7038b2d3a1e98a220\": container with ID starting with 16ff286957d3358e721e746161c7e17e6a01ca0b33dbb6a7038b2d3a1e98a220 not found: ID does not exist" containerID="16ff286957d3358e721e746161c7e17e6a01ca0b33dbb6a7038b2d3a1e98a220"
Sep 09 20:19:18.210876 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:18.210845    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16ff286957d3358e721e746161c7e17e6a01ca0b33dbb6a7038b2d3a1e98a220"} err="failed to get container status \"16ff286957d3358e721e746161c7e17e6a01ca0b33dbb6a7038b2d3a1e98a220\": rpc error: code = NotFound desc = could not find container \"16ff286957d3358e721e746161c7e17e6a01ca0b33dbb6a7038b2d3a1e98a220\": container with ID starting with 16ff286957d3358e721e746161c7e17e6a01ca0b33dbb6a7038b2d3a1e98a220 not found: ID does not exist"
Sep 09 20:19:18.213668 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:18.213648    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-f7v6p"]
Sep 09 20:19:18.217610 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:18.217591    2523 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-c6b88b959-f7v6p"]
Sep 09 20:19:18.277118 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:18.277100    2523 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f670a4b2-cef5-4d1f-8295-5ed166e49fd0-cabundle-cert\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:19:18.277118 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:18.277119    2523 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f670a4b2-cef5-4d1f-8295-5ed166e49fd0-proxy-tls\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:19:18.277239 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:18.277128    2523 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f670a4b2-cef5-4d1f-8295-5ed166e49fd0-kserve-provision-location\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:19:18.277239 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:18.277136    2523 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgq9k\" (UniqueName: \"kubernetes.io/projected/f670a4b2-cef5-4d1f-8295-5ed166e49fd0-kube-api-access-pgq9k\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:19:18.277239 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:18.277145    2523 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f670a4b2-cef5-4d1f-8295-5ed166e49fd0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:19:19.807553 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:19.807524    2523 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f670a4b2-cef5-4d1f-8295-5ed166e49fd0" path="/var/lib/kubelet/pods/f670a4b2-cef5-4d1f-8295-5ed166e49fd0/volumes"
Sep 09 20:19:31.811635 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:31.811606    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t"
Sep 09 20:19:42.849309 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:42.849266    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t" podUID="07673bbf-0ab7-45cb-ba27-ba6cc385cc10" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.26:8080/v2/models/isvc-autogluon-v2-variants/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 20:19:48.547099 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:48.547059    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t"]
Sep 09 20:19:48.547575 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:48.547382    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t" podUID="07673bbf-0ab7-45cb-ba27-ba6cc385cc10" containerName="kserve-container" containerID="cri-o://cd97815d9516b53a2a27f3d51b3a240e1d9d61f7684de647f479602fb9814e85" gracePeriod=30
Sep 09 20:19:48.547575 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:48.547445    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t" podUID="07673bbf-0ab7-45cb-ba27-ba6cc385cc10" containerName="kube-rbac-proxy" containerID="cri-o://a832cb7c312bd41912751a8af9f11191824476540a7a8b2bba18f11f7595ee6c" gracePeriod=30
Sep 09 20:19:49.305860 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:49.305816    2523 generic.go:356] "Generic (PLEG): container finished" podID="07673bbf-0ab7-45cb-ba27-ba6cc385cc10" containerID="a832cb7c312bd41912751a8af9f11191824476540a7a8b2bba18f11f7595ee6c" exitCode=2
Sep 09 20:19:49.306024 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:49.305883    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t" event={"ID":"07673bbf-0ab7-45cb-ba27-ba6cc385cc10","Type":"ContainerDied","Data":"a832cb7c312bd41912751a8af9f11191824476540a7a8b2bba18f11f7595ee6c"}
Sep 09 20:19:49.784970 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:49.784944    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t"
Sep 09 20:19:49.895669 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:49.895609    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/07673bbf-0ab7-45cb-ba27-ba6cc385cc10-kube-rbac-proxy-sar-config\") pod \"07673bbf-0ab7-45cb-ba27-ba6cc385cc10\" (UID: \"07673bbf-0ab7-45cb-ba27-ba6cc385cc10\") "
Sep 09 20:19:49.895669 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:49.895642    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/07673bbf-0ab7-45cb-ba27-ba6cc385cc10-cabundle-cert\") pod \"07673bbf-0ab7-45cb-ba27-ba6cc385cc10\" (UID: \"07673bbf-0ab7-45cb-ba27-ba6cc385cc10\") "
Sep 09 20:19:49.895859 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:49.895674    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/07673bbf-0ab7-45cb-ba27-ba6cc385cc10-kserve-provision-location\") pod \"07673bbf-0ab7-45cb-ba27-ba6cc385cc10\" (UID: \"07673bbf-0ab7-45cb-ba27-ba6cc385cc10\") "
Sep 09 20:19:49.895859 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:49.895713    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07673bbf-0ab7-45cb-ba27-ba6cc385cc10-proxy-tls\") pod \"07673bbf-0ab7-45cb-ba27-ba6cc385cc10\" (UID: \"07673bbf-0ab7-45cb-ba27-ba6cc385cc10\") "
Sep 09 20:19:49.895859 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:49.895747    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lfrc8\" (UniqueName: \"kubernetes.io/projected/07673bbf-0ab7-45cb-ba27-ba6cc385cc10-kube-api-access-lfrc8\") pod \"07673bbf-0ab7-45cb-ba27-ba6cc385cc10\" (UID: \"07673bbf-0ab7-45cb-ba27-ba6cc385cc10\") "
Sep 09 20:19:49.896046 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:49.896022    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07673bbf-0ab7-45cb-ba27-ba6cc385cc10-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "07673bbf-0ab7-45cb-ba27-ba6cc385cc10" (UID: "07673bbf-0ab7-45cb-ba27-ba6cc385cc10"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:19:49.896046 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:49.896031    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07673bbf-0ab7-45cb-ba27-ba6cc385cc10-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "07673bbf-0ab7-45cb-ba27-ba6cc385cc10" (UID: "07673bbf-0ab7-45cb-ba27-ba6cc385cc10"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:19:49.897879 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:49.897852    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07673bbf-0ab7-45cb-ba27-ba6cc385cc10-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "07673bbf-0ab7-45cb-ba27-ba6cc385cc10" (UID: "07673bbf-0ab7-45cb-ba27-ba6cc385cc10"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:19:49.897957 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:49.897928    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07673bbf-0ab7-45cb-ba27-ba6cc385cc10-kube-api-access-lfrc8" (OuterVolumeSpecName: "kube-api-access-lfrc8") pod "07673bbf-0ab7-45cb-ba27-ba6cc385cc10" (UID: "07673bbf-0ab7-45cb-ba27-ba6cc385cc10"). InnerVolumeSpecName "kube-api-access-lfrc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:19:49.900271 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:49.900251    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07673bbf-0ab7-45cb-ba27-ba6cc385cc10-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "07673bbf-0ab7-45cb-ba27-ba6cc385cc10" (UID: "07673bbf-0ab7-45cb-ba27-ba6cc385cc10"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:19:49.996404 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:49.996373    2523 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/07673bbf-0ab7-45cb-ba27-ba6cc385cc10-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:19:49.996404 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:49.996402    2523 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/07673bbf-0ab7-45cb-ba27-ba6cc385cc10-cabundle-cert\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:19:49.996404 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:49.996411    2523 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/07673bbf-0ab7-45cb-ba27-ba6cc385cc10-kserve-provision-location\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:19:49.996550 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:49.996420    2523 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/07673bbf-0ab7-45cb-ba27-ba6cc385cc10-proxy-tls\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:19:49.996550 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:49.996428    2523 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfrc8\" (UniqueName: \"kubernetes.io/projected/07673bbf-0ab7-45cb-ba27-ba6cc385cc10-kube-api-access-lfrc8\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:19:50.311791 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:50.311738    2523 generic.go:356] "Generic (PLEG): container finished" podID="07673bbf-0ab7-45cb-ba27-ba6cc385cc10" containerID="cd97815d9516b53a2a27f3d51b3a240e1d9d61f7684de647f479602fb9814e85" exitCode=0
Sep 09 20:19:50.311890 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:50.311791    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t"
Sep 09 20:19:50.311890 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:50.311826    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t" event={"ID":"07673bbf-0ab7-45cb-ba27-ba6cc385cc10","Type":"ContainerDied","Data":"cd97815d9516b53a2a27f3d51b3a240e1d9d61f7684de647f479602fb9814e85"}
Sep 09 20:19:50.312000 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:50.311899    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t" event={"ID":"07673bbf-0ab7-45cb-ba27-ba6cc385cc10","Type":"ContainerDied","Data":"e451bf7de2f50cc43b9bf2f3cd56e31f6fe9f305c375a46124cec30577b797d8"}
Sep 09 20:19:50.312000 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:50.311939    2523 scope.go:117] "RemoveContainer" containerID="a832cb7c312bd41912751a8af9f11191824476540a7a8b2bba18f11f7595ee6c"
Sep 09 20:19:50.320481 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:50.320460    2523 scope.go:117] "RemoveContainer" containerID="cd97815d9516b53a2a27f3d51b3a240e1d9d61f7684de647f479602fb9814e85"
Sep 09 20:19:50.328217 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:50.328198    2523 scope.go:117] "RemoveContainer" containerID="f1d1211def067bb8181cb734a69b65a84530ed664268419b80b7bbe4c91c2db4"
Sep 09 20:19:50.334898 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:50.334877    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t"]
Sep 09 20:19:50.335473 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:50.335457    2523 scope.go:117] "RemoveContainer" containerID="a832cb7c312bd41912751a8af9f11191824476540a7a8b2bba18f11f7595ee6c"
Sep 09 20:19:50.335744 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:19:50.335726    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a832cb7c312bd41912751a8af9f11191824476540a7a8b2bba18f11f7595ee6c\": container with ID starting with a832cb7c312bd41912751a8af9f11191824476540a7a8b2bba18f11f7595ee6c not found: ID does not exist" containerID="a832cb7c312bd41912751a8af9f11191824476540a7a8b2bba18f11f7595ee6c"
Sep 09 20:19:50.335788 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:50.335760    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a832cb7c312bd41912751a8af9f11191824476540a7a8b2bba18f11f7595ee6c"} err="failed to get container status \"a832cb7c312bd41912751a8af9f11191824476540a7a8b2bba18f11f7595ee6c\": rpc error: code = NotFound desc = could not find container \"a832cb7c312bd41912751a8af9f11191824476540a7a8b2bba18f11f7595ee6c\": container with ID starting with a832cb7c312bd41912751a8af9f11191824476540a7a8b2bba18f11f7595ee6c not found: ID does not exist"
Sep 09 20:19:50.335788 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:50.335778    2523 scope.go:117] "RemoveContainer" containerID="cd97815d9516b53a2a27f3d51b3a240e1d9d61f7684de647f479602fb9814e85"
Sep 09 20:19:50.335998 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:19:50.335982    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd97815d9516b53a2a27f3d51b3a240e1d9d61f7684de647f479602fb9814e85\": container with ID starting with cd97815d9516b53a2a27f3d51b3a240e1d9d61f7684de647f479602fb9814e85 not found: ID does not exist" containerID="cd97815d9516b53a2a27f3d51b3a240e1d9d61f7684de647f479602fb9814e85"
Sep 09 20:19:50.336041 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:50.336011    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd97815d9516b53a2a27f3d51b3a240e1d9d61f7684de647f479602fb9814e85"} err="failed to get container status \"cd97815d9516b53a2a27f3d51b3a240e1d9d61f7684de647f479602fb9814e85\": rpc error: code = NotFound desc = could not find container \"cd97815d9516b53a2a27f3d51b3a240e1d9d61f7684de647f479602fb9814e85\": container with ID starting with cd97815d9516b53a2a27f3d51b3a240e1d9d61f7684de647f479602fb9814e85 not found: ID does not exist"
Sep 09 20:19:50.336041 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:50.336025    2523 scope.go:117] "RemoveContainer" containerID="f1d1211def067bb8181cb734a69b65a84530ed664268419b80b7bbe4c91c2db4"
Sep 09 20:19:50.336257 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:19:50.336239    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1d1211def067bb8181cb734a69b65a84530ed664268419b80b7bbe4c91c2db4\": container with ID starting with f1d1211def067bb8181cb734a69b65a84530ed664268419b80b7bbe4c91c2db4 not found: ID does not exist" containerID="f1d1211def067bb8181cb734a69b65a84530ed664268419b80b7bbe4c91c2db4"
Sep 09 20:19:50.336305 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:50.336259    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1d1211def067bb8181cb734a69b65a84530ed664268419b80b7bbe4c91c2db4"} err="failed to get container status \"f1d1211def067bb8181cb734a69b65a84530ed664268419b80b7bbe4c91c2db4\": rpc error: code = NotFound desc = could not find container \"f1d1211def067bb8181cb734a69b65a84530ed664268419b80b7bbe4c91c2db4\": container with ID starting with f1d1211def067bb8181cb734a69b65a84530ed664268419b80b7bbe4c91c2db4 not found: ID does not exist"
Sep 09 20:19:50.338647 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:50.338627    2523 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-xlm7t"]
Sep 09 20:19:51.806979 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:51.806941    2523 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07673bbf-0ab7-45cb-ba27-ba6cc385cc10" path="/var/lib/kubelet/pods/07673bbf-0ab7-45cb-ba27-ba6cc385cc10/volumes"
Sep 09 20:19:53.713081 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:53.713048    2523 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74"]
Sep 09 20:19:53.713453 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:53.713328    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07673bbf-0ab7-45cb-ba27-ba6cc385cc10" containerName="kserve-container"
Sep 09 20:19:53.713453 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:53.713340    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="07673bbf-0ab7-45cb-ba27-ba6cc385cc10" containerName="kserve-container"
Sep 09 20:19:53.713453 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:53.713348    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f670a4b2-cef5-4d1f-8295-5ed166e49fd0" containerName="kube-rbac-proxy"
Sep 09 20:19:53.713453 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:53.713354    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="f670a4b2-cef5-4d1f-8295-5ed166e49fd0" containerName="kube-rbac-proxy"
Sep 09 20:19:53.713453 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:53.713365    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07673bbf-0ab7-45cb-ba27-ba6cc385cc10" containerName="storage-initializer"
Sep 09 20:19:53.713453 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:53.713371    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="07673bbf-0ab7-45cb-ba27-ba6cc385cc10" containerName="storage-initializer"
Sep 09 20:19:53.713453 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:53.713379    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f670a4b2-cef5-4d1f-8295-5ed166e49fd0" containerName="kserve-container"
Sep 09 20:19:53.713453 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:53.713384    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="f670a4b2-cef5-4d1f-8295-5ed166e49fd0" containerName="kserve-container"
Sep 09 20:19:53.713453 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:53.713390    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f670a4b2-cef5-4d1f-8295-5ed166e49fd0" containerName="storage-initializer"
Sep 09 20:19:53.713453 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:53.713395    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="f670a4b2-cef5-4d1f-8295-5ed166e49fd0" containerName="storage-initializer"
Sep 09 20:19:53.713453 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:53.713403    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07673bbf-0ab7-45cb-ba27-ba6cc385cc10" containerName="kube-rbac-proxy"
Sep 09 20:19:53.713453 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:53.713408    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="07673bbf-0ab7-45cb-ba27-ba6cc385cc10" containerName="kube-rbac-proxy"
Sep 09 20:19:53.713453 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:53.713455    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="f670a4b2-cef5-4d1f-8295-5ed166e49fd0" containerName="kube-rbac-proxy"
Sep 09 20:19:53.713453 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:53.713462    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="07673bbf-0ab7-45cb-ba27-ba6cc385cc10" containerName="kserve-container"
Sep 09 20:19:53.713853 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:53.713467    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="f670a4b2-cef5-4d1f-8295-5ed166e49fd0" containerName="kserve-container"
Sep 09 20:19:53.713853 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:53.713475    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="07673bbf-0ab7-45cb-ba27-ba6cc385cc10" containerName="kube-rbac-proxy"
Sep 09 20:19:53.718214 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:53.718197    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74"
Sep 09 20:19:53.721197 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:53.721179    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Sep 09 20:19:53.721654 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:53.721638    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 20:19:53.721959 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:53.721943    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Sep 09 20:19:53.722019 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:53.721943    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-ks7mb\""
Sep 09 20:19:53.722338 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:53.722318    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 20:19:53.722423 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:53.722328    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:19:53.722619 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:53.722608    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 20:19:53.728432 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:53.728414    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74"]
Sep 09 20:19:53.820104 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:53.820080    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9eefa4b4-792d-4c1e-830a-5a1dbaaef92f-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74\" (UID: \"9eefa4b4-792d-4c1e-830a-5a1dbaaef92f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74"
Sep 09 20:19:53.820271 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:53.820120    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq2l8\" (UniqueName: \"kubernetes.io/projected/9eefa4b4-792d-4c1e-830a-5a1dbaaef92f-kube-api-access-qq2l8\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74\" (UID: \"9eefa4b4-792d-4c1e-830a-5a1dbaaef92f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74"
Sep 09 20:19:53.820271 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:53.820140    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9eefa4b4-792d-4c1e-830a-5a1dbaaef92f-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74\" (UID: \"9eefa4b4-792d-4c1e-830a-5a1dbaaef92f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74"
Sep 09 20:19:53.820382 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:53.820264    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9eefa4b4-792d-4c1e-830a-5a1dbaaef92f-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74\" (UID: \"9eefa4b4-792d-4c1e-830a-5a1dbaaef92f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74"
Sep 09 20:19:53.820382 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:53.820296    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9eefa4b4-792d-4c1e-830a-5a1dbaaef92f-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74\" (UID: \"9eefa4b4-792d-4c1e-830a-5a1dbaaef92f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74"
Sep 09 20:19:53.920941 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:53.920919    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq2l8\" (UniqueName: \"kubernetes.io/projected/9eefa4b4-792d-4c1e-830a-5a1dbaaef92f-kube-api-access-qq2l8\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74\" (UID: \"9eefa4b4-792d-4c1e-830a-5a1dbaaef92f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74"
Sep 09 20:19:53.921045 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:53.920947    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9eefa4b4-792d-4c1e-830a-5a1dbaaef92f-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74\" (UID: \"9eefa4b4-792d-4c1e-830a-5a1dbaaef92f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74"
Sep 09 20:19:53.921045 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:53.921039    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9eefa4b4-792d-4c1e-830a-5a1dbaaef92f-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74\" (UID: \"9eefa4b4-792d-4c1e-830a-5a1dbaaef92f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74"
Sep 09 20:19:53.921190 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:19:53.921128    2523 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-serving-cert: secret "isvc-autogluon-v2-noslash-predictor-serving-cert" not found
Sep 09 20:19:53.921255 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:19:53.921221    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9eefa4b4-792d-4c1e-830a-5a1dbaaef92f-proxy-tls podName:9eefa4b4-792d-4c1e-830a-5a1dbaaef92f nodeName:}" failed. No retries permitted until 2026-09-09 20:19:54.421201135 +0000 UTC m=+1499.093220117 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9eefa4b4-792d-4c1e-830a-5a1dbaaef92f-proxy-tls") pod "isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74" (UID: "9eefa4b4-792d-4c1e-830a-5a1dbaaef92f") : secret "isvc-autogluon-v2-noslash-predictor-serving-cert" not found
Sep 09 20:19:53.921319 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:53.921280    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9eefa4b4-792d-4c1e-830a-5a1dbaaef92f-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74\" (UID: \"9eefa4b4-792d-4c1e-830a-5a1dbaaef92f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74"
Sep 09 20:19:53.921376 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:53.921342    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9eefa4b4-792d-4c1e-830a-5a1dbaaef92f-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74\" (UID: \"9eefa4b4-792d-4c1e-830a-5a1dbaaef92f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74"
Sep 09 20:19:53.921376 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:53.921368    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9eefa4b4-792d-4c1e-830a-5a1dbaaef92f-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74\" (UID: \"9eefa4b4-792d-4c1e-830a-5a1dbaaef92f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74"
Sep 09 20:19:53.921914 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:53.921892    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9eefa4b4-792d-4c1e-830a-5a1dbaaef92f-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74\" (UID: \"9eefa4b4-792d-4c1e-830a-5a1dbaaef92f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74"
Sep 09 20:19:53.921984 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:53.921899    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9eefa4b4-792d-4c1e-830a-5a1dbaaef92f-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74\" (UID: \"9eefa4b4-792d-4c1e-830a-5a1dbaaef92f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74"
Sep 09 20:19:53.930884 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:53.930863    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq2l8\" (UniqueName: \"kubernetes.io/projected/9eefa4b4-792d-4c1e-830a-5a1dbaaef92f-kube-api-access-qq2l8\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74\" (UID: \"9eefa4b4-792d-4c1e-830a-5a1dbaaef92f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74"
Sep 09 20:19:54.423626 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:54.423603    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9eefa4b4-792d-4c1e-830a-5a1dbaaef92f-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74\" (UID: \"9eefa4b4-792d-4c1e-830a-5a1dbaaef92f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74"
Sep 09 20:19:54.425926 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:54.425900    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9eefa4b4-792d-4c1e-830a-5a1dbaaef92f-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74\" (UID: \"9eefa4b4-792d-4c1e-830a-5a1dbaaef92f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74"
Sep 09 20:19:54.628762 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:54.628730    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74"
Sep 09 20:19:54.904069 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:54.904039    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74"]
Sep 09 20:19:54.907246 ip-10-0-132-226 kubenswrapper[2523]: W0909 20:19:54.907217    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eefa4b4_792d_4c1e_830a_5a1dbaaef92f.slice/crio-e803aca31c381442b5089c76f52c71e7667f2853da7e9f1366229c21cb1a21a2 WatchSource:0}: Error finding container e803aca31c381442b5089c76f52c71e7667f2853da7e9f1366229c21cb1a21a2: Status 404 returned error can't find the container with id e803aca31c381442b5089c76f52c71e7667f2853da7e9f1366229c21cb1a21a2
Sep 09 20:19:55.333946 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:55.333910    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74" event={"ID":"9eefa4b4-792d-4c1e-830a-5a1dbaaef92f","Type":"ContainerStarted","Data":"8fdd96afec2895fe83bc12e116af4070a4d2b619d18649440a75983690d7e326"}
Sep 09 20:19:55.334131 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:19:55.333983    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74" event={"ID":"9eefa4b4-792d-4c1e-830a-5a1dbaaef92f","Type":"ContainerStarted","Data":"e803aca31c381442b5089c76f52c71e7667f2853da7e9f1366229c21cb1a21a2"}
Sep 09 20:20:24.433331 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:20:24.433293    2523 generic.go:356] "Generic (PLEG): container finished" podID="9eefa4b4-792d-4c1e-830a-5a1dbaaef92f" containerID="8fdd96afec2895fe83bc12e116af4070a4d2b619d18649440a75983690d7e326" exitCode=0
Sep 09 20:20:24.433664 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:20:24.433363    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74" event={"ID":"9eefa4b4-792d-4c1e-830a-5a1dbaaef92f","Type":"ContainerDied","Data":"8fdd96afec2895fe83bc12e116af4070a4d2b619d18649440a75983690d7e326"}
Sep 09 20:20:25.440068 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:20:25.440032    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74" event={"ID":"9eefa4b4-792d-4c1e-830a-5a1dbaaef92f","Type":"ContainerStarted","Data":"7fb00908956409d72522377652c8ab02a762bb38ea733768b63f4445cfb373a1"}
Sep 09 20:20:25.440517 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:20:25.440095    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74" event={"ID":"9eefa4b4-792d-4c1e-830a-5a1dbaaef92f","Type":"ContainerStarted","Data":"327a509ef54d871318f4fcdfa2e8e763cc9d9bb13a27c49110241e0171e0ed58"}
Sep 09 20:20:25.440517 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:20:25.440296    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74"
Sep 09 20:20:25.461480 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:20:25.461437    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74" podStartSLOduration=3.167135693 podStartE2EDuration="32.461423776s" totalImagesPullingTime="0s" totalInitContainerRuntime="29.294288083s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:19:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:20:25.460326864 +0000 UTC m=+1530.132345896" watchObservedRunningTime="2026-09-09 20:20:25.461423776 +0000 UTC m=+1530.133442773"
Sep 09 20:20:26.444716 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:20:26.444685    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74"
Sep 09 20:20:32.455084 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:20:32.455051    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74"
Sep 09 20:21:59.807239 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:21:59.807139    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74"
Sep 09 20:22:09.871762 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:09.871722    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74"]
Sep 09 20:22:09.872263 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:09.871937    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74" podUID="9eefa4b4-792d-4c1e-830a-5a1dbaaef92f" containerName="kserve-container" containerID="cri-o://327a509ef54d871318f4fcdfa2e8e763cc9d9bb13a27c49110241e0171e0ed58" gracePeriod=30
Sep 09 20:22:09.872263 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:09.871983    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74" podUID="9eefa4b4-792d-4c1e-830a-5a1dbaaef92f" containerName="kube-rbac-proxy" containerID="cri-o://7fb00908956409d72522377652c8ab02a762bb38ea733768b63f4445cfb373a1" gracePeriod=30
Sep 09 20:22:10.829454 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:10.829410    2523 generic.go:356] "Generic (PLEG): container finished" podID="9eefa4b4-792d-4c1e-830a-5a1dbaaef92f" containerID="7fb00908956409d72522377652c8ab02a762bb38ea733768b63f4445cfb373a1" exitCode=2
Sep 09 20:22:10.829623 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:10.829457    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74" event={"ID":"9eefa4b4-792d-4c1e-830a-5a1dbaaef92f","Type":"ContainerDied","Data":"7fb00908956409d72522377652c8ab02a762bb38ea733768b63f4445cfb373a1"}
Sep 09 20:22:11.117328 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:11.117305    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74"
Sep 09 20:22:11.231738 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:11.231704    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9eefa4b4-792d-4c1e-830a-5a1dbaaef92f-proxy-tls\") pod \"9eefa4b4-792d-4c1e-830a-5a1dbaaef92f\" (UID: \"9eefa4b4-792d-4c1e-830a-5a1dbaaef92f\") "
Sep 09 20:22:11.231886 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:11.231770    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qq2l8\" (UniqueName: \"kubernetes.io/projected/9eefa4b4-792d-4c1e-830a-5a1dbaaef92f-kube-api-access-qq2l8\") pod \"9eefa4b4-792d-4c1e-830a-5a1dbaaef92f\" (UID: \"9eefa4b4-792d-4c1e-830a-5a1dbaaef92f\") "
Sep 09 20:22:11.231886 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:11.231818    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9eefa4b4-792d-4c1e-830a-5a1dbaaef92f-kube-rbac-proxy-sar-config\") pod \"9eefa4b4-792d-4c1e-830a-5a1dbaaef92f\" (UID: \"9eefa4b4-792d-4c1e-830a-5a1dbaaef92f\") "
Sep 09 20:22:11.231886 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:11.231845    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9eefa4b4-792d-4c1e-830a-5a1dbaaef92f-kserve-provision-location\") pod \"9eefa4b4-792d-4c1e-830a-5a1dbaaef92f\" (UID: \"9eefa4b4-792d-4c1e-830a-5a1dbaaef92f\") "
Sep 09 20:22:11.231886 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:11.231866    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9eefa4b4-792d-4c1e-830a-5a1dbaaef92f-cabundle-cert\") pod \"9eefa4b4-792d-4c1e-830a-5a1dbaaef92f\" (UID: \"9eefa4b4-792d-4c1e-830a-5a1dbaaef92f\") "
Sep 09 20:22:11.232185 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:11.232136    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9eefa4b4-792d-4c1e-830a-5a1dbaaef92f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9eefa4b4-792d-4c1e-830a-5a1dbaaef92f" (UID: "9eefa4b4-792d-4c1e-830a-5a1dbaaef92f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:22:11.232324 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:11.232258    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9eefa4b4-792d-4c1e-830a-5a1dbaaef92f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9eefa4b4-792d-4c1e-830a-5a1dbaaef92f" (UID: "9eefa4b4-792d-4c1e-830a-5a1dbaaef92f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:22:11.233957 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:11.233929    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9eefa4b4-792d-4c1e-830a-5a1dbaaef92f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9eefa4b4-792d-4c1e-830a-5a1dbaaef92f" (UID: "9eefa4b4-792d-4c1e-830a-5a1dbaaef92f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:22:11.234040 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:11.233956    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eefa4b4-792d-4c1e-830a-5a1dbaaef92f-kube-api-access-qq2l8" (OuterVolumeSpecName: "kube-api-access-qq2l8") pod "9eefa4b4-792d-4c1e-830a-5a1dbaaef92f" (UID: "9eefa4b4-792d-4c1e-830a-5a1dbaaef92f"). InnerVolumeSpecName "kube-api-access-qq2l8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:22:11.235899 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:11.235876    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eefa4b4-792d-4c1e-830a-5a1dbaaef92f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9eefa4b4-792d-4c1e-830a-5a1dbaaef92f" (UID: "9eefa4b4-792d-4c1e-830a-5a1dbaaef92f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:22:11.333044 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:11.333021    2523 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9eefa4b4-792d-4c1e-830a-5a1dbaaef92f-proxy-tls\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:22:11.333044 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:11.333041    2523 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq2l8\" (UniqueName: \"kubernetes.io/projected/9eefa4b4-792d-4c1e-830a-5a1dbaaef92f-kube-api-access-qq2l8\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:22:11.333193 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:11.333051    2523 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9eefa4b4-792d-4c1e-830a-5a1dbaaef92f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:22:11.333193 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:11.333060    2523 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9eefa4b4-792d-4c1e-830a-5a1dbaaef92f-kserve-provision-location\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:22:11.333193 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:11.333069    2523 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9eefa4b4-792d-4c1e-830a-5a1dbaaef92f-cabundle-cert\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:22:11.835012 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:11.834984    2523 generic.go:356] "Generic (PLEG): container finished" podID="9eefa4b4-792d-4c1e-830a-5a1dbaaef92f" containerID="327a509ef54d871318f4fcdfa2e8e763cc9d9bb13a27c49110241e0171e0ed58" exitCode=0
Sep 09 20:22:11.835118 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:11.835039    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74"
Sep 09 20:22:11.835118 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:11.835043    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74" event={"ID":"9eefa4b4-792d-4c1e-830a-5a1dbaaef92f","Type":"ContainerDied","Data":"327a509ef54d871318f4fcdfa2e8e763cc9d9bb13a27c49110241e0171e0ed58"}
Sep 09 20:22:11.835118 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:11.835092    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74" event={"ID":"9eefa4b4-792d-4c1e-830a-5a1dbaaef92f","Type":"ContainerDied","Data":"e803aca31c381442b5089c76f52c71e7667f2853da7e9f1366229c21cb1a21a2"}
Sep 09 20:22:11.835232 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:11.835120    2523 scope.go:117] "RemoveContainer" containerID="7fb00908956409d72522377652c8ab02a762bb38ea733768b63f4445cfb373a1"
Sep 09 20:22:11.846653 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:11.846633    2523 scope.go:117] "RemoveContainer" containerID="327a509ef54d871318f4fcdfa2e8e763cc9d9bb13a27c49110241e0171e0ed58"
Sep 09 20:22:11.853854 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:11.853838    2523 scope.go:117] "RemoveContainer" containerID="8fdd96afec2895fe83bc12e116af4070a4d2b619d18649440a75983690d7e326"
Sep 09 20:22:11.857787 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:11.857759    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74"]
Sep 09 20:22:11.861320 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:11.861302    2523 scope.go:117] "RemoveContainer" containerID="7fb00908956409d72522377652c8ab02a762bb38ea733768b63f4445cfb373a1"
Sep 09 20:22:11.861605 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:22:11.861586    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fb00908956409d72522377652c8ab02a762bb38ea733768b63f4445cfb373a1\": container with ID starting with 7fb00908956409d72522377652c8ab02a762bb38ea733768b63f4445cfb373a1 not found: ID does not exist" containerID="7fb00908956409d72522377652c8ab02a762bb38ea733768b63f4445cfb373a1"
Sep 09 20:22:11.861650 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:11.861612    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fb00908956409d72522377652c8ab02a762bb38ea733768b63f4445cfb373a1"} err="failed to get container status \"7fb00908956409d72522377652c8ab02a762bb38ea733768b63f4445cfb373a1\": rpc error: code = NotFound desc = could not find container \"7fb00908956409d72522377652c8ab02a762bb38ea733768b63f4445cfb373a1\": container with ID starting with 7fb00908956409d72522377652c8ab02a762bb38ea733768b63f4445cfb373a1 not found: ID does not exist"
Sep 09 20:22:11.861650 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:11.861626    2523 scope.go:117] "RemoveContainer" containerID="327a509ef54d871318f4fcdfa2e8e763cc9d9bb13a27c49110241e0171e0ed58"
Sep 09 20:22:11.861815 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:11.861799    2523 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-mhm74"]
Sep 09 20:22:11.861873 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:22:11.861858    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"327a509ef54d871318f4fcdfa2e8e763cc9d9bb13a27c49110241e0171e0ed58\": container with ID starting with 327a509ef54d871318f4fcdfa2e8e763cc9d9bb13a27c49110241e0171e0ed58 not found: ID does not exist" containerID="327a509ef54d871318f4fcdfa2e8e763cc9d9bb13a27c49110241e0171e0ed58"
Sep 09 20:22:11.861917 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:11.861879    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"327a509ef54d871318f4fcdfa2e8e763cc9d9bb13a27c49110241e0171e0ed58"} err="failed to get container status \"327a509ef54d871318f4fcdfa2e8e763cc9d9bb13a27c49110241e0171e0ed58\": rpc error: code = NotFound desc = could not find container \"327a509ef54d871318f4fcdfa2e8e763cc9d9bb13a27c49110241e0171e0ed58\": container with ID starting with 327a509ef54d871318f4fcdfa2e8e763cc9d9bb13a27c49110241e0171e0ed58 not found: ID does not exist"
Sep 09 20:22:11.861917 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:11.861895    2523 scope.go:117] "RemoveContainer" containerID="8fdd96afec2895fe83bc12e116af4070a4d2b619d18649440a75983690d7e326"
Sep 09 20:22:11.862134 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:22:11.862119    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fdd96afec2895fe83bc12e116af4070a4d2b619d18649440a75983690d7e326\": container with ID starting with 8fdd96afec2895fe83bc12e116af4070a4d2b619d18649440a75983690d7e326 not found: ID does not exist" containerID="8fdd96afec2895fe83bc12e116af4070a4d2b619d18649440a75983690d7e326"
Sep 09 20:22:11.862209 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:11.862137    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fdd96afec2895fe83bc12e116af4070a4d2b619d18649440a75983690d7e326"} err="failed to get container status \"8fdd96afec2895fe83bc12e116af4070a4d2b619d18649440a75983690d7e326\": rpc error: code = NotFound desc = could not find container \"8fdd96afec2895fe83bc12e116af4070a4d2b619d18649440a75983690d7e326\": container with ID starting with 8fdd96afec2895fe83bc12e116af4070a4d2b619d18649440a75983690d7e326 not found: ID does not exist"
Sep 09 20:22:13.807504 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:13.807472    2523 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9eefa4b4-792d-4c1e-830a-5a1dbaaef92f" path="/var/lib/kubelet/pods/9eefa4b4-792d-4c1e-830a-5a1dbaaef92f/volumes"
Sep 09 20:22:15.056570 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:15.056542    2523 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh"]
Sep 09 20:22:15.056951 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:15.056794    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9eefa4b4-792d-4c1e-830a-5a1dbaaef92f" containerName="kube-rbac-proxy"
Sep 09 20:22:15.056951 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:15.056805    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="9eefa4b4-792d-4c1e-830a-5a1dbaaef92f" containerName="kube-rbac-proxy"
Sep 09 20:22:15.056951 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:15.056820    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9eefa4b4-792d-4c1e-830a-5a1dbaaef92f" containerName="storage-initializer"
Sep 09 20:22:15.056951 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:15.056825    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="9eefa4b4-792d-4c1e-830a-5a1dbaaef92f" containerName="storage-initializer"
Sep 09 20:22:15.056951 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:15.056835    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9eefa4b4-792d-4c1e-830a-5a1dbaaef92f" containerName="kserve-container"
Sep 09 20:22:15.056951 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:15.056840    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="9eefa4b4-792d-4c1e-830a-5a1dbaaef92f" containerName="kserve-container"
Sep 09 20:22:15.056951 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:15.056883    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="9eefa4b4-792d-4c1e-830a-5a1dbaaef92f" containerName="kserve-container"
Sep 09 20:22:15.056951 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:15.056890    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="9eefa4b4-792d-4c1e-830a-5a1dbaaef92f" containerName="kube-rbac-proxy"
Sep 09 20:22:15.061421 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:15.061404    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh"
Sep 09 20:22:15.064083 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:15.064055    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:22:15.064232 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:15.064055    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Sep 09 20:22:15.064232 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:15.064110    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-ks7mb\""
Sep 09 20:22:15.064232 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:15.064062    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 20:22:15.064232 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:15.064095    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Sep 09 20:22:15.064489 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:15.064450    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 20:22:15.065068 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:15.065049    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 20:22:15.071803 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:15.071782    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh"]
Sep 09 20:22:15.159756 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:15.159731    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh\" (UID: \"fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh"
Sep 09 20:22:15.159839 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:15.159770    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh\" (UID: \"fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh"
Sep 09 20:22:15.159839 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:15.159798    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh\" (UID: \"fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh"
Sep 09 20:22:15.159839 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:15.159822    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh\" (UID: \"fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh"
Sep 09 20:22:15.159953 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:15.159851    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7mtm\" (UniqueName: \"kubernetes.io/projected/fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c-kube-api-access-m7mtm\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh\" (UID: \"fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh"
Sep 09 20:22:15.260437 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:15.260414    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh\" (UID: \"fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh"
Sep 09 20:22:15.260540 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:15.260443    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh\" (UID: \"fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh"
Sep 09 20:22:15.260540 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:15.260462    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh\" (UID: \"fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh"
Sep 09 20:22:15.260540 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:15.260490    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7mtm\" (UniqueName: \"kubernetes.io/projected/fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c-kube-api-access-m7mtm\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh\" (UID: \"fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh"
Sep 09 20:22:15.260688 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:15.260602    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh\" (UID: \"fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh"
Sep 09 20:22:15.260861 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:15.260841    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh\" (UID: \"fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh"
Sep 09 20:22:15.261085 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:15.261069    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh\" (UID: \"fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh"
Sep 09 20:22:15.261190 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:15.261171    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh\" (UID: \"fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh"
Sep 09 20:22:15.262966 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:15.262945    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh\" (UID: \"fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh"
Sep 09 20:22:15.271357 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:15.271337    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7mtm\" (UniqueName: \"kubernetes.io/projected/fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c-kube-api-access-m7mtm\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh\" (UID: \"fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh"
Sep 09 20:22:15.370531 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:15.370510    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh"
Sep 09 20:22:15.640767 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:15.640743    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh"]
Sep 09 20:22:15.643229 ip-10-0-132-226 kubenswrapper[2523]: W0909 20:22:15.643205    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa859455_d6d5_4d80_9e23_dc6a8a1ffc8c.slice/crio-3769584d38e56d205b6280d3fdb8522cf689d1c5d224a88e3cfd4e93def90e79 WatchSource:0}: Error finding container 3769584d38e56d205b6280d3fdb8522cf689d1c5d224a88e3cfd4e93def90e79: Status 404 returned error can't find the container with id 3769584d38e56d205b6280d3fdb8522cf689d1c5d224a88e3cfd4e93def90e79
Sep 09 20:22:15.850887 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:15.850844    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh" event={"ID":"fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c","Type":"ContainerStarted","Data":"99dde4f6d6db3c2d2fc647ea6ec76f35142b6c36ebbda05812cca2da98ee42bf"}
Sep 09 20:22:15.851048 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:15.850919    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh" event={"ID":"fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c","Type":"ContainerStarted","Data":"3769584d38e56d205b6280d3fdb8522cf689d1c5d224a88e3cfd4e93def90e79"}
Sep 09 20:22:24.883515 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:24.883464    2523 generic.go:356] "Generic (PLEG): container finished" podID="fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c" containerID="99dde4f6d6db3c2d2fc647ea6ec76f35142b6c36ebbda05812cca2da98ee42bf" exitCode=0
Sep 09 20:22:24.883799 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:24.883537    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh" event={"ID":"fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c","Type":"ContainerDied","Data":"99dde4f6d6db3c2d2fc647ea6ec76f35142b6c36ebbda05812cca2da98ee42bf"}
Sep 09 20:22:25.889212 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:25.889175    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh" event={"ID":"fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c","Type":"ContainerStarted","Data":"c97132a6f4afa1b609248836c42991257840506fe6b02db5f517a85ee2c1e041"}
Sep 09 20:22:25.889677 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:25.889249    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh" event={"ID":"fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c","Type":"ContainerStarted","Data":"7a7d957227a3eff03d59e03f9abb674d3a688e891ff5643d90fb0225aec5dc85"}
Sep 09 20:22:25.889677 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:25.889458    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh"
Sep 09 20:22:25.915764 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:25.915720    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh" podStartSLOduration=1.812956969 podStartE2EDuration="10.915706013s" totalImagesPullingTime="0s" totalInitContainerRuntime="9.102749044s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:22:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:22:25.914211379 +0000 UTC m=+1650.586230376" watchObservedRunningTime="2026-09-09 20:22:25.915706013 +0000 UTC m=+1650.587725011"
Sep 09 20:22:26.893590 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:26.893559    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh"
Sep 09 20:22:26.894866 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:26.894838    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh" podUID="fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 09 20:22:27.902171 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:27.902106    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh" podUID="fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 09 20:22:32.905549 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:32.905521    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh"
Sep 09 20:22:32.906121 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:32.905974    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh" podUID="fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 09 20:22:34.870324 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:34.870290    2523 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk"]
Sep 09 20:22:34.874219 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:34.874200    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk"
Sep 09 20:22:34.878082 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:34.878051    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
Sep 09 20:22:34.878451 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:34.878434    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
Sep 09 20:22:34.889120 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:34.889092    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36a2ffd2-7556-4fea-a4d6-3c8555418097-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk\" (UID: \"36a2ffd2-7556-4fea-a4d6-3c8555418097\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk"
Sep 09 20:22:34.889225 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:34.889132    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/36a2ffd2-7556-4fea-a4d6-3c8555418097-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk\" (UID: \"36a2ffd2-7556-4fea-a4d6-3c8555418097\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk"
Sep 09 20:22:34.889278 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:34.889255    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36a2ffd2-7556-4fea-a4d6-3c8555418097-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk\" (UID: \"36a2ffd2-7556-4fea-a4d6-3c8555418097\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk"
Sep 09 20:22:34.889324 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:34.889303    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nkpn\" (UniqueName: \"kubernetes.io/projected/36a2ffd2-7556-4fea-a4d6-3c8555418097-kube-api-access-8nkpn\") pod \"isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk\" (UID: \"36a2ffd2-7556-4fea-a4d6-3c8555418097\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk"
Sep 09 20:22:34.889366 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:34.889343    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/36a2ffd2-7556-4fea-a4d6-3c8555418097-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk\" (UID: \"36a2ffd2-7556-4fea-a4d6-3c8555418097\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk"
Sep 09 20:22:34.892296 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:34.892275    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk"]
Sep 09 20:22:34.989946 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:34.989922    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36a2ffd2-7556-4fea-a4d6-3c8555418097-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk\" (UID: \"36a2ffd2-7556-4fea-a4d6-3c8555418097\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk"
Sep 09 20:22:34.990094 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:34.989954    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nkpn\" (UniqueName: \"kubernetes.io/projected/36a2ffd2-7556-4fea-a4d6-3c8555418097-kube-api-access-8nkpn\") pod \"isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk\" (UID: \"36a2ffd2-7556-4fea-a4d6-3c8555418097\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk"
Sep 09 20:22:34.990094 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:34.989977    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/36a2ffd2-7556-4fea-a4d6-3c8555418097-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk\" (UID: \"36a2ffd2-7556-4fea-a4d6-3c8555418097\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk"
Sep 09 20:22:34.990094 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:34.990023    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36a2ffd2-7556-4fea-a4d6-3c8555418097-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk\" (UID: \"36a2ffd2-7556-4fea-a4d6-3c8555418097\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk"
Sep 09 20:22:34.990094 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:34.990056    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/36a2ffd2-7556-4fea-a4d6-3c8555418097-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk\" (UID: \"36a2ffd2-7556-4fea-a4d6-3c8555418097\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk"
Sep 09 20:22:34.990094 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:22:34.990090    2523 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-serving-cert: secret "isvc-sklearn-runtime-predictor-serving-cert" not found
Sep 09 20:22:34.990407 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:22:34.990185    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36a2ffd2-7556-4fea-a4d6-3c8555418097-proxy-tls podName:36a2ffd2-7556-4fea-a4d6-3c8555418097 nodeName:}" failed. No retries permitted until 2026-09-09 20:22:35.49013658 +0000 UTC m=+1660.162155573 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/36a2ffd2-7556-4fea-a4d6-3c8555418097-proxy-tls") pod "isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk" (UID: "36a2ffd2-7556-4fea-a4d6-3c8555418097") : secret "isvc-sklearn-runtime-predictor-serving-cert" not found
Sep 09 20:22:34.990509 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:34.990488    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36a2ffd2-7556-4fea-a4d6-3c8555418097-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk\" (UID: \"36a2ffd2-7556-4fea-a4d6-3c8555418097\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk"
Sep 09 20:22:34.990665 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:34.990648    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/36a2ffd2-7556-4fea-a4d6-3c8555418097-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk\" (UID: \"36a2ffd2-7556-4fea-a4d6-3c8555418097\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk"
Sep 09 20:22:34.990702 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:34.990680    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/36a2ffd2-7556-4fea-a4d6-3c8555418097-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk\" (UID: \"36a2ffd2-7556-4fea-a4d6-3c8555418097\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk"
Sep 09 20:22:35.000597 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:35.000572    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nkpn\" (UniqueName: \"kubernetes.io/projected/36a2ffd2-7556-4fea-a4d6-3c8555418097-kube-api-access-8nkpn\") pod \"isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk\" (UID: \"36a2ffd2-7556-4fea-a4d6-3c8555418097\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk"
Sep 09 20:22:35.493660 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:35.493628    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36a2ffd2-7556-4fea-a4d6-3c8555418097-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk\" (UID: \"36a2ffd2-7556-4fea-a4d6-3c8555418097\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk"
Sep 09 20:22:35.496056 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:35.496038    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36a2ffd2-7556-4fea-a4d6-3c8555418097-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk\" (UID: \"36a2ffd2-7556-4fea-a4d6-3c8555418097\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk"
Sep 09 20:22:35.783313 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:35.783249    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk"
Sep 09 20:22:36.056089 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:36.056066    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk"]
Sep 09 20:22:36.058547 ip-10-0-132-226 kubenswrapper[2523]: W0909 20:22:36.058518    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36a2ffd2_7556_4fea_a4d6_3c8555418097.slice/crio-cf7bab3141a05564baf19a820f9b5c1b21e7595a62aefeff8a9db4f15efcde2f WatchSource:0}: Error finding container cf7bab3141a05564baf19a820f9b5c1b21e7595a62aefeff8a9db4f15efcde2f: Status 404 returned error can't find the container with id cf7bab3141a05564baf19a820f9b5c1b21e7595a62aefeff8a9db4f15efcde2f
Sep 09 20:22:36.932624 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:36.932588    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk" event={"ID":"36a2ffd2-7556-4fea-a4d6-3c8555418097","Type":"ContainerStarted","Data":"85cf2fc3a48f43060a92539d75bd7a02e1586a29f86cc772bc75d227317a066e"}
Sep 09 20:22:36.932788 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:36.932645    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk" event={"ID":"36a2ffd2-7556-4fea-a4d6-3c8555418097","Type":"ContainerStarted","Data":"cf7bab3141a05564baf19a820f9b5c1b21e7595a62aefeff8a9db4f15efcde2f"}
Sep 09 20:22:41.952677 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:41.952607    2523 generic.go:356] "Generic (PLEG): container finished" podID="36a2ffd2-7556-4fea-a4d6-3c8555418097" containerID="85cf2fc3a48f43060a92539d75bd7a02e1586a29f86cc772bc75d227317a066e" exitCode=0
Sep 09 20:22:41.952990 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:41.952679    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk" event={"ID":"36a2ffd2-7556-4fea-a4d6-3c8555418097","Type":"ContainerDied","Data":"85cf2fc3a48f43060a92539d75bd7a02e1586a29f86cc772bc75d227317a066e"}
Sep 09 20:22:42.907294 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:42.907266    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh"
Sep 09 20:22:42.959748 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:42.959717    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk" event={"ID":"36a2ffd2-7556-4fea-a4d6-3c8555418097","Type":"ContainerStarted","Data":"879c47088e52f399e05f35438b3854c6e8744aebbd2b193b3f7022e833a64beb"}
Sep 09 20:22:42.960269 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:42.959778    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk" event={"ID":"36a2ffd2-7556-4fea-a4d6-3c8555418097","Type":"ContainerStarted","Data":"ffcd89d543e406758788b28350fdd3f89f938c83d0d64bbdeb07e2da7694b13b"}
Sep 09 20:22:42.960269 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:42.959962    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk"
Sep 09 20:22:42.981916 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:42.981864    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk" podStartSLOduration=3.413884091 podStartE2EDuration="8.981849419s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.567965328s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:22:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:22:42.980255826 +0000 UTC m=+1667.652274821" watchObservedRunningTime="2026-09-09 20:22:42.981849419 +0000 UTC m=+1667.653868413"
Sep 09 20:22:43.963202 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:43.963173    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk"
Sep 09 20:22:43.964350 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:43.964319    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk" podUID="36a2ffd2-7556-4fea-a4d6-3c8555418097" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 09 20:22:44.967353 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:44.967299    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk" podUID="36a2ffd2-7556-4fea-a4d6-3c8555418097" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 09 20:22:49.972074 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:49.972042    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk"
Sep 09 20:22:49.972672 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:49.972645    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk" podUID="36a2ffd2-7556-4fea-a4d6-3c8555418097" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 09 20:22:57.446692 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:57.446659    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh"]
Sep 09 20:22:57.447090 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:57.446892    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh" podUID="fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c" containerName="kserve-container" containerID="cri-o://7a7d957227a3eff03d59e03f9abb674d3a688e891ff5643d90fb0225aec5dc85" gracePeriod=30
Sep 09 20:22:57.447090 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:57.446938    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh" podUID="fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c" containerName="kube-rbac-proxy" containerID="cri-o://c97132a6f4afa1b609248836c42991257840506fe6b02db5f517a85ee2c1e041" gracePeriod=30
Sep 09 20:22:57.901486 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:57.901440    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh" podUID="fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Sep 09 20:22:58.025041 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:58.025000    2523 generic.go:356] "Generic (PLEG): container finished" podID="fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c" containerID="c97132a6f4afa1b609248836c42991257840506fe6b02db5f517a85ee2c1e041" exitCode=2
Sep 09 20:22:58.025239 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:58.025075    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh" event={"ID":"fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c","Type":"ContainerDied","Data":"c97132a6f4afa1b609248836c42991257840506fe6b02db5f517a85ee2c1e041"}
Sep 09 20:22:59.031961 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:59.031915    2523 generic.go:356] "Generic (PLEG): container finished" podID="fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c" containerID="7a7d957227a3eff03d59e03f9abb674d3a688e891ff5643d90fb0225aec5dc85" exitCode=0
Sep 09 20:22:59.032314 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:59.031999    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh" event={"ID":"fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c","Type":"ContainerDied","Data":"7a7d957227a3eff03d59e03f9abb674d3a688e891ff5643d90fb0225aec5dc85"}
Sep 09 20:22:59.090475 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:59.090452    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh"
Sep 09 20:22:59.179800 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:59.179730    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c-proxy-tls\") pod \"fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c\" (UID: \"fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c\") "
Sep 09 20:22:59.179800 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:59.179786    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c-kserve-provision-location\") pod \"fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c\" (UID: \"fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c\") "
Sep 09 20:22:59.180005 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:59.179817    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c-kube-rbac-proxy-sar-config\") pod \"fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c\" (UID: \"fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c\") "
Sep 09 20:22:59.180005 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:59.179856    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7mtm\" (UniqueName: \"kubernetes.io/projected/fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c-kube-api-access-m7mtm\") pod \"fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c\" (UID: \"fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c\") "
Sep 09 20:22:59.180005 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:59.179937    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c-cabundle-cert\") pod \"fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c\" (UID: \"fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c\") "
Sep 09 20:22:59.180189 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:59.180144    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c" (UID: "fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:22:59.180573 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:59.180479    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c" (UID: "fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:22:59.180747 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:59.180667    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c" (UID: "fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:22:59.181881 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:59.181853    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c" (UID: "fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:22:59.182014 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:59.181998    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c-kube-api-access-m7mtm" (OuterVolumeSpecName: "kube-api-access-m7mtm") pod "fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c" (UID: "fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c"). InnerVolumeSpecName "kube-api-access-m7mtm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:22:59.280415 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:59.280390    2523 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c-cabundle-cert\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:22:59.280415 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:59.280412    2523 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c-proxy-tls\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:22:59.280556 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:59.280422    2523 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c-kserve-provision-location\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:22:59.280556 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:59.280431    2523 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:22:59.280556 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:59.280440    2523 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7mtm\" (UniqueName: \"kubernetes.io/projected/fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c-kube-api-access-m7mtm\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:22:59.972959 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:22:59.972934    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk"
Sep 09 20:23:00.038393 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:23:00.038351    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh" event={"ID":"fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c","Type":"ContainerDied","Data":"3769584d38e56d205b6280d3fdb8522cf689d1c5d224a88e3cfd4e93def90e79"}
Sep 09 20:23:00.038755 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:23:00.038450    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh"
Sep 09 20:23:00.038755 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:23:00.038453    2523 scope.go:117] "RemoveContainer" containerID="c97132a6f4afa1b609248836c42991257840506fe6b02db5f517a85ee2c1e041"
Sep 09 20:23:00.048081 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:23:00.048063    2523 scope.go:117] "RemoveContainer" containerID="7a7d957227a3eff03d59e03f9abb674d3a688e891ff5643d90fb0225aec5dc85"
Sep 09 20:23:00.055684 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:23:00.055660    2523 scope.go:117] "RemoveContainer" containerID="99dde4f6d6db3c2d2fc647ea6ec76f35142b6c36ebbda05812cca2da98ee42bf"
Sep 09 20:23:00.056608 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:23:00.056580    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh"]
Sep 09 20:23:00.058662 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:23:00.058639    2523 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-ln9bh"]
Sep 09 20:23:01.807379 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:23:01.807346    2523 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c" path="/var/lib/kubelet/pods/fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c/volumes"
Sep 09 20:23:06.942930 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:23:06.942901    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk"]
Sep 09 20:23:06.943327 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:23:06.943119    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk" podUID="36a2ffd2-7556-4fea-a4d6-3c8555418097" containerName="kserve-container" containerID="cri-o://ffcd89d543e406758788b28350fdd3f89f938c83d0d64bbdeb07e2da7694b13b" gracePeriod=30
Sep 09 20:23:06.943327 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:23:06.943187    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk" podUID="36a2ffd2-7556-4fea-a4d6-3c8555418097" containerName="kube-rbac-proxy" containerID="cri-o://879c47088e52f399e05f35438b3854c6e8744aebbd2b193b3f7022e833a64beb" gracePeriod=30
Sep 09 20:23:07.068991 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:23:07.068965    2523 generic.go:356] "Generic (PLEG): container finished" podID="36a2ffd2-7556-4fea-a4d6-3c8555418097" containerID="879c47088e52f399e05f35438b3854c6e8744aebbd2b193b3f7022e833a64beb" exitCode=2
Sep 09 20:23:07.069083 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:23:07.069032    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk" event={"ID":"36a2ffd2-7556-4fea-a4d6-3c8555418097","Type":"ContainerDied","Data":"879c47088e52f399e05f35438b3854c6e8744aebbd2b193b3f7022e833a64beb"}
Sep 09 20:23:08.075464 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:23:08.075437    2523 generic.go:356] "Generic (PLEG): container finished" podID="36a2ffd2-7556-4fea-a4d6-3c8555418097" containerID="ffcd89d543e406758788b28350fdd3f89f938c83d0d64bbdeb07e2da7694b13b" exitCode=0
Sep 09 20:23:08.075850 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:23:08.075507    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk" event={"ID":"36a2ffd2-7556-4fea-a4d6-3c8555418097","Type":"ContainerDied","Data":"ffcd89d543e406758788b28350fdd3f89f938c83d0d64bbdeb07e2da7694b13b"}
Sep 09 20:23:08.075850 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:23:08.075559    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk" event={"ID":"36a2ffd2-7556-4fea-a4d6-3c8555418097","Type":"ContainerDied","Data":"cf7bab3141a05564baf19a820f9b5c1b21e7595a62aefeff8a9db4f15efcde2f"}
Sep 09 20:23:08.075850 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:23:08.075581    2523 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf7bab3141a05564baf19a820f9b5c1b21e7595a62aefeff8a9db4f15efcde2f"
Sep 09 20:23:08.085008 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:23:08.084988    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk"
Sep 09 20:23:08.147791 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:23:08.147730    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36a2ffd2-7556-4fea-a4d6-3c8555418097-proxy-tls\") pod \"36a2ffd2-7556-4fea-a4d6-3c8555418097\" (UID: \"36a2ffd2-7556-4fea-a4d6-3c8555418097\") "
Sep 09 20:23:08.147791 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:23:08.147769    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/36a2ffd2-7556-4fea-a4d6-3c8555418097-kube-rbac-proxy-sar-config\") pod \"36a2ffd2-7556-4fea-a4d6-3c8555418097\" (UID: \"36a2ffd2-7556-4fea-a4d6-3c8555418097\") "
Sep 09 20:23:08.147999 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:23:08.147797    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/36a2ffd2-7556-4fea-a4d6-3c8555418097-cabundle-cert\") pod \"36a2ffd2-7556-4fea-a4d6-3c8555418097\" (UID: \"36a2ffd2-7556-4fea-a4d6-3c8555418097\") "
Sep 09 20:23:08.147999 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:23:08.147821    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8nkpn\" (UniqueName: \"kubernetes.io/projected/36a2ffd2-7556-4fea-a4d6-3c8555418097-kube-api-access-8nkpn\") pod \"36a2ffd2-7556-4fea-a4d6-3c8555418097\" (UID: \"36a2ffd2-7556-4fea-a4d6-3c8555418097\") "
Sep 09 20:23:08.147999 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:23:08.147885    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36a2ffd2-7556-4fea-a4d6-3c8555418097-kserve-provision-location\") pod \"36a2ffd2-7556-4fea-a4d6-3c8555418097\" (UID: \"36a2ffd2-7556-4fea-a4d6-3c8555418097\") "
Sep 09 20:23:08.148184 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:23:08.148134    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36a2ffd2-7556-4fea-a4d6-3c8555418097-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "36a2ffd2-7556-4fea-a4d6-3c8555418097" (UID: "36a2ffd2-7556-4fea-a4d6-3c8555418097"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:23:08.148250 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:23:08.148218    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36a2ffd2-7556-4fea-a4d6-3c8555418097-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "36a2ffd2-7556-4fea-a4d6-3c8555418097" (UID: "36a2ffd2-7556-4fea-a4d6-3c8555418097"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:23:08.149978 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:23:08.149943    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36a2ffd2-7556-4fea-a4d6-3c8555418097-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "36a2ffd2-7556-4fea-a4d6-3c8555418097" (UID: "36a2ffd2-7556-4fea-a4d6-3c8555418097"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:23:08.150082 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:23:08.150010    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36a2ffd2-7556-4fea-a4d6-3c8555418097-kube-api-access-8nkpn" (OuterVolumeSpecName: "kube-api-access-8nkpn") pod "36a2ffd2-7556-4fea-a4d6-3c8555418097" (UID: "36a2ffd2-7556-4fea-a4d6-3c8555418097"). InnerVolumeSpecName "kube-api-access-8nkpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:23:08.169634 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:23:08.169611    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36a2ffd2-7556-4fea-a4d6-3c8555418097-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "36a2ffd2-7556-4fea-a4d6-3c8555418097" (UID: "36a2ffd2-7556-4fea-a4d6-3c8555418097"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:23:08.248750 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:23:08.248725    2523 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36a2ffd2-7556-4fea-a4d6-3c8555418097-proxy-tls\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:23:08.248750 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:23:08.248748    2523 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/36a2ffd2-7556-4fea-a4d6-3c8555418097-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:23:08.248873 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:23:08.248762    2523 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/36a2ffd2-7556-4fea-a4d6-3c8555418097-cabundle-cert\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:23:08.248873 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:23:08.248776    2523 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nkpn\" (UniqueName: \"kubernetes.io/projected/36a2ffd2-7556-4fea-a4d6-3c8555418097-kube-api-access-8nkpn\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:23:08.248873 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:23:08.248789    2523 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36a2ffd2-7556-4fea-a4d6-3c8555418097-kserve-provision-location\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:23:09.078842 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:23:09.078817    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk"
Sep 09 20:23:09.118411 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:23:09.118384    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk"]
Sep 09 20:23:09.123893 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:23:09.123868    2523 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-79f4b454d7-rgmdk"]
Sep 09 20:23:09.810349 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:23:09.810318    2523 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36a2ffd2-7556-4fea-a4d6-3c8555418097" path="/var/lib/kubelet/pods/36a2ffd2-7556-4fea-a4d6-3c8555418097/volumes"
Sep 09 20:24:27.365853 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:27.365808    2523 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs"]
Sep 09 20:24:27.366321 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:27.366075    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c" containerName="kserve-container"
Sep 09 20:24:27.366321 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:27.366085    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c" containerName="kserve-container"
Sep 09 20:24:27.366321 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:27.366092    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36a2ffd2-7556-4fea-a4d6-3c8555418097" containerName="storage-initializer"
Sep 09 20:24:27.366321 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:27.366100    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="36a2ffd2-7556-4fea-a4d6-3c8555418097" containerName="storage-initializer"
Sep 09 20:24:27.366321 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:27.366106    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36a2ffd2-7556-4fea-a4d6-3c8555418097" containerName="kserve-container"
Sep 09 20:24:27.366321 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:27.366111    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="36a2ffd2-7556-4fea-a4d6-3c8555418097" containerName="kserve-container"
Sep 09 20:24:27.366321 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:27.366124    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c" containerName="storage-initializer"
Sep 09 20:24:27.366321 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:27.366128    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c" containerName="storage-initializer"
Sep 09 20:24:27.366321 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:27.366136    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36a2ffd2-7556-4fea-a4d6-3c8555418097" containerName="kube-rbac-proxy"
Sep 09 20:24:27.366321 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:27.366141    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="36a2ffd2-7556-4fea-a4d6-3c8555418097" containerName="kube-rbac-proxy"
Sep 09 20:24:27.366321 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:27.366164    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c" containerName="kube-rbac-proxy"
Sep 09 20:24:27.366321 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:27.366170    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c" containerName="kube-rbac-proxy"
Sep 09 20:24:27.366321 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:27.366215    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c" containerName="kserve-container"
Sep 09 20:24:27.366321 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:27.366222    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="36a2ffd2-7556-4fea-a4d6-3c8555418097" containerName="kube-rbac-proxy"
Sep 09 20:24:27.366321 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:27.366228    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa859455-d6d5-4d80-9e23-dc6a8a1ffc8c" containerName="kube-rbac-proxy"
Sep 09 20:24:27.366321 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:27.366235    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="36a2ffd2-7556-4fea-a4d6-3c8555418097" containerName="kserve-container"
Sep 09 20:24:27.369163 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:27.369135    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs"
Sep 09 20:24:27.372216 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:27.372192    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 20:24:27.373075 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:27.373053    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:24:27.373231 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:27.373079    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
Sep 09 20:24:27.373231 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:27.373085    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 20:24:27.373231 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:27.373186    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
Sep 09 20:24:27.373694 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:27.373258    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 20:24:27.373694 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:27.373348    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-ks7mb\""
Sep 09 20:24:27.400951 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:27.400928    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs"]
Sep 09 20:24:27.456110 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:27.456086    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8314408d-2b45-4592-999b-720bcf902e61-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs\" (UID: \"8314408d-2b45-4592-999b-720bcf902e61\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs"
Sep 09 20:24:27.456232 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:27.456117    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8314408d-2b45-4592-999b-720bcf902e61-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs\" (UID: \"8314408d-2b45-4592-999b-720bcf902e61\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs"
Sep 09 20:24:27.456232 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:27.456171    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8314408d-2b45-4592-999b-720bcf902e61-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs\" (UID: \"8314408d-2b45-4592-999b-720bcf902e61\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs"
Sep 09 20:24:27.456365 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:27.456243    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clsfd\" (UniqueName: \"kubernetes.io/projected/8314408d-2b45-4592-999b-720bcf902e61-kube-api-access-clsfd\") pod \"isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs\" (UID: \"8314408d-2b45-4592-999b-720bcf902e61\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs"
Sep 09 20:24:27.456365 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:27.456292    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8314408d-2b45-4592-999b-720bcf902e61-proxy-tls\") pod \"isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs\" (UID: \"8314408d-2b45-4592-999b-720bcf902e61\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs"
Sep 09 20:24:27.557076 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:27.557054    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8314408d-2b45-4592-999b-720bcf902e61-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs\" (UID: \"8314408d-2b45-4592-999b-720bcf902e61\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs"
Sep 09 20:24:27.557207 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:27.557081    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clsfd\" (UniqueName: \"kubernetes.io/projected/8314408d-2b45-4592-999b-720bcf902e61-kube-api-access-clsfd\") pod \"isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs\" (UID: \"8314408d-2b45-4592-999b-720bcf902e61\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs"
Sep 09 20:24:27.557207 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:27.557103    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8314408d-2b45-4592-999b-720bcf902e61-proxy-tls\") pod \"isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs\" (UID: \"8314408d-2b45-4592-999b-720bcf902e61\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs"
Sep 09 20:24:27.557207 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:27.557136    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8314408d-2b45-4592-999b-720bcf902e61-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs\" (UID: \"8314408d-2b45-4592-999b-720bcf902e61\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs"
Sep 09 20:24:27.557207 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:27.557171    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8314408d-2b45-4592-999b-720bcf902e61-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs\" (UID: \"8314408d-2b45-4592-999b-720bcf902e61\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs"
Sep 09 20:24:27.557380 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:24:27.557239    2523 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-predictor-serving-cert: secret "isvc-sklearn-v2-predictor-serving-cert" not found
Sep 09 20:24:27.557380 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:24:27.557301    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8314408d-2b45-4592-999b-720bcf902e61-proxy-tls podName:8314408d-2b45-4592-999b-720bcf902e61 nodeName:}" failed. No retries permitted until 2026-09-09 20:24:28.057280406 +0000 UTC m=+1772.729299382 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8314408d-2b45-4592-999b-720bcf902e61-proxy-tls") pod "isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs" (UID: "8314408d-2b45-4592-999b-720bcf902e61") : secret "isvc-sklearn-v2-predictor-serving-cert" not found
Sep 09 20:24:27.557570 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:27.557554    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8314408d-2b45-4592-999b-720bcf902e61-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs\" (UID: \"8314408d-2b45-4592-999b-720bcf902e61\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs"
Sep 09 20:24:27.557763 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:27.557747    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8314408d-2b45-4592-999b-720bcf902e61-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs\" (UID: \"8314408d-2b45-4592-999b-720bcf902e61\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs"
Sep 09 20:24:27.557798 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:27.557748    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8314408d-2b45-4592-999b-720bcf902e61-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs\" (UID: \"8314408d-2b45-4592-999b-720bcf902e61\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs"
Sep 09 20:24:27.567036 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:27.567014    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clsfd\" (UniqueName: \"kubernetes.io/projected/8314408d-2b45-4592-999b-720bcf902e61-kube-api-access-clsfd\") pod \"isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs\" (UID: \"8314408d-2b45-4592-999b-720bcf902e61\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs"
Sep 09 20:24:28.060463 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:28.060429    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8314408d-2b45-4592-999b-720bcf902e61-proxy-tls\") pod \"isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs\" (UID: \"8314408d-2b45-4592-999b-720bcf902e61\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs"
Sep 09 20:24:28.060595 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:24:28.060549    2523 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-predictor-serving-cert: secret "isvc-sklearn-v2-predictor-serving-cert" not found
Sep 09 20:24:28.060638 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:24:28.060601    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8314408d-2b45-4592-999b-720bcf902e61-proxy-tls podName:8314408d-2b45-4592-999b-720bcf902e61 nodeName:}" failed. No retries permitted until 2026-09-09 20:24:29.060587244 +0000 UTC m=+1773.732606218 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8314408d-2b45-4592-999b-720bcf902e61-proxy-tls") pod "isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs" (UID: "8314408d-2b45-4592-999b-720bcf902e61") : secret "isvc-sklearn-v2-predictor-serving-cert" not found
Sep 09 20:24:29.066591 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:29.066557    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8314408d-2b45-4592-999b-720bcf902e61-proxy-tls\") pod \"isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs\" (UID: \"8314408d-2b45-4592-999b-720bcf902e61\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs"
Sep 09 20:24:29.068934 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:29.068916    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8314408d-2b45-4592-999b-720bcf902e61-proxy-tls\") pod \"isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs\" (UID: \"8314408d-2b45-4592-999b-720bcf902e61\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs"
Sep 09 20:24:29.179016 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:29.178991    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs"
Sep 09 20:24:29.449791 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:29.449718    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs"]
Sep 09 20:24:29.452335 ip-10-0-132-226 kubenswrapper[2523]: W0909 20:24:29.452308    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8314408d_2b45_4592_999b_720bcf902e61.slice/crio-f669922aad88c723bab484f08b3ce0a749b95185a3cba1813516303332732d72 WatchSource:0}: Error finding container f669922aad88c723bab484f08b3ce0a749b95185a3cba1813516303332732d72: Status 404 returned error can't find the container with id f669922aad88c723bab484f08b3ce0a749b95185a3cba1813516303332732d72
Sep 09 20:24:30.371174 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:30.371123    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs" event={"ID":"8314408d-2b45-4592-999b-720bcf902e61","Type":"ContainerStarted","Data":"5cb6d5fbd930755e161a9dc30e9c58dd8d2c54a652233bf85039d5328aec6c02"}
Sep 09 20:24:30.371633 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:30.371208    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs" event={"ID":"8314408d-2b45-4592-999b-720bcf902e61","Type":"ContainerStarted","Data":"f669922aad88c723bab484f08b3ce0a749b95185a3cba1813516303332732d72"}
Sep 09 20:24:33.384842 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:33.384772    2523 generic.go:356] "Generic (PLEG): container finished" podID="8314408d-2b45-4592-999b-720bcf902e61" containerID="5cb6d5fbd930755e161a9dc30e9c58dd8d2c54a652233bf85039d5328aec6c02" exitCode=0
Sep 09 20:24:33.385271 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:33.384849    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs" event={"ID":"8314408d-2b45-4592-999b-720bcf902e61","Type":"ContainerDied","Data":"5cb6d5fbd930755e161a9dc30e9c58dd8d2c54a652233bf85039d5328aec6c02"}
Sep 09 20:24:34.390749 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:34.390716    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs" event={"ID":"8314408d-2b45-4592-999b-720bcf902e61","Type":"ContainerStarted","Data":"ddbbbeb9d53a913c5ddf46381a4d8a2c31ed7d390ae902b0dcd947c7287975f0"}
Sep 09 20:24:34.391189 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:34.390775    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs" event={"ID":"8314408d-2b45-4592-999b-720bcf902e61","Type":"ContainerStarted","Data":"802498ddfd40617ab870827c9bbea4e57fc8d268a4bbb95540ce28fb7e69cd45"}
Sep 09 20:24:34.391189 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:34.391090    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs"
Sep 09 20:24:34.391298 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:34.391245    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs"
Sep 09 20:24:34.392511 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:34.392488    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs" podUID="8314408d-2b45-4592-999b-720bcf902e61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 09 20:24:34.422812 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:34.422753    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs" podStartSLOduration=3.839026461 podStartE2EDuration="7.422737115s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.583710654s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:24:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:24:34.422393726 +0000 UTC m=+1779.094412724" watchObservedRunningTime="2026-09-09 20:24:34.422737115 +0000 UTC m=+1779.094756114"
Sep 09 20:24:35.394200 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:35.394161    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs" podUID="8314408d-2b45-4592-999b-720bcf902e61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 09 20:24:40.399341 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:40.399314    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs"
Sep 09 20:24:40.399798 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:40.399773    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs" podUID="8314408d-2b45-4592-999b-720bcf902e61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 09 20:24:50.400733 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:24:50.400648    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs" podUID="8314408d-2b45-4592-999b-720bcf902e61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 09 20:25:00.400517 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:00.400482    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs" podUID="8314408d-2b45-4592-999b-720bcf902e61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 09 20:25:10.400443 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:10.400406    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs" podUID="8314408d-2b45-4592-999b-720bcf902e61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 09 20:25:20.400526 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:20.400487    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs" podUID="8314408d-2b45-4592-999b-720bcf902e61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 09 20:25:30.400600 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:30.400561    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs" podUID="8314408d-2b45-4592-999b-720bcf902e61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 09 20:25:40.400676 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:40.400638    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs"
Sep 09 20:25:47.439746 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:47.439710    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs"]
Sep 09 20:25:47.440370 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:47.439938    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs" podUID="8314408d-2b45-4592-999b-720bcf902e61" containerName="kserve-container" containerID="cri-o://802498ddfd40617ab870827c9bbea4e57fc8d268a4bbb95540ce28fb7e69cd45" gracePeriod=30
Sep 09 20:25:47.440370 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:47.439998    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs" podUID="8314408d-2b45-4592-999b-720bcf902e61" containerName="kube-rbac-proxy" containerID="cri-o://ddbbbeb9d53a913c5ddf46381a4d8a2c31ed7d390ae902b0dcd947c7287975f0" gracePeriod=30
Sep 09 20:25:47.660828 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:47.660798    2523 generic.go:356] "Generic (PLEG): container finished" podID="8314408d-2b45-4592-999b-720bcf902e61" containerID="ddbbbeb9d53a913c5ddf46381a4d8a2c31ed7d390ae902b0dcd947c7287975f0" exitCode=2
Sep 09 20:25:47.660964 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:47.660840    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs" event={"ID":"8314408d-2b45-4592-999b-720bcf902e61","Type":"ContainerDied","Data":"ddbbbeb9d53a913c5ddf46381a4d8a2c31ed7d390ae902b0dcd947c7287975f0"}
Sep 09 20:25:50.395208 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:50.395142    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs" podUID="8314408d-2b45-4592-999b-720bcf902e61" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Sep 09 20:25:50.400511 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:50.400478    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs" podUID="8314408d-2b45-4592-999b-720bcf902e61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 09 20:25:51.272884 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:51.272862    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs"
Sep 09 20:25:51.427939 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:51.427913    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8314408d-2b45-4592-999b-720bcf902e61-cabundle-cert\") pod \"8314408d-2b45-4592-999b-720bcf902e61\" (UID: \"8314408d-2b45-4592-999b-720bcf902e61\") "
Sep 09 20:25:51.428365 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:51.427954    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-clsfd\" (UniqueName: \"kubernetes.io/projected/8314408d-2b45-4592-999b-720bcf902e61-kube-api-access-clsfd\") pod \"8314408d-2b45-4592-999b-720bcf902e61\" (UID: \"8314408d-2b45-4592-999b-720bcf902e61\") "
Sep 09 20:25:51.428365 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:51.427978    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8314408d-2b45-4592-999b-720bcf902e61-kube-rbac-proxy-sar-config\") pod \"8314408d-2b45-4592-999b-720bcf902e61\" (UID: \"8314408d-2b45-4592-999b-720bcf902e61\") "
Sep 09 20:25:51.428365 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:51.428003    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8314408d-2b45-4592-999b-720bcf902e61-kserve-provision-location\") pod \"8314408d-2b45-4592-999b-720bcf902e61\" (UID: \"8314408d-2b45-4592-999b-720bcf902e61\") "
Sep 09 20:25:51.428365 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:51.428029    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8314408d-2b45-4592-999b-720bcf902e61-proxy-tls\") pod \"8314408d-2b45-4592-999b-720bcf902e61\" (UID: \"8314408d-2b45-4592-999b-720bcf902e61\") "
Sep 09 20:25:51.428365 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:51.428352    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8314408d-2b45-4592-999b-720bcf902e61-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8314408d-2b45-4592-999b-720bcf902e61" (UID: "8314408d-2b45-4592-999b-720bcf902e61"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:25:51.428568 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:51.428374    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8314408d-2b45-4592-999b-720bcf902e61-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8314408d-2b45-4592-999b-720bcf902e61" (UID: "8314408d-2b45-4592-999b-720bcf902e61"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:25:51.428568 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:51.428417    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8314408d-2b45-4592-999b-720bcf902e61-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8314408d-2b45-4592-999b-720bcf902e61" (UID: "8314408d-2b45-4592-999b-720bcf902e61"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:25:51.430117 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:51.430095    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8314408d-2b45-4592-999b-720bcf902e61-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8314408d-2b45-4592-999b-720bcf902e61" (UID: "8314408d-2b45-4592-999b-720bcf902e61"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:25:51.430277 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:51.430260    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8314408d-2b45-4592-999b-720bcf902e61-kube-api-access-clsfd" (OuterVolumeSpecName: "kube-api-access-clsfd") pod "8314408d-2b45-4592-999b-720bcf902e61" (UID: "8314408d-2b45-4592-999b-720bcf902e61"). InnerVolumeSpecName "kube-api-access-clsfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:25:51.529195 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:51.529173    2523 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8314408d-2b45-4592-999b-720bcf902e61-proxy-tls\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:25:51.529195 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:51.529194    2523 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8314408d-2b45-4592-999b-720bcf902e61-cabundle-cert\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:25:51.529333 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:51.529203    2523 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clsfd\" (UniqueName: \"kubernetes.io/projected/8314408d-2b45-4592-999b-720bcf902e61-kube-api-access-clsfd\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:25:51.529333 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:51.529211    2523 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8314408d-2b45-4592-999b-720bcf902e61-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:25:51.529333 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:51.529220    2523 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8314408d-2b45-4592-999b-720bcf902e61-kserve-provision-location\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:25:51.677855 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:51.677826    2523 generic.go:356] "Generic (PLEG): container finished" podID="8314408d-2b45-4592-999b-720bcf902e61" containerID="802498ddfd40617ab870827c9bbea4e57fc8d268a4bbb95540ce28fb7e69cd45" exitCode=0
Sep 09 20:25:51.677967 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:51.677887    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs"
Sep 09 20:25:51.677967 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:51.677911    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs" event={"ID":"8314408d-2b45-4592-999b-720bcf902e61","Type":"ContainerDied","Data":"802498ddfd40617ab870827c9bbea4e57fc8d268a4bbb95540ce28fb7e69cd45"}
Sep 09 20:25:51.678045 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:51.677972    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs" event={"ID":"8314408d-2b45-4592-999b-720bcf902e61","Type":"ContainerDied","Data":"f669922aad88c723bab484f08b3ce0a749b95185a3cba1813516303332732d72"}
Sep 09 20:25:51.678045 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:51.677999    2523 scope.go:117] "RemoveContainer" containerID="ddbbbeb9d53a913c5ddf46381a4d8a2c31ed7d390ae902b0dcd947c7287975f0"
Sep 09 20:25:51.686464 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:51.686442    2523 scope.go:117] "RemoveContainer" containerID="802498ddfd40617ab870827c9bbea4e57fc8d268a4bbb95540ce28fb7e69cd45"
Sep 09 20:25:51.693829 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:51.693815    2523 scope.go:117] "RemoveContainer" containerID="5cb6d5fbd930755e161a9dc30e9c58dd8d2c54a652233bf85039d5328aec6c02"
Sep 09 20:25:51.699904 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:51.699860    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs"]
Sep 09 20:25:51.701125 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:51.701093    2523 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-7fc6cff9ff-ftggs"]
Sep 09 20:25:51.701403 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:51.701388    2523 scope.go:117] "RemoveContainer" containerID="ddbbbeb9d53a913c5ddf46381a4d8a2c31ed7d390ae902b0dcd947c7287975f0"
Sep 09 20:25:51.701674 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:25:51.701658    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddbbbeb9d53a913c5ddf46381a4d8a2c31ed7d390ae902b0dcd947c7287975f0\": container with ID starting with ddbbbeb9d53a913c5ddf46381a4d8a2c31ed7d390ae902b0dcd947c7287975f0 not found: ID does not exist" containerID="ddbbbeb9d53a913c5ddf46381a4d8a2c31ed7d390ae902b0dcd947c7287975f0"
Sep 09 20:25:51.701752 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:51.701684    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddbbbeb9d53a913c5ddf46381a4d8a2c31ed7d390ae902b0dcd947c7287975f0"} err="failed to get container status \"ddbbbeb9d53a913c5ddf46381a4d8a2c31ed7d390ae902b0dcd947c7287975f0\": rpc error: code = NotFound desc = could not find container \"ddbbbeb9d53a913c5ddf46381a4d8a2c31ed7d390ae902b0dcd947c7287975f0\": container with ID starting with ddbbbeb9d53a913c5ddf46381a4d8a2c31ed7d390ae902b0dcd947c7287975f0 not found: ID does not exist"
Sep 09 20:25:51.701752 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:51.701707    2523 scope.go:117] "RemoveContainer" containerID="802498ddfd40617ab870827c9bbea4e57fc8d268a4bbb95540ce28fb7e69cd45"
Sep 09 20:25:51.701952 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:25:51.701935    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"802498ddfd40617ab870827c9bbea4e57fc8d268a4bbb95540ce28fb7e69cd45\": container with ID starting with 802498ddfd40617ab870827c9bbea4e57fc8d268a4bbb95540ce28fb7e69cd45 not found: ID does not exist" containerID="802498ddfd40617ab870827c9bbea4e57fc8d268a4bbb95540ce28fb7e69cd45"
Sep 09 20:25:51.701993 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:51.701959    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"802498ddfd40617ab870827c9bbea4e57fc8d268a4bbb95540ce28fb7e69cd45"} err="failed to get container status \"802498ddfd40617ab870827c9bbea4e57fc8d268a4bbb95540ce28fb7e69cd45\": rpc error: code = NotFound desc = could not find container \"802498ddfd40617ab870827c9bbea4e57fc8d268a4bbb95540ce28fb7e69cd45\": container with ID starting with 802498ddfd40617ab870827c9bbea4e57fc8d268a4bbb95540ce28fb7e69cd45 not found: ID does not exist"
Sep 09 20:25:51.701993 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:51.701975    2523 scope.go:117] "RemoveContainer" containerID="5cb6d5fbd930755e161a9dc30e9c58dd8d2c54a652233bf85039d5328aec6c02"
Sep 09 20:25:51.702228 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:25:51.702214    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cb6d5fbd930755e161a9dc30e9c58dd8d2c54a652233bf85039d5328aec6c02\": container with ID starting with 5cb6d5fbd930755e161a9dc30e9c58dd8d2c54a652233bf85039d5328aec6c02 not found: ID does not exist" containerID="5cb6d5fbd930755e161a9dc30e9c58dd8d2c54a652233bf85039d5328aec6c02"
Sep 09 20:25:51.702275 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:51.702233    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cb6d5fbd930755e161a9dc30e9c58dd8d2c54a652233bf85039d5328aec6c02"} err="failed to get container status \"5cb6d5fbd930755e161a9dc30e9c58dd8d2c54a652233bf85039d5328aec6c02\": rpc error: code = NotFound desc = could not find container \"5cb6d5fbd930755e161a9dc30e9c58dd8d2c54a652233bf85039d5328aec6c02\": container with ID starting with 5cb6d5fbd930755e161a9dc30e9c58dd8d2c54a652233bf85039d5328aec6c02 not found: ID does not exist"
Sep 09 20:25:51.807024 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:51.806997    2523 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8314408d-2b45-4592-999b-720bcf902e61" path="/var/lib/kubelet/pods/8314408d-2b45-4592-999b-720bcf902e61/volumes"
Sep 09 20:25:52.646041 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:52.646010    2523 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh"]
Sep 09 20:25:52.646478 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:52.646278    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8314408d-2b45-4592-999b-720bcf902e61" containerName="kube-rbac-proxy"
Sep 09 20:25:52.646478 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:52.646290    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="8314408d-2b45-4592-999b-720bcf902e61" containerName="kube-rbac-proxy"
Sep 09 20:25:52.646478 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:52.646303    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8314408d-2b45-4592-999b-720bcf902e61" containerName="storage-initializer"
Sep 09 20:25:52.646478 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:52.646309    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="8314408d-2b45-4592-999b-720bcf902e61" containerName="storage-initializer"
Sep 09 20:25:52.646478 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:52.646323    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8314408d-2b45-4592-999b-720bcf902e61" containerName="kserve-container"
Sep 09 20:25:52.646478 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:52.646328    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="8314408d-2b45-4592-999b-720bcf902e61" containerName="kserve-container"
Sep 09 20:25:52.646478 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:52.646370    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="8314408d-2b45-4592-999b-720bcf902e61" containerName="kube-rbac-proxy"
Sep 09 20:25:52.646478 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:52.646378    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="8314408d-2b45-4592-999b-720bcf902e61" containerName="kserve-container"
Sep 09 20:25:52.650638 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:52.650622    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh"
Sep 09 20:25:52.653057 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:52.653040    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 20:25:52.653251 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:52.653234    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 20:25:52.653311 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:52.653268    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-ks7mb\""
Sep 09 20:25:52.654047 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:52.654020    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Sep 09 20:25:52.654134 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:52.654100    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Sep 09 20:25:52.654227 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:52.654209    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:25:52.654432 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:52.654416    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 20:25:52.659383 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:52.659363    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh"]
Sep 09 20:25:52.738055 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:52.738029    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a359bd28-730b-4de8-91fd-48338328ac99-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh\" (UID: \"a359bd28-730b-4de8-91fd-48338328ac99\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh"
Sep 09 20:25:52.738185 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:52.738063    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a359bd28-730b-4de8-91fd-48338328ac99-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh\" (UID: \"a359bd28-730b-4de8-91fd-48338328ac99\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh"
Sep 09 20:25:52.738185 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:52.738136    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz69g\" (UniqueName: \"kubernetes.io/projected/a359bd28-730b-4de8-91fd-48338328ac99-kube-api-access-fz69g\") pod \"isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh\" (UID: \"a359bd28-730b-4de8-91fd-48338328ac99\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh"
Sep 09 20:25:52.738270 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:52.738200    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a359bd28-730b-4de8-91fd-48338328ac99-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh\" (UID: \"a359bd28-730b-4de8-91fd-48338328ac99\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh"
Sep 09 20:25:52.738270 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:52.738228    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a359bd28-730b-4de8-91fd-48338328ac99-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh\" (UID: \"a359bd28-730b-4de8-91fd-48338328ac99\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh"
Sep 09 20:25:52.838634 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:52.838609    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz69g\" (UniqueName: \"kubernetes.io/projected/a359bd28-730b-4de8-91fd-48338328ac99-kube-api-access-fz69g\") pod \"isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh\" (UID: \"a359bd28-730b-4de8-91fd-48338328ac99\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh"
Sep 09 20:25:52.838743 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:52.838640    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a359bd28-730b-4de8-91fd-48338328ac99-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh\" (UID: \"a359bd28-730b-4de8-91fd-48338328ac99\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh"
Sep 09 20:25:52.838743 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:52.838672    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a359bd28-730b-4de8-91fd-48338328ac99-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh\" (UID: \"a359bd28-730b-4de8-91fd-48338328ac99\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh"
Sep 09 20:25:52.838743 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:52.838703    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a359bd28-730b-4de8-91fd-48338328ac99-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh\" (UID: \"a359bd28-730b-4de8-91fd-48338328ac99\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh"
Sep 09 20:25:52.838743 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:52.838736    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a359bd28-730b-4de8-91fd-48338328ac99-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh\" (UID: \"a359bd28-730b-4de8-91fd-48338328ac99\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh"
Sep 09 20:25:52.839043 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:52.839022    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a359bd28-730b-4de8-91fd-48338328ac99-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh\" (UID: \"a359bd28-730b-4de8-91fd-48338328ac99\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh"
Sep 09 20:25:52.839341 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:52.839321    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a359bd28-730b-4de8-91fd-48338328ac99-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh\" (UID: \"a359bd28-730b-4de8-91fd-48338328ac99\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh"
Sep 09 20:25:52.839435 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:52.839330    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a359bd28-730b-4de8-91fd-48338328ac99-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh\" (UID: \"a359bd28-730b-4de8-91fd-48338328ac99\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh"
Sep 09 20:25:52.841270 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:52.841245    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a359bd28-730b-4de8-91fd-48338328ac99-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh\" (UID: \"a359bd28-730b-4de8-91fd-48338328ac99\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh"
Sep 09 20:25:52.846822 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:52.846802    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz69g\" (UniqueName: \"kubernetes.io/projected/a359bd28-730b-4de8-91fd-48338328ac99-kube-api-access-fz69g\") pod \"isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh\" (UID: \"a359bd28-730b-4de8-91fd-48338328ac99\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh"
Sep 09 20:25:52.960586 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:52.960521    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh"
Sep 09 20:25:52.992951 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:52.992923    2523 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-6kf8r"]
Sep 09 20:25:52.996995 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:52.996512    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-6kf8r"
Sep 09 20:25:52.998894 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:52.998872    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 09 20:25:52.999010 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:52.998972    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 09 20:25:53.006215 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:53.006191    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-6kf8r"]
Sep 09 20:25:53.035191 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:53.034741    2523 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-7x7bm"]
Sep 09 20:25:53.038130 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:53.037913    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-7x7bm"
Sep 09 20:25:53.041879 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:53.041858    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 09 20:25:53.049164 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:53.049124    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-7x7bm"]
Sep 09 20:25:53.141489 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:53.141465    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e36dc0c-fb8a-4970-9d19-1a52f717bedc-proxy-tls\") pod \"isvc-canary-create-predictor-68984b9498-7x7bm\" (UID: \"6e36dc0c-fb8a-4970-9d19-1a52f717bedc\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-7x7bm"
Sep 09 20:25:53.141595 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:53.141498    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45cc6423-62bf-4b6b-bb9b-2f3a1cde1639-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-579784d7d7-6kf8r\" (UID: \"45cc6423-62bf-4b6b-bb9b-2f3a1cde1639\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-6kf8r"
Sep 09 20:25:53.141595 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:53.141521    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsnsv\" (UniqueName: \"kubernetes.io/projected/45cc6423-62bf-4b6b-bb9b-2f3a1cde1639-kube-api-access-vsnsv\") pod \"isvc-canary-create-v2-predictor-579784d7d7-6kf8r\" (UID: \"45cc6423-62bf-4b6b-bb9b-2f3a1cde1639\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-6kf8r"
Sep 09 20:25:53.141595 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:53.141555    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45cc6423-62bf-4b6b-bb9b-2f3a1cde1639-proxy-tls\") pod \"isvc-canary-create-v2-predictor-579784d7d7-6kf8r\" (UID: \"45cc6423-62bf-4b6b-bb9b-2f3a1cde1639\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-6kf8r"
Sep 09 20:25:53.141595 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:53.141570    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e36dc0c-fb8a-4970-9d19-1a52f717bedc-kserve-provision-location\") pod \"isvc-canary-create-predictor-68984b9498-7x7bm\" (UID: \"6e36dc0c-fb8a-4970-9d19-1a52f717bedc\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-7x7bm"
Sep 09 20:25:53.141595 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:53.141592    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e36dc0c-fb8a-4970-9d19-1a52f717bedc-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-68984b9498-7x7bm\" (UID: \"6e36dc0c-fb8a-4970-9d19-1a52f717bedc\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-7x7bm"
Sep 09 20:25:53.141856 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:53.141607    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e36dc0c-fb8a-4970-9d19-1a52f717bedc-cabundle-cert\") pod \"isvc-canary-create-predictor-68984b9498-7x7bm\" (UID: \"6e36dc0c-fb8a-4970-9d19-1a52f717bedc\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-7x7bm"
Sep 09 20:25:53.141856 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:53.141627    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45cc6423-62bf-4b6b-bb9b-2f3a1cde1639-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-579784d7d7-6kf8r\" (UID: \"45cc6423-62bf-4b6b-bb9b-2f3a1cde1639\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-6kf8r"
Sep 09 20:25:53.141856 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:53.141679    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45cc6423-62bf-4b6b-bb9b-2f3a1cde1639-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-579784d7d7-6kf8r\" (UID: \"45cc6423-62bf-4b6b-bb9b-2f3a1cde1639\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-6kf8r"
Sep 09 20:25:53.141856 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:53.141781    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxm26\" (UniqueName: \"kubernetes.io/projected/6e36dc0c-fb8a-4970-9d19-1a52f717bedc-kube-api-access-nxm26\") pod \"isvc-canary-create-predictor-68984b9498-7x7bm\" (UID: \"6e36dc0c-fb8a-4970-9d19-1a52f717bedc\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-7x7bm"
Sep 09 20:25:53.232651 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:53.232631    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh"]
Sep 09 20:25:53.235136 ip-10-0-132-226 kubenswrapper[2523]: W0909 20:25:53.235113    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda359bd28_730b_4de8_91fd_48338328ac99.slice/crio-118fcd64652340c9aa4fa5021ea07e4c246b0f343f1121f82124f028d9b3c53c WatchSource:0}: Error finding container 118fcd64652340c9aa4fa5021ea07e4c246b0f343f1121f82124f028d9b3c53c: Status 404 returned error can't find the container with id 118fcd64652340c9aa4fa5021ea07e4c246b0f343f1121f82124f028d9b3c53c
Sep 09 20:25:53.242576 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:53.242558    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45cc6423-62bf-4b6b-bb9b-2f3a1cde1639-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-579784d7d7-6kf8r\" (UID: \"45cc6423-62bf-4b6b-bb9b-2f3a1cde1639\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-6kf8r"
Sep 09 20:25:53.242661 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:53.242590    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsnsv\" (UniqueName: \"kubernetes.io/projected/45cc6423-62bf-4b6b-bb9b-2f3a1cde1639-kube-api-access-vsnsv\") pod \"isvc-canary-create-v2-predictor-579784d7d7-6kf8r\" (UID: \"45cc6423-62bf-4b6b-bb9b-2f3a1cde1639\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-6kf8r"
Sep 09 20:25:53.242661 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:53.242609    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45cc6423-62bf-4b6b-bb9b-2f3a1cde1639-proxy-tls\") pod \"isvc-canary-create-v2-predictor-579784d7d7-6kf8r\" (UID: \"45cc6423-62bf-4b6b-bb9b-2f3a1cde1639\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-6kf8r"
Sep 09 20:25:53.242767 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:25:53.242705    2523 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-v2-predictor-serving-cert: secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 09 20:25:53.242767 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:53.242725    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e36dc0c-fb8a-4970-9d19-1a52f717bedc-kserve-provision-location\") pod \"isvc-canary-create-predictor-68984b9498-7x7bm\" (UID: \"6e36dc0c-fb8a-4970-9d19-1a52f717bedc\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-7x7bm"
Sep 09 20:25:53.242767 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:25:53.242758    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45cc6423-62bf-4b6b-bb9b-2f3a1cde1639-proxy-tls podName:45cc6423-62bf-4b6b-bb9b-2f3a1cde1639 nodeName:}" failed. No retries permitted until 2026-09-09 20:25:53.742738421 +0000 UTC m=+1858.414757400 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/45cc6423-62bf-4b6b-bb9b-2f3a1cde1639-proxy-tls") pod "isvc-canary-create-v2-predictor-579784d7d7-6kf8r" (UID: "45cc6423-62bf-4b6b-bb9b-2f3a1cde1639") : secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 09 20:25:53.242924 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:53.242797    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e36dc0c-fb8a-4970-9d19-1a52f717bedc-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-68984b9498-7x7bm\" (UID: \"6e36dc0c-fb8a-4970-9d19-1a52f717bedc\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-7x7bm"
Sep 09 20:25:53.242924 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:53.242821    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e36dc0c-fb8a-4970-9d19-1a52f717bedc-cabundle-cert\") pod \"isvc-canary-create-predictor-68984b9498-7x7bm\" (UID: \"6e36dc0c-fb8a-4970-9d19-1a52f717bedc\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-7x7bm"
Sep 09 20:25:53.243027 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:53.242918    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45cc6423-62bf-4b6b-bb9b-2f3a1cde1639-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-579784d7d7-6kf8r\" (UID: \"45cc6423-62bf-4b6b-bb9b-2f3a1cde1639\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-6kf8r"
Sep 09 20:25:53.243027 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:53.242961    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45cc6423-62bf-4b6b-bb9b-2f3a1cde1639-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-579784d7d7-6kf8r\" (UID: \"45cc6423-62bf-4b6b-bb9b-2f3a1cde1639\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-6kf8r"
Sep 09 20:25:53.243027 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:53.242968    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45cc6423-62bf-4b6b-bb9b-2f3a1cde1639-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-579784d7d7-6kf8r\" (UID: \"45cc6423-62bf-4b6b-bb9b-2f3a1cde1639\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-6kf8r"
Sep 09 20:25:53.243027 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:53.243006    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxm26\" (UniqueName: \"kubernetes.io/projected/6e36dc0c-fb8a-4970-9d19-1a52f717bedc-kube-api-access-nxm26\") pod \"isvc-canary-create-predictor-68984b9498-7x7bm\" (UID: \"6e36dc0c-fb8a-4970-9d19-1a52f717bedc\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-7x7bm"
Sep 09 20:25:53.243254 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:53.243040    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e36dc0c-fb8a-4970-9d19-1a52f717bedc-kserve-provision-location\") pod \"isvc-canary-create-predictor-68984b9498-7x7bm\" (UID: \"6e36dc0c-fb8a-4970-9d19-1a52f717bedc\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-7x7bm"
Sep 09 20:25:53.243254 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:53.243047    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e36dc0c-fb8a-4970-9d19-1a52f717bedc-proxy-tls\") pod \"isvc-canary-create-predictor-68984b9498-7x7bm\" (UID: \"6e36dc0c-fb8a-4970-9d19-1a52f717bedc\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-7x7bm"
Sep 09 20:25:53.243634 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:53.243609    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e36dc0c-fb8a-4970-9d19-1a52f717bedc-cabundle-cert\") pod \"isvc-canary-create-predictor-68984b9498-7x7bm\" (UID: \"6e36dc0c-fb8a-4970-9d19-1a52f717bedc\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-7x7bm"
Sep 09 20:25:53.243634 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:53.243626    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e36dc0c-fb8a-4970-9d19-1a52f717bedc-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-68984b9498-7x7bm\" (UID: \"6e36dc0c-fb8a-4970-9d19-1a52f717bedc\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-7x7bm"
Sep 09 20:25:53.243804 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:53.243726    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45cc6423-62bf-4b6b-bb9b-2f3a1cde1639-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-579784d7d7-6kf8r\" (UID: \"45cc6423-62bf-4b6b-bb9b-2f3a1cde1639\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-6kf8r"
Sep 09 20:25:53.243919 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:53.243893    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45cc6423-62bf-4b6b-bb9b-2f3a1cde1639-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-579784d7d7-6kf8r\" (UID: \"45cc6423-62bf-4b6b-bb9b-2f3a1cde1639\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-6kf8r"
Sep 09 20:25:53.245527 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:53.245507    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e36dc0c-fb8a-4970-9d19-1a52f717bedc-proxy-tls\") pod \"isvc-canary-create-predictor-68984b9498-7x7bm\" (UID: \"6e36dc0c-fb8a-4970-9d19-1a52f717bedc\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-7x7bm"
Sep 09 20:25:53.253686 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:53.253655    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsnsv\" (UniqueName: \"kubernetes.io/projected/45cc6423-62bf-4b6b-bb9b-2f3a1cde1639-kube-api-access-vsnsv\") pod \"isvc-canary-create-v2-predictor-579784d7d7-6kf8r\" (UID: \"45cc6423-62bf-4b6b-bb9b-2f3a1cde1639\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-6kf8r"
Sep 09 20:25:53.254936 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:53.254916    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxm26\" (UniqueName: \"kubernetes.io/projected/6e36dc0c-fb8a-4970-9d19-1a52f717bedc-kube-api-access-nxm26\") pod \"isvc-canary-create-predictor-68984b9498-7x7bm\" (UID: \"6e36dc0c-fb8a-4970-9d19-1a52f717bedc\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-7x7bm"
Sep 09 20:25:53.348200 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:53.348171    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-7x7bm"
Sep 09 20:25:53.618946 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:53.618904    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-7x7bm"]
Sep 09 20:25:53.621066 ip-10-0-132-226 kubenswrapper[2523]: W0909 20:25:53.621039    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e36dc0c_fb8a_4970_9d19_1a52f717bedc.slice/crio-69ce4d6f589f3120f3641e272bdf58f836e9b30b0cb5d7c2fc7ae0f404b80b74 WatchSource:0}: Error finding container 69ce4d6f589f3120f3641e272bdf58f836e9b30b0cb5d7c2fc7ae0f404b80b74: Status 404 returned error can't find the container with id 69ce4d6f589f3120f3641e272bdf58f836e9b30b0cb5d7c2fc7ae0f404b80b74
Sep 09 20:25:53.689226 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:53.689195    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-7x7bm" event={"ID":"6e36dc0c-fb8a-4970-9d19-1a52f717bedc","Type":"ContainerStarted","Data":"d5cb6534e9300ef99f4107776fdaa6603af3a90082364edaa478d4a5b74f5353"}
Sep 09 20:25:53.689545 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:53.689269    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-7x7bm" event={"ID":"6e36dc0c-fb8a-4970-9d19-1a52f717bedc","Type":"ContainerStarted","Data":"69ce4d6f589f3120f3641e272bdf58f836e9b30b0cb5d7c2fc7ae0f404b80b74"}
Sep 09 20:25:53.690960 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:53.690938    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh" event={"ID":"a359bd28-730b-4de8-91fd-48338328ac99","Type":"ContainerStarted","Data":"50339aad15389bf5b3509c49373dd38269bc342cbd5fa052eecb4718a5cd0b6c"}
Sep 09 20:25:53.691059 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:53.690984    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh" event={"ID":"a359bd28-730b-4de8-91fd-48338328ac99","Type":"ContainerStarted","Data":"118fcd64652340c9aa4fa5021ea07e4c246b0f343f1121f82124f028d9b3c53c"}
Sep 09 20:25:53.747059 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:53.747032    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45cc6423-62bf-4b6b-bb9b-2f3a1cde1639-proxy-tls\") pod \"isvc-canary-create-v2-predictor-579784d7d7-6kf8r\" (UID: \"45cc6423-62bf-4b6b-bb9b-2f3a1cde1639\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-6kf8r"
Sep 09 20:25:53.747252 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:25:53.747231    2523 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-v2-predictor-serving-cert: secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 09 20:25:53.747323 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:25:53.747309    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45cc6423-62bf-4b6b-bb9b-2f3a1cde1639-proxy-tls podName:45cc6423-62bf-4b6b-bb9b-2f3a1cde1639 nodeName:}" failed. No retries permitted until 2026-09-09 20:25:54.747288658 +0000 UTC m=+1859.419307656 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/45cc6423-62bf-4b6b-bb9b-2f3a1cde1639-proxy-tls") pod "isvc-canary-create-v2-predictor-579784d7d7-6kf8r" (UID: "45cc6423-62bf-4b6b-bb9b-2f3a1cde1639") : secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 09 20:25:54.753541 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:54.753508    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45cc6423-62bf-4b6b-bb9b-2f3a1cde1639-proxy-tls\") pod \"isvc-canary-create-v2-predictor-579784d7d7-6kf8r\" (UID: \"45cc6423-62bf-4b6b-bb9b-2f3a1cde1639\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-6kf8r"
Sep 09 20:25:54.755955 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:54.755932    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45cc6423-62bf-4b6b-bb9b-2f3a1cde1639-proxy-tls\") pod \"isvc-canary-create-v2-predictor-579784d7d7-6kf8r\" (UID: \"45cc6423-62bf-4b6b-bb9b-2f3a1cde1639\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-6kf8r"
Sep 09 20:25:54.808763 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:54.808731    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-6kf8r"
Sep 09 20:25:55.081301 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:55.081275    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-6kf8r"]
Sep 09 20:25:55.083651 ip-10-0-132-226 kubenswrapper[2523]: W0909 20:25:55.083627    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45cc6423_62bf_4b6b_bb9b_2f3a1cde1639.slice/crio-348d8054d51612be3b44c7d8f0d5a4a2fc9b589b85215e53ed0ca231aa2dcffd WatchSource:0}: Error finding container 348d8054d51612be3b44c7d8f0d5a4a2fc9b589b85215e53ed0ca231aa2dcffd: Status 404 returned error can't find the container with id 348d8054d51612be3b44c7d8f0d5a4a2fc9b589b85215e53ed0ca231aa2dcffd
Sep 09 20:25:55.700531 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:55.700490    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-6kf8r" event={"ID":"45cc6423-62bf-4b6b-bb9b-2f3a1cde1639","Type":"ContainerStarted","Data":"7eb0ba2c179e7e2513036b01d280fbb83ac1cea098273877ed6a78bff62291b2"}
Sep 09 20:25:55.700726 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:55.700543    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-6kf8r" event={"ID":"45cc6423-62bf-4b6b-bb9b-2f3a1cde1639","Type":"ContainerStarted","Data":"348d8054d51612be3b44c7d8f0d5a4a2fc9b589b85215e53ed0ca231aa2dcffd"}
Sep 09 20:25:57.708839 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:57.708806    2523 generic.go:356] "Generic (PLEG): container finished" podID="a359bd28-730b-4de8-91fd-48338328ac99" containerID="50339aad15389bf5b3509c49373dd38269bc342cbd5fa052eecb4718a5cd0b6c" exitCode=0
Sep 09 20:25:57.709235 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:57.708876    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh" event={"ID":"a359bd28-730b-4de8-91fd-48338328ac99","Type":"ContainerDied","Data":"50339aad15389bf5b3509c49373dd38269bc342cbd5fa052eecb4718a5cd0b6c"}
Sep 09 20:25:57.710458 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:57.710436    2523 generic.go:356] "Generic (PLEG): container finished" podID="6e36dc0c-fb8a-4970-9d19-1a52f717bedc" containerID="d5cb6534e9300ef99f4107776fdaa6603af3a90082364edaa478d4a5b74f5353" exitCode=0
Sep 09 20:25:57.710591 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:57.710490    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-7x7bm" event={"ID":"6e36dc0c-fb8a-4970-9d19-1a52f717bedc","Type":"ContainerDied","Data":"d5cb6534e9300ef99f4107776fdaa6603af3a90082364edaa478d4a5b74f5353"}
Sep 09 20:25:58.717015 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:58.716983    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh" event={"ID":"a359bd28-730b-4de8-91fd-48338328ac99","Type":"ContainerStarted","Data":"61a8503a036eb93d720186774c2ced1f69aba2915b98442c123fb4a3d4619984"}
Sep 09 20:25:58.717464 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:58.717059    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh" event={"ID":"a359bd28-730b-4de8-91fd-48338328ac99","Type":"ContainerStarted","Data":"0b8f96a930a64d5f36ef14ee4409a4440d882fb598c65b8d225f38752548a671"}
Sep 09 20:25:58.717464 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:58.717362    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh"
Sep 09 20:25:58.717588 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:58.717535    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh"
Sep 09 20:25:58.718920 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:58.718885    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh" podUID="a359bd28-730b-4de8-91fd-48338328ac99" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 09 20:25:58.719303 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:58.719284    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-7x7bm" event={"ID":"6e36dc0c-fb8a-4970-9d19-1a52f717bedc","Type":"ContainerStarted","Data":"38f3ac85a9ad0693ce645ff5ffbddff733d3975962aad0f1490d41d7f1203d57"}
Sep 09 20:25:58.719381 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:58.719337    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-7x7bm" event={"ID":"6e36dc0c-fb8a-4970-9d19-1a52f717bedc","Type":"ContainerStarted","Data":"8357cabd357f2f84d9a02aa9cc9ce841151bfd09658052e61b6c86209bb16d19"}
Sep 09 20:25:58.719525 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:58.719510    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-7x7bm"
Sep 09 20:25:58.740067 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:58.740019    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh" podStartSLOduration=2.944618626 podStartE2EDuration="6.740006703s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.795388077s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:25:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:25:58.738577527 +0000 UTC m=+1863.410596537" watchObservedRunningTime="2026-09-09 20:25:58.740006703 +0000 UTC m=+1863.412025699"
Sep 09 20:25:58.763516 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:58.763480    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-7x7bm" podStartSLOduration=2.170844366 podStartE2EDuration="5.763467333s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.592622967s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:25:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:25:58.762173829 +0000 UTC m=+1863.434192827" watchObservedRunningTime="2026-09-09 20:25:58.763467333 +0000 UTC m=+1863.435486329"
Sep 09 20:25:59.727661 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:59.727628    2523 generic.go:356] "Generic (PLEG): container finished" podID="45cc6423-62bf-4b6b-bb9b-2f3a1cde1639" containerID="7eb0ba2c179e7e2513036b01d280fbb83ac1cea098273877ed6a78bff62291b2" exitCode=0
Sep 09 20:25:59.728043 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:59.727698    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-6kf8r" event={"ID":"45cc6423-62bf-4b6b-bb9b-2f3a1cde1639","Type":"ContainerDied","Data":"7eb0ba2c179e7e2513036b01d280fbb83ac1cea098273877ed6a78bff62291b2"}
Sep 09 20:25:59.728137 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:59.728119    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-7x7bm"
Sep 09 20:25:59.728444 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:59.728409    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh" podUID="a359bd28-730b-4de8-91fd-48338328ac99" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 09 20:25:59.729498 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:25:59.729473    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-7x7bm" podUID="6e36dc0c-fb8a-4970-9d19-1a52f717bedc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 09 20:26:00.734052 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:26:00.734014    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-6kf8r" event={"ID":"45cc6423-62bf-4b6b-bb9b-2f3a1cde1639","Type":"ContainerStarted","Data":"73b2bd718c7f7f60658ce8584ceefa9b620d93813264ce753d9e7ad6a9cff4d0"}
Sep 09 20:26:00.734507 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:26:00.734076    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-6kf8r" event={"ID":"45cc6423-62bf-4b6b-bb9b-2f3a1cde1639","Type":"ContainerStarted","Data":"621d1b74910a40437a1247a066f700f7f7f166191260ca6ae028fb24e49bd9a0"}
Sep 09 20:26:00.734507 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:26:00.734238    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-7x7bm" podUID="6e36dc0c-fb8a-4970-9d19-1a52f717bedc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 09 20:26:00.734507 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:26:00.734387    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-6kf8r"
Sep 09 20:26:00.734507 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:26:00.734441    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-6kf8r"
Sep 09 20:26:00.735499 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:26:00.735476    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-6kf8r" podUID="45cc6423-62bf-4b6b-bb9b-2f3a1cde1639" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 09 20:26:00.754451 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:26:00.754404    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-6kf8r" podStartSLOduration=4.794703008 podStartE2EDuration="8.754390904s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.959687896s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:25:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:26:00.752976442 +0000 UTC m=+1865.424995441" watchObservedRunningTime="2026-09-09 20:26:00.754390904 +0000 UTC m=+1865.426409900"
Sep 09 20:26:01.737876 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:26:01.737835    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-6kf8r" podUID="45cc6423-62bf-4b6b-bb9b-2f3a1cde1639" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 09 20:26:04.733349 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:26:04.733321    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh"
Sep 09 20:26:04.733869 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:26:04.733840    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh" podUID="a359bd28-730b-4de8-91fd-48338328ac99" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 09 20:26:05.738740 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:26:05.738713    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-7x7bm"
Sep 09 20:26:05.739257 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:26:05.739226    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-7x7bm" podUID="6e36dc0c-fb8a-4970-9d19-1a52f717bedc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 09 20:26:06.742823 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:26:06.742791    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-6kf8r"
Sep 09 20:26:06.743285 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:26:06.743260    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-6kf8r" podUID="45cc6423-62bf-4b6b-bb9b-2f3a1cde1639" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 09 20:26:14.734180 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:26:14.734121    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh" podUID="a359bd28-730b-4de8-91fd-48338328ac99" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 09 20:26:15.739424 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:26:15.739387    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-7x7bm" podUID="6e36dc0c-fb8a-4970-9d19-1a52f717bedc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 09 20:26:16.743405 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:26:16.743365    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-6kf8r" podUID="45cc6423-62bf-4b6b-bb9b-2f3a1cde1639" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 09 20:26:24.734340 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:26:24.734295    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh" podUID="a359bd28-730b-4de8-91fd-48338328ac99" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 09 20:26:25.739765 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:26:25.739721    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-7x7bm" podUID="6e36dc0c-fb8a-4970-9d19-1a52f717bedc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 09 20:26:26.743489 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:26:26.743443    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-6kf8r" podUID="45cc6423-62bf-4b6b-bb9b-2f3a1cde1639" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 09 20:26:34.734217 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:26:34.734171    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh" podUID="a359bd28-730b-4de8-91fd-48338328ac99" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 09 20:26:35.739785 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:26:35.739750    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-7x7bm" podUID="6e36dc0c-fb8a-4970-9d19-1a52f717bedc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 09 20:26:36.743753 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:26:36.743714    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-6kf8r" podUID="45cc6423-62bf-4b6b-bb9b-2f3a1cde1639" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 09 20:26:44.733859 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:26:44.733811    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh" podUID="a359bd28-730b-4de8-91fd-48338328ac99" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 09 20:26:45.739967 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:26:45.739930    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-7x7bm" podUID="6e36dc0c-fb8a-4970-9d19-1a52f717bedc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 09 20:26:46.743665 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:26:46.743627    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-6kf8r" podUID="45cc6423-62bf-4b6b-bb9b-2f3a1cde1639" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 09 20:26:54.733899 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:26:54.733857    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh" podUID="a359bd28-730b-4de8-91fd-48338328ac99" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 09 20:26:55.739187 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:26:55.739133    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-7x7bm" podUID="6e36dc0c-fb8a-4970-9d19-1a52f717bedc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 09 20:26:56.744182 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:26:56.744106    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-6kf8r" podUID="45cc6423-62bf-4b6b-bb9b-2f3a1cde1639" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 09 20:27:04.734653 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:04.734617    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh" podUID="a359bd28-730b-4de8-91fd-48338328ac99" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 09 20:27:05.739589 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:05.739549    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-7x7bm" podUID="6e36dc0c-fb8a-4970-9d19-1a52f717bedc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 09 20:27:06.744301 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:06.744259    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-6kf8r" podUID="45cc6423-62bf-4b6b-bb9b-2f3a1cde1639" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 09 20:27:12.803990 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:12.803962    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-7x7bm"
Sep 09 20:27:14.734531 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:14.734494    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh"
Sep 09 20:27:16.744237 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:16.744211    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-6kf8r"
Sep 09 20:27:22.759177 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:22.759124    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh"]
Sep 09 20:27:22.759566 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:22.759411    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh" podUID="a359bd28-730b-4de8-91fd-48338328ac99" containerName="kserve-container" containerID="cri-o://0b8f96a930a64d5f36ef14ee4409a4440d882fb598c65b8d225f38752548a671" gracePeriod=30
Sep 09 20:27:22.759566 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:22.759445    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh" podUID="a359bd28-730b-4de8-91fd-48338328ac99" containerName="kube-rbac-proxy" containerID="cri-o://61a8503a036eb93d720186774c2ced1f69aba2915b98442c123fb4a3d4619984" gracePeriod=30
Sep 09 20:27:23.066355 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:23.066330    2523 generic.go:356] "Generic (PLEG): container finished" podID="a359bd28-730b-4de8-91fd-48338328ac99" containerID="61a8503a036eb93d720186774c2ced1f69aba2915b98442c123fb4a3d4619984" exitCode=2
Sep 09 20:27:23.066521 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:23.066403    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh" event={"ID":"a359bd28-730b-4de8-91fd-48338328ac99","Type":"ContainerDied","Data":"61a8503a036eb93d720186774c2ced1f69aba2915b98442c123fb4a3d4619984"}
Sep 09 20:27:23.079391 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:23.079361    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-7x7bm"]
Sep 09 20:27:23.079688 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:23.079588    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-7x7bm" podUID="6e36dc0c-fb8a-4970-9d19-1a52f717bedc" containerName="kserve-container" containerID="cri-o://8357cabd357f2f84d9a02aa9cc9ce841151bfd09658052e61b6c86209bb16d19" gracePeriod=30
Sep 09 20:27:23.079857 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:23.079613    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-7x7bm" podUID="6e36dc0c-fb8a-4970-9d19-1a52f717bedc" containerName="kube-rbac-proxy" containerID="cri-o://38f3ac85a9ad0693ce645ff5ffbddff733d3975962aad0f1490d41d7f1203d57" gracePeriod=30
Sep 09 20:27:23.082827 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:23.082800    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-6kf8r"]
Sep 09 20:27:23.083195 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:23.083104    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-6kf8r" podUID="45cc6423-62bf-4b6b-bb9b-2f3a1cde1639" containerName="kserve-container" containerID="cri-o://621d1b74910a40437a1247a066f700f7f7f166191260ca6ae028fb24e49bd9a0" gracePeriod=30
Sep 09 20:27:23.083322 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:23.083222    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-6kf8r" podUID="45cc6423-62bf-4b6b-bb9b-2f3a1cde1639" containerName="kube-rbac-proxy" containerID="cri-o://73b2bd718c7f7f60658ce8584ceefa9b620d93813264ce753d9e7ad6a9cff4d0" gracePeriod=30
Sep 09 20:27:24.072687 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:24.072658    2523 generic.go:356] "Generic (PLEG): container finished" podID="6e36dc0c-fb8a-4970-9d19-1a52f717bedc" containerID="38f3ac85a9ad0693ce645ff5ffbddff733d3975962aad0f1490d41d7f1203d57" exitCode=2
Sep 09 20:27:24.073079 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:24.072741    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-7x7bm" event={"ID":"6e36dc0c-fb8a-4970-9d19-1a52f717bedc","Type":"ContainerDied","Data":"38f3ac85a9ad0693ce645ff5ffbddff733d3975962aad0f1490d41d7f1203d57"}
Sep 09 20:27:24.074689 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:24.074669    2523 generic.go:356] "Generic (PLEG): container finished" podID="45cc6423-62bf-4b6b-bb9b-2f3a1cde1639" containerID="73b2bd718c7f7f60658ce8584ceefa9b620d93813264ce753d9e7ad6a9cff4d0" exitCode=2
Sep 09 20:27:24.074742 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:24.074698    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-6kf8r" event={"ID":"45cc6423-62bf-4b6b-bb9b-2f3a1cde1639","Type":"ContainerDied","Data":"73b2bd718c7f7f60658ce8584ceefa9b620d93813264ce753d9e7ad6a9cff4d0"}
Sep 09 20:27:24.729088 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:24.729050    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh" podUID="a359bd28-730b-4de8-91fd-48338328ac99" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Sep 09 20:27:24.734353 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:24.734329    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh" podUID="a359bd28-730b-4de8-91fd-48338328ac99" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 09 20:27:25.735435 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:25.735393    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-7x7bm" podUID="6e36dc0c-fb8a-4970-9d19-1a52f717bedc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Sep 09 20:27:26.738305 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:26.738262    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-6kf8r" podUID="45cc6423-62bf-4b6b-bb9b-2f3a1cde1639" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Sep 09 20:27:26.743223 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:26.743198    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-6kf8r" podUID="45cc6423-62bf-4b6b-bb9b-2f3a1cde1639" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 09 20:27:27.330162 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.330128    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh"
Sep 09 20:27:27.405764 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.405682    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fz69g\" (UniqueName: \"kubernetes.io/projected/a359bd28-730b-4de8-91fd-48338328ac99-kube-api-access-fz69g\") pod \"a359bd28-730b-4de8-91fd-48338328ac99\" (UID: \"a359bd28-730b-4de8-91fd-48338328ac99\") "
Sep 09 20:27:27.405924 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.405773    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a359bd28-730b-4de8-91fd-48338328ac99-kube-rbac-proxy-sar-config\") pod \"a359bd28-730b-4de8-91fd-48338328ac99\" (UID: \"a359bd28-730b-4de8-91fd-48338328ac99\") "
Sep 09 20:27:27.405924 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.405824    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a359bd28-730b-4de8-91fd-48338328ac99-cabundle-cert\") pod \"a359bd28-730b-4de8-91fd-48338328ac99\" (UID: \"a359bd28-730b-4de8-91fd-48338328ac99\") "
Sep 09 20:27:27.405924 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.405881    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a359bd28-730b-4de8-91fd-48338328ac99-kserve-provision-location\") pod \"a359bd28-730b-4de8-91fd-48338328ac99\" (UID: \"a359bd28-730b-4de8-91fd-48338328ac99\") "
Sep 09 20:27:27.405924 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.405913    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a359bd28-730b-4de8-91fd-48338328ac99-proxy-tls\") pod \"a359bd28-730b-4de8-91fd-48338328ac99\" (UID: \"a359bd28-730b-4de8-91fd-48338328ac99\") "
Sep 09 20:27:27.406382 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.406319    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a359bd28-730b-4de8-91fd-48338328ac99-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a359bd28-730b-4de8-91fd-48338328ac99" (UID: "a359bd28-730b-4de8-91fd-48338328ac99"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:27:27.406595 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.406548    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a359bd28-730b-4de8-91fd-48338328ac99-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a359bd28-730b-4de8-91fd-48338328ac99" (UID: "a359bd28-730b-4de8-91fd-48338328ac99"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:27:27.406813 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.406784    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a359bd28-730b-4de8-91fd-48338328ac99-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a359bd28-730b-4de8-91fd-48338328ac99" (UID: "a359bd28-730b-4de8-91fd-48338328ac99"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:27:27.408109 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.408078    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a359bd28-730b-4de8-91fd-48338328ac99-kube-api-access-fz69g" (OuterVolumeSpecName: "kube-api-access-fz69g") pod "a359bd28-730b-4de8-91fd-48338328ac99" (UID: "a359bd28-730b-4de8-91fd-48338328ac99"). InnerVolumeSpecName "kube-api-access-fz69g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:27:27.408501 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.408473    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a359bd28-730b-4de8-91fd-48338328ac99-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a359bd28-730b-4de8-91fd-48338328ac99" (UID: "a359bd28-730b-4de8-91fd-48338328ac99"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:27:27.427384 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.427367    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-7x7bm"
Sep 09 20:27:27.507263 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.507238    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxm26\" (UniqueName: \"kubernetes.io/projected/6e36dc0c-fb8a-4970-9d19-1a52f717bedc-kube-api-access-nxm26\") pod \"6e36dc0c-fb8a-4970-9d19-1a52f717bedc\" (UID: \"6e36dc0c-fb8a-4970-9d19-1a52f717bedc\") "
Sep 09 20:27:27.507376 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.507283    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e36dc0c-fb8a-4970-9d19-1a52f717bedc-kserve-provision-location\") pod \"6e36dc0c-fb8a-4970-9d19-1a52f717bedc\" (UID: \"6e36dc0c-fb8a-4970-9d19-1a52f717bedc\") "
Sep 09 20:27:27.507376 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.507308    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e36dc0c-fb8a-4970-9d19-1a52f717bedc-cabundle-cert\") pod \"6e36dc0c-fb8a-4970-9d19-1a52f717bedc\" (UID: \"6e36dc0c-fb8a-4970-9d19-1a52f717bedc\") "
Sep 09 20:27:27.507376 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.507335    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e36dc0c-fb8a-4970-9d19-1a52f717bedc-kube-rbac-proxy-sar-config\") pod \"6e36dc0c-fb8a-4970-9d19-1a52f717bedc\" (UID: \"6e36dc0c-fb8a-4970-9d19-1a52f717bedc\") "
Sep 09 20:27:27.507500 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.507376    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e36dc0c-fb8a-4970-9d19-1a52f717bedc-proxy-tls\") pod \"6e36dc0c-fb8a-4970-9d19-1a52f717bedc\" (UID: \"6e36dc0c-fb8a-4970-9d19-1a52f717bedc\") "
Sep 09 20:27:27.507621 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.507598    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e36dc0c-fb8a-4970-9d19-1a52f717bedc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6e36dc0c-fb8a-4970-9d19-1a52f717bedc" (UID: "6e36dc0c-fb8a-4970-9d19-1a52f717bedc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:27:27.507690 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.507611    2523 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a359bd28-730b-4de8-91fd-48338328ac99-kserve-provision-location\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:27:27.507690 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.507637    2523 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a359bd28-730b-4de8-91fd-48338328ac99-proxy-tls\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:27:27.507690 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.507648    2523 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz69g\" (UniqueName: \"kubernetes.io/projected/a359bd28-730b-4de8-91fd-48338328ac99-kube-api-access-fz69g\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:27:27.507690 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.507658    2523 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a359bd28-730b-4de8-91fd-48338328ac99-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:27:27.507690 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.507668    2523 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a359bd28-730b-4de8-91fd-48338328ac99-cabundle-cert\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:27:27.507892 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.507710    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e36dc0c-fb8a-4970-9d19-1a52f717bedc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6e36dc0c-fb8a-4970-9d19-1a52f717bedc" (UID: "6e36dc0c-fb8a-4970-9d19-1a52f717bedc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:27:27.507892 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.507715    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e36dc0c-fb8a-4970-9d19-1a52f717bedc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6e36dc0c-fb8a-4970-9d19-1a52f717bedc" (UID: "6e36dc0c-fb8a-4970-9d19-1a52f717bedc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:27:27.509370 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.509352    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e36dc0c-fb8a-4970-9d19-1a52f717bedc-kube-api-access-nxm26" (OuterVolumeSpecName: "kube-api-access-nxm26") pod "6e36dc0c-fb8a-4970-9d19-1a52f717bedc" (UID: "6e36dc0c-fb8a-4970-9d19-1a52f717bedc"). InnerVolumeSpecName "kube-api-access-nxm26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:27:27.509442 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.509413    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e36dc0c-fb8a-4970-9d19-1a52f717bedc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6e36dc0c-fb8a-4970-9d19-1a52f717bedc" (UID: "6e36dc0c-fb8a-4970-9d19-1a52f717bedc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:27:27.608524 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.608499    2523 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e36dc0c-fb8a-4970-9d19-1a52f717bedc-proxy-tls\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:27:27.608524 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.608521    2523 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxm26\" (UniqueName: \"kubernetes.io/projected/6e36dc0c-fb8a-4970-9d19-1a52f717bedc-kube-api-access-nxm26\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:27:27.608654 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.608530    2523 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e36dc0c-fb8a-4970-9d19-1a52f717bedc-kserve-provision-location\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:27:27.608654 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.608539    2523 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e36dc0c-fb8a-4970-9d19-1a52f717bedc-cabundle-cert\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:27:27.608654 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.608547    2523 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e36dc0c-fb8a-4970-9d19-1a52f717bedc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:27:27.822193 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.822171    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-6kf8r"
Sep 09 20:27:27.910908 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.910840    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45cc6423-62bf-4b6b-bb9b-2f3a1cde1639-cabundle-cert\") pod \"45cc6423-62bf-4b6b-bb9b-2f3a1cde1639\" (UID: \"45cc6423-62bf-4b6b-bb9b-2f3a1cde1639\") "
Sep 09 20:27:27.910908 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.910875    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vsnsv\" (UniqueName: \"kubernetes.io/projected/45cc6423-62bf-4b6b-bb9b-2f3a1cde1639-kube-api-access-vsnsv\") pod \"45cc6423-62bf-4b6b-bb9b-2f3a1cde1639\" (UID: \"45cc6423-62bf-4b6b-bb9b-2f3a1cde1639\") "
Sep 09 20:27:27.910908 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.910900    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45cc6423-62bf-4b6b-bb9b-2f3a1cde1639-kube-rbac-proxy-sar-config\") pod \"45cc6423-62bf-4b6b-bb9b-2f3a1cde1639\" (UID: \"45cc6423-62bf-4b6b-bb9b-2f3a1cde1639\") "
Sep 09 20:27:27.911120 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.911096    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45cc6423-62bf-4b6b-bb9b-2f3a1cde1639-proxy-tls\") pod \"45cc6423-62bf-4b6b-bb9b-2f3a1cde1639\" (UID: \"45cc6423-62bf-4b6b-bb9b-2f3a1cde1639\") "
Sep 09 20:27:27.911198 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.911139    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45cc6423-62bf-4b6b-bb9b-2f3a1cde1639-kserve-provision-location\") pod \"45cc6423-62bf-4b6b-bb9b-2f3a1cde1639\" (UID: \"45cc6423-62bf-4b6b-bb9b-2f3a1cde1639\") "
Sep 09 20:27:27.911264 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.911243    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45cc6423-62bf-4b6b-bb9b-2f3a1cde1639-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "45cc6423-62bf-4b6b-bb9b-2f3a1cde1639" (UID: "45cc6423-62bf-4b6b-bb9b-2f3a1cde1639"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:27:27.911326 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.911271    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45cc6423-62bf-4b6b-bb9b-2f3a1cde1639-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "45cc6423-62bf-4b6b-bb9b-2f3a1cde1639" (UID: "45cc6423-62bf-4b6b-bb9b-2f3a1cde1639"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:27:27.911439 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.911417    2523 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45cc6423-62bf-4b6b-bb9b-2f3a1cde1639-cabundle-cert\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:27:27.911535 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.911441    2523 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45cc6423-62bf-4b6b-bb9b-2f3a1cde1639-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:27:27.911535 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.911507    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45cc6423-62bf-4b6b-bb9b-2f3a1cde1639-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "45cc6423-62bf-4b6b-bb9b-2f3a1cde1639" (UID: "45cc6423-62bf-4b6b-bb9b-2f3a1cde1639"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:27:27.913093 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.913072    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45cc6423-62bf-4b6b-bb9b-2f3a1cde1639-kube-api-access-vsnsv" (OuterVolumeSpecName: "kube-api-access-vsnsv") pod "45cc6423-62bf-4b6b-bb9b-2f3a1cde1639" (UID: "45cc6423-62bf-4b6b-bb9b-2f3a1cde1639"). InnerVolumeSpecName "kube-api-access-vsnsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:27:27.913229 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.913213    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45cc6423-62bf-4b6b-bb9b-2f3a1cde1639-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "45cc6423-62bf-4b6b-bb9b-2f3a1cde1639" (UID: "45cc6423-62bf-4b6b-bb9b-2f3a1cde1639"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:27:27.964553 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.964533    2523 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-j2mt4"]
Sep 09 20:27:27.964809 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.964798    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a359bd28-730b-4de8-91fd-48338328ac99" containerName="kserve-container"
Sep 09 20:27:27.964860 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.964811    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="a359bd28-730b-4de8-91fd-48338328ac99" containerName="kserve-container"
Sep 09 20:27:27.964860 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.964824    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e36dc0c-fb8a-4970-9d19-1a52f717bedc" containerName="storage-initializer"
Sep 09 20:27:27.964860 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.964830    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e36dc0c-fb8a-4970-9d19-1a52f717bedc" containerName="storage-initializer"
Sep 09 20:27:27.964860 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.964837    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45cc6423-62bf-4b6b-bb9b-2f3a1cde1639" containerName="kube-rbac-proxy"
Sep 09 20:27:27.964860 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.964842    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="45cc6423-62bf-4b6b-bb9b-2f3a1cde1639" containerName="kube-rbac-proxy"
Sep 09 20:27:27.964860 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.964851    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a359bd28-730b-4de8-91fd-48338328ac99" containerName="storage-initializer"
Sep 09 20:27:27.964860 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.964856    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="a359bd28-730b-4de8-91fd-48338328ac99" containerName="storage-initializer"
Sep 09 20:27:27.965077 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.964864    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45cc6423-62bf-4b6b-bb9b-2f3a1cde1639" containerName="storage-initializer"
Sep 09 20:27:27.965077 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.964869    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="45cc6423-62bf-4b6b-bb9b-2f3a1cde1639" containerName="storage-initializer"
Sep 09 20:27:27.965077 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.964878    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a359bd28-730b-4de8-91fd-48338328ac99" containerName="kube-rbac-proxy"
Sep 09 20:27:27.965077 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.964883    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="a359bd28-730b-4de8-91fd-48338328ac99" containerName="kube-rbac-proxy"
Sep 09 20:27:27.965077 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.964890    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e36dc0c-fb8a-4970-9d19-1a52f717bedc" containerName="kserve-container"
Sep 09 20:27:27.965077 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.964895    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e36dc0c-fb8a-4970-9d19-1a52f717bedc" containerName="kserve-container"
Sep 09 20:27:27.965077 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.964901    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e36dc0c-fb8a-4970-9d19-1a52f717bedc" containerName="kube-rbac-proxy"
Sep 09 20:27:27.965077 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.964906    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e36dc0c-fb8a-4970-9d19-1a52f717bedc" containerName="kube-rbac-proxy"
Sep 09 20:27:27.965077 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.964911    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45cc6423-62bf-4b6b-bb9b-2f3a1cde1639" containerName="kserve-container"
Sep 09 20:27:27.965077 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.964916    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="45cc6423-62bf-4b6b-bb9b-2f3a1cde1639" containerName="kserve-container"
Sep 09 20:27:27.965077 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.964958    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e36dc0c-fb8a-4970-9d19-1a52f717bedc" containerName="kube-rbac-proxy"
Sep 09 20:27:27.965077 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.964965    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e36dc0c-fb8a-4970-9d19-1a52f717bedc" containerName="kserve-container"
Sep 09 20:27:27.965077 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.964971    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="a359bd28-730b-4de8-91fd-48338328ac99" containerName="kserve-container"
Sep 09 20:27:27.965077 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.964977    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="a359bd28-730b-4de8-91fd-48338328ac99" containerName="kube-rbac-proxy"
Sep 09 20:27:27.965077 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.964983    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="45cc6423-62bf-4b6b-bb9b-2f3a1cde1639" containerName="kserve-container"
Sep 09 20:27:27.965077 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.964988    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="45cc6423-62bf-4b6b-bb9b-2f3a1cde1639" containerName="kube-rbac-proxy"
Sep 09 20:27:27.968204 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.968187    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-j2mt4"
Sep 09 20:27:27.970541 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.970511    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-predictor-serving-cert\""
Sep 09 20:27:27.970633 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.970586    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
Sep 09 20:27:27.978229 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:27.978205    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-j2mt4"]
Sep 09 20:27:28.012077 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.012051    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm946\" (UniqueName: \"kubernetes.io/projected/dab4510d-a522-4a64-b70d-518cd56be114-kube-api-access-nm946\") pod \"isvc-tensorflow-predictor-547dfc5669-j2mt4\" (UID: \"dab4510d-a522-4a64-b70d-518cd56be114\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-j2mt4"
Sep 09 20:27:28.012202 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.012085    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dab4510d-a522-4a64-b70d-518cd56be114-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-j2mt4\" (UID: \"dab4510d-a522-4a64-b70d-518cd56be114\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-j2mt4"
Sep 09 20:27:28.012202 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.012105    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dab4510d-a522-4a64-b70d-518cd56be114-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-j2mt4\" (UID: \"dab4510d-a522-4a64-b70d-518cd56be114\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-j2mt4"
Sep 09 20:27:28.012202 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.012192    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dab4510d-a522-4a64-b70d-518cd56be114-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-j2mt4\" (UID: \"dab4510d-a522-4a64-b70d-518cd56be114\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-j2mt4"
Sep 09 20:27:28.012374 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.012258    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dab4510d-a522-4a64-b70d-518cd56be114-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-j2mt4\" (UID: \"dab4510d-a522-4a64-b70d-518cd56be114\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-j2mt4"
Sep 09 20:27:28.012374 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.012317    2523 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45cc6423-62bf-4b6b-bb9b-2f3a1cde1639-proxy-tls\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:27:28.012374 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.012333    2523 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45cc6423-62bf-4b6b-bb9b-2f3a1cde1639-kserve-provision-location\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:27:28.012374 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.012348    2523 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsnsv\" (UniqueName: \"kubernetes.io/projected/45cc6423-62bf-4b6b-bb9b-2f3a1cde1639-kube-api-access-vsnsv\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:27:28.091773 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.091747    2523 generic.go:356] "Generic (PLEG): container finished" podID="6e36dc0c-fb8a-4970-9d19-1a52f717bedc" containerID="8357cabd357f2f84d9a02aa9cc9ce841151bfd09658052e61b6c86209bb16d19" exitCode=0
Sep 09 20:27:28.091873 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.091804    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-7x7bm"
Sep 09 20:27:28.091873 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.091834    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-7x7bm" event={"ID":"6e36dc0c-fb8a-4970-9d19-1a52f717bedc","Type":"ContainerDied","Data":"8357cabd357f2f84d9a02aa9cc9ce841151bfd09658052e61b6c86209bb16d19"}
Sep 09 20:27:28.091987 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.091904    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-7x7bm" event={"ID":"6e36dc0c-fb8a-4970-9d19-1a52f717bedc","Type":"ContainerDied","Data":"69ce4d6f589f3120f3641e272bdf58f836e9b30b0cb5d7c2fc7ae0f404b80b74"}
Sep 09 20:27:28.091987 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.091923    2523 scope.go:117] "RemoveContainer" containerID="38f3ac85a9ad0693ce645ff5ffbddff733d3975962aad0f1490d41d7f1203d57"
Sep 09 20:27:28.093951 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.093930    2523 generic.go:356] "Generic (PLEG): container finished" podID="45cc6423-62bf-4b6b-bb9b-2f3a1cde1639" containerID="621d1b74910a40437a1247a066f700f7f7f166191260ca6ae028fb24e49bd9a0" exitCode=0
Sep 09 20:27:28.094063 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.093958    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-6kf8r" event={"ID":"45cc6423-62bf-4b6b-bb9b-2f3a1cde1639","Type":"ContainerDied","Data":"621d1b74910a40437a1247a066f700f7f7f166191260ca6ae028fb24e49bd9a0"}
Sep 09 20:27:28.094063 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.094001    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-6kf8r"
Sep 09 20:27:28.094063 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.094015    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-6kf8r" event={"ID":"45cc6423-62bf-4b6b-bb9b-2f3a1cde1639","Type":"ContainerDied","Data":"348d8054d51612be3b44c7d8f0d5a4a2fc9b589b85215e53ed0ca231aa2dcffd"}
Sep 09 20:27:28.096276 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.096256    2523 generic.go:356] "Generic (PLEG): container finished" podID="a359bd28-730b-4de8-91fd-48338328ac99" containerID="0b8f96a930a64d5f36ef14ee4409a4440d882fb598c65b8d225f38752548a671" exitCode=0
Sep 09 20:27:28.096374 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.096317    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh" event={"ID":"a359bd28-730b-4de8-91fd-48338328ac99","Type":"ContainerDied","Data":"0b8f96a930a64d5f36ef14ee4409a4440d882fb598c65b8d225f38752548a671"}
Sep 09 20:27:28.096374 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.096343    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh"
Sep 09 20:27:28.096374 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.096364    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh" event={"ID":"a359bd28-730b-4de8-91fd-48338328ac99","Type":"ContainerDied","Data":"118fcd64652340c9aa4fa5021ea07e4c246b0f343f1121f82124f028d9b3c53c"}
Sep 09 20:27:28.100891 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.100863    2523 scope.go:117] "RemoveContainer" containerID="8357cabd357f2f84d9a02aa9cc9ce841151bfd09658052e61b6c86209bb16d19"
Sep 09 20:27:28.108081 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.108067    2523 scope.go:117] "RemoveContainer" containerID="d5cb6534e9300ef99f4107776fdaa6603af3a90082364edaa478d4a5b74f5353"
Sep 09 20:27:28.113199 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.113180    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dab4510d-a522-4a64-b70d-518cd56be114-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-j2mt4\" (UID: \"dab4510d-a522-4a64-b70d-518cd56be114\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-j2mt4"
Sep 09 20:27:28.113306 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.113216    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dab4510d-a522-4a64-b70d-518cd56be114-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-j2mt4\" (UID: \"dab4510d-a522-4a64-b70d-518cd56be114\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-j2mt4"
Sep 09 20:27:28.113378 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:27:28.113335    2523 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-tensorflow-predictor-serving-cert: secret "isvc-tensorflow-predictor-serving-cert" not found
Sep 09 20:27:28.113433 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.113335    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm946\" (UniqueName: \"kubernetes.io/projected/dab4510d-a522-4a64-b70d-518cd56be114-kube-api-access-nm946\") pod \"isvc-tensorflow-predictor-547dfc5669-j2mt4\" (UID: \"dab4510d-a522-4a64-b70d-518cd56be114\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-j2mt4"
Sep 09 20:27:28.113433 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:27:28.113404    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dab4510d-a522-4a64-b70d-518cd56be114-proxy-tls podName:dab4510d-a522-4a64-b70d-518cd56be114 nodeName:}" failed. No retries permitted until 2026-09-09 20:27:28.613384384 +0000 UTC m=+1953.285403372 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/dab4510d-a522-4a64-b70d-518cd56be114-proxy-tls") pod "isvc-tensorflow-predictor-547dfc5669-j2mt4" (UID: "dab4510d-a522-4a64-b70d-518cd56be114") : secret "isvc-tensorflow-predictor-serving-cert" not found
Sep 09 20:27:28.113553 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.113447    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dab4510d-a522-4a64-b70d-518cd56be114-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-j2mt4\" (UID: \"dab4510d-a522-4a64-b70d-518cd56be114\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-j2mt4"
Sep 09 20:27:28.113553 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.113481    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dab4510d-a522-4a64-b70d-518cd56be114-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-j2mt4\" (UID: \"dab4510d-a522-4a64-b70d-518cd56be114\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-j2mt4"
Sep 09 20:27:28.113762 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.113744    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dab4510d-a522-4a64-b70d-518cd56be114-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-j2mt4\" (UID: \"dab4510d-a522-4a64-b70d-518cd56be114\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-j2mt4"
Sep 09 20:27:28.113917 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.113897    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dab4510d-a522-4a64-b70d-518cd56be114-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-j2mt4\" (UID: \"dab4510d-a522-4a64-b70d-518cd56be114\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-j2mt4"
Sep 09 20:27:28.114091 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.114073    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dab4510d-a522-4a64-b70d-518cd56be114-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-j2mt4\" (UID: \"dab4510d-a522-4a64-b70d-518cd56be114\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-j2mt4"
Sep 09 20:27:28.115660 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.115633    2523 scope.go:117] "RemoveContainer" containerID="38f3ac85a9ad0693ce645ff5ffbddff733d3975962aad0f1490d41d7f1203d57"
Sep 09 20:27:28.116273 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:27:28.116126    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38f3ac85a9ad0693ce645ff5ffbddff733d3975962aad0f1490d41d7f1203d57\": container with ID starting with 38f3ac85a9ad0693ce645ff5ffbddff733d3975962aad0f1490d41d7f1203d57 not found: ID does not exist" containerID="38f3ac85a9ad0693ce645ff5ffbddff733d3975962aad0f1490d41d7f1203d57"
Sep 09 20:27:28.116362 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.116280    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38f3ac85a9ad0693ce645ff5ffbddff733d3975962aad0f1490d41d7f1203d57"} err="failed to get container status \"38f3ac85a9ad0693ce645ff5ffbddff733d3975962aad0f1490d41d7f1203d57\": rpc error: code = NotFound desc = could not find container \"38f3ac85a9ad0693ce645ff5ffbddff733d3975962aad0f1490d41d7f1203d57\": container with ID starting with 38f3ac85a9ad0693ce645ff5ffbddff733d3975962aad0f1490d41d7f1203d57 not found: ID does not exist"
Sep 09 20:27:28.116362 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.116306    2523 scope.go:117] "RemoveContainer" containerID="8357cabd357f2f84d9a02aa9cc9ce841151bfd09658052e61b6c86209bb16d19"
Sep 09 20:27:28.116648 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:27:28.116621    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8357cabd357f2f84d9a02aa9cc9ce841151bfd09658052e61b6c86209bb16d19\": container with ID starting with 8357cabd357f2f84d9a02aa9cc9ce841151bfd09658052e61b6c86209bb16d19 not found: ID does not exist" containerID="8357cabd357f2f84d9a02aa9cc9ce841151bfd09658052e61b6c86209bb16d19"
Sep 09 20:27:28.116730 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.116653    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8357cabd357f2f84d9a02aa9cc9ce841151bfd09658052e61b6c86209bb16d19"} err="failed to get container status \"8357cabd357f2f84d9a02aa9cc9ce841151bfd09658052e61b6c86209bb16d19\": rpc error: code = NotFound desc = could not find container \"8357cabd357f2f84d9a02aa9cc9ce841151bfd09658052e61b6c86209bb16d19\": container with ID starting with 8357cabd357f2f84d9a02aa9cc9ce841151bfd09658052e61b6c86209bb16d19 not found: ID does not exist"
Sep 09 20:27:28.116730 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.116704    2523 scope.go:117] "RemoveContainer" containerID="d5cb6534e9300ef99f4107776fdaa6603af3a90082364edaa478d4a5b74f5353"
Sep 09 20:27:28.117081 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:27:28.117058    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5cb6534e9300ef99f4107776fdaa6603af3a90082364edaa478d4a5b74f5353\": container with ID starting with d5cb6534e9300ef99f4107776fdaa6603af3a90082364edaa478d4a5b74f5353 not found: ID does not exist" containerID="d5cb6534e9300ef99f4107776fdaa6603af3a90082364edaa478d4a5b74f5353"
Sep 09 20:27:28.117283 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.117255    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5cb6534e9300ef99f4107776fdaa6603af3a90082364edaa478d4a5b74f5353"} err="failed to get container status \"d5cb6534e9300ef99f4107776fdaa6603af3a90082364edaa478d4a5b74f5353\": rpc error: code = NotFound desc = could not find container \"d5cb6534e9300ef99f4107776fdaa6603af3a90082364edaa478d4a5b74f5353\": container with ID starting with d5cb6534e9300ef99f4107776fdaa6603af3a90082364edaa478d4a5b74f5353 not found: ID does not exist"
Sep 09 20:27:28.117446 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.117431    2523 scope.go:117] "RemoveContainer" containerID="73b2bd718c7f7f60658ce8584ceefa9b620d93813264ce753d9e7ad6a9cff4d0"
Sep 09 20:27:28.118747 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.118602    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-7x7bm"]
Sep 09 20:27:28.121674 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.121653    2523 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-7x7bm"]
Sep 09 20:27:28.122701 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.122683    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm946\" (UniqueName: \"kubernetes.io/projected/dab4510d-a522-4a64-b70d-518cd56be114-kube-api-access-nm946\") pod \"isvc-tensorflow-predictor-547dfc5669-j2mt4\" (UID: \"dab4510d-a522-4a64-b70d-518cd56be114\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-j2mt4"
Sep 09 20:27:28.127045 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.126950    2523 scope.go:117] "RemoveContainer" containerID="621d1b74910a40437a1247a066f700f7f7f166191260ca6ae028fb24e49bd9a0"
Sep 09 20:27:28.132838 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.132350    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh"]
Sep 09 20:27:28.135762 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.135746    2523 scope.go:117] "RemoveContainer" containerID="7eb0ba2c179e7e2513036b01d280fbb83ac1cea098273877ed6a78bff62291b2"
Sep 09 20:27:28.139524 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.139502    2523 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6dcddf96b6-lwhlh"]
Sep 09 20:27:28.145131 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.145101    2523 scope.go:117] "RemoveContainer" containerID="73b2bd718c7f7f60658ce8584ceefa9b620d93813264ce753d9e7ad6a9cff4d0"
Sep 09 20:27:28.145448 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:27:28.145427    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73b2bd718c7f7f60658ce8584ceefa9b620d93813264ce753d9e7ad6a9cff4d0\": container with ID starting with 73b2bd718c7f7f60658ce8584ceefa9b620d93813264ce753d9e7ad6a9cff4d0 not found: ID does not exist" containerID="73b2bd718c7f7f60658ce8584ceefa9b620d93813264ce753d9e7ad6a9cff4d0"
Sep 09 20:27:28.145545 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.145454    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73b2bd718c7f7f60658ce8584ceefa9b620d93813264ce753d9e7ad6a9cff4d0"} err="failed to get container status \"73b2bd718c7f7f60658ce8584ceefa9b620d93813264ce753d9e7ad6a9cff4d0\": rpc error: code = NotFound desc = could not find container \"73b2bd718c7f7f60658ce8584ceefa9b620d93813264ce753d9e7ad6a9cff4d0\": container with ID starting with 73b2bd718c7f7f60658ce8584ceefa9b620d93813264ce753d9e7ad6a9cff4d0 not found: ID does not exist"
Sep 09 20:27:28.145545 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.145469    2523 scope.go:117] "RemoveContainer" containerID="621d1b74910a40437a1247a066f700f7f7f166191260ca6ae028fb24e49bd9a0"
Sep 09 20:27:28.145713 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:27:28.145696    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"621d1b74910a40437a1247a066f700f7f7f166191260ca6ae028fb24e49bd9a0\": container with ID starting with 621d1b74910a40437a1247a066f700f7f7f166191260ca6ae028fb24e49bd9a0 not found: ID does not exist" containerID="621d1b74910a40437a1247a066f700f7f7f166191260ca6ae028fb24e49bd9a0"
Sep 09 20:27:28.145774 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.145716    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"621d1b74910a40437a1247a066f700f7f7f166191260ca6ae028fb24e49bd9a0"} err="failed to get container status \"621d1b74910a40437a1247a066f700f7f7f166191260ca6ae028fb24e49bd9a0\": rpc error: code = NotFound desc = could not find container \"621d1b74910a40437a1247a066f700f7f7f166191260ca6ae028fb24e49bd9a0\": container with ID starting with 621d1b74910a40437a1247a066f700f7f7f166191260ca6ae028fb24e49bd9a0 not found: ID does not exist"
Sep 09 20:27:28.145774 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.145729    2523 scope.go:117] "RemoveContainer" containerID="7eb0ba2c179e7e2513036b01d280fbb83ac1cea098273877ed6a78bff62291b2"
Sep 09 20:27:28.145970 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:27:28.145954    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7eb0ba2c179e7e2513036b01d280fbb83ac1cea098273877ed6a78bff62291b2\": container with ID starting with 7eb0ba2c179e7e2513036b01d280fbb83ac1cea098273877ed6a78bff62291b2 not found: ID does not exist" containerID="7eb0ba2c179e7e2513036b01d280fbb83ac1cea098273877ed6a78bff62291b2"
Sep 09 20:27:28.146041 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.145975    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7eb0ba2c179e7e2513036b01d280fbb83ac1cea098273877ed6a78bff62291b2"} err="failed to get container status \"7eb0ba2c179e7e2513036b01d280fbb83ac1cea098273877ed6a78bff62291b2\": rpc error: code = NotFound desc = could not find container \"7eb0ba2c179e7e2513036b01d280fbb83ac1cea098273877ed6a78bff62291b2\": container with ID starting with 7eb0ba2c179e7e2513036b01d280fbb83ac1cea098273877ed6a78bff62291b2 not found: ID does not exist"
Sep 09 20:27:28.146041 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.145990    2523 scope.go:117] "RemoveContainer" containerID="61a8503a036eb93d720186774c2ced1f69aba2915b98442c123fb4a3d4619984"
Sep 09 20:27:28.154375 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.154353    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-6kf8r"]
Sep 09 20:27:28.157988 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.157969    2523 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-6kf8r"]
Sep 09 20:27:28.163326 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.163277    2523 scope.go:117] "RemoveContainer" containerID="0b8f96a930a64d5f36ef14ee4409a4440d882fb598c65b8d225f38752548a671"
Sep 09 20:27:28.170357 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.170341    2523 scope.go:117] "RemoveContainer" containerID="50339aad15389bf5b3509c49373dd38269bc342cbd5fa052eecb4718a5cd0b6c"
Sep 09 20:27:28.177862 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.177672    2523 scope.go:117] "RemoveContainer" containerID="61a8503a036eb93d720186774c2ced1f69aba2915b98442c123fb4a3d4619984"
Sep 09 20:27:28.177980 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:27:28.177959    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61a8503a036eb93d720186774c2ced1f69aba2915b98442c123fb4a3d4619984\": container with ID starting with 61a8503a036eb93d720186774c2ced1f69aba2915b98442c123fb4a3d4619984 not found: ID does not exist" containerID="61a8503a036eb93d720186774c2ced1f69aba2915b98442c123fb4a3d4619984"
Sep 09 20:27:28.178045 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.177985    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61a8503a036eb93d720186774c2ced1f69aba2915b98442c123fb4a3d4619984"} err="failed to get container status \"61a8503a036eb93d720186774c2ced1f69aba2915b98442c123fb4a3d4619984\": rpc error: code = NotFound desc = could not find container \"61a8503a036eb93d720186774c2ced1f69aba2915b98442c123fb4a3d4619984\": container with ID starting with 61a8503a036eb93d720186774c2ced1f69aba2915b98442c123fb4a3d4619984 not found: ID does not exist"
Sep 09 20:27:28.178045 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.178009    2523 scope.go:117] "RemoveContainer" containerID="0b8f96a930a64d5f36ef14ee4409a4440d882fb598c65b8d225f38752548a671"
Sep 09 20:27:28.178338 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:27:28.178316    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b8f96a930a64d5f36ef14ee4409a4440d882fb598c65b8d225f38752548a671\": container with ID starting with 0b8f96a930a64d5f36ef14ee4409a4440d882fb598c65b8d225f38752548a671 not found: ID does not exist" containerID="0b8f96a930a64d5f36ef14ee4409a4440d882fb598c65b8d225f38752548a671"
Sep 09 20:27:28.178426 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.178343    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b8f96a930a64d5f36ef14ee4409a4440d882fb598c65b8d225f38752548a671"} err="failed to get container status \"0b8f96a930a64d5f36ef14ee4409a4440d882fb598c65b8d225f38752548a671\": rpc error: code = NotFound desc = could not find container \"0b8f96a930a64d5f36ef14ee4409a4440d882fb598c65b8d225f38752548a671\": container with ID starting with 0b8f96a930a64d5f36ef14ee4409a4440d882fb598c65b8d225f38752548a671 not found: ID does not exist"
Sep 09 20:27:28.178426 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.178357    2523 scope.go:117] "RemoveContainer" containerID="50339aad15389bf5b3509c49373dd38269bc342cbd5fa052eecb4718a5cd0b6c"
Sep 09 20:27:28.178646 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:27:28.178630    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50339aad15389bf5b3509c49373dd38269bc342cbd5fa052eecb4718a5cd0b6c\": container with ID starting with 50339aad15389bf5b3509c49373dd38269bc342cbd5fa052eecb4718a5cd0b6c not found: ID does not exist" containerID="50339aad15389bf5b3509c49373dd38269bc342cbd5fa052eecb4718a5cd0b6c"
Sep 09 20:27:28.178703 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.178662    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50339aad15389bf5b3509c49373dd38269bc342cbd5fa052eecb4718a5cd0b6c"} err="failed to get container status \"50339aad15389bf5b3509c49373dd38269bc342cbd5fa052eecb4718a5cd0b6c\": rpc error: code = NotFound desc = could not find container \"50339aad15389bf5b3509c49373dd38269bc342cbd5fa052eecb4718a5cd0b6c\": container with ID starting with 50339aad15389bf5b3509c49373dd38269bc342cbd5fa052eecb4718a5cd0b6c not found: ID does not exist"
Sep 09 20:27:28.258076 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.258043    2523 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-wj88p"]
Sep 09 20:27:28.263235 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.263214    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-wj88p"
Sep 09 20:27:28.265685 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.265648    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 09 20:27:28.265787 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.265760    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 09 20:27:28.273257 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.273235    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-wj88p"]
Sep 09 20:27:28.314590 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.314563    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0051a6a-812d-406c-8292-97ff6440ee0e-proxy-tls\") pod \"isvc-canary-promote-predictor-cb75746f8-wj88p\" (UID: \"c0051a6a-812d-406c-8292-97ff6440ee0e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-wj88p"
Sep 09 20:27:28.314687 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.314603    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0051a6a-812d-406c-8292-97ff6440ee0e-cabundle-cert\") pod \"isvc-canary-promote-predictor-cb75746f8-wj88p\" (UID: \"c0051a6a-812d-406c-8292-97ff6440ee0e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-wj88p"
Sep 09 20:27:28.314687 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.314637    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0051a6a-812d-406c-8292-97ff6440ee0e-kserve-provision-location\") pod \"isvc-canary-promote-predictor-cb75746f8-wj88p\" (UID: \"c0051a6a-812d-406c-8292-97ff6440ee0e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-wj88p"
Sep 09 20:27:28.314687 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.314660    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shdt4\" (UniqueName: \"kubernetes.io/projected/c0051a6a-812d-406c-8292-97ff6440ee0e-kube-api-access-shdt4\") pod \"isvc-canary-promote-predictor-cb75746f8-wj88p\" (UID: \"c0051a6a-812d-406c-8292-97ff6440ee0e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-wj88p"
Sep 09 20:27:28.314787 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.314713    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0051a6a-812d-406c-8292-97ff6440ee0e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-cb75746f8-wj88p\" (UID: \"c0051a6a-812d-406c-8292-97ff6440ee0e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-wj88p"
Sep 09 20:27:28.415402 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.415349    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0051a6a-812d-406c-8292-97ff6440ee0e-proxy-tls\") pod \"isvc-canary-promote-predictor-cb75746f8-wj88p\" (UID: \"c0051a6a-812d-406c-8292-97ff6440ee0e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-wj88p"
Sep 09 20:27:28.415402 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.415380    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0051a6a-812d-406c-8292-97ff6440ee0e-cabundle-cert\") pod \"isvc-canary-promote-predictor-cb75746f8-wj88p\" (UID: \"c0051a6a-812d-406c-8292-97ff6440ee0e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-wj88p"
Sep 09 20:27:28.415538 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.415407    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0051a6a-812d-406c-8292-97ff6440ee0e-kserve-provision-location\") pod \"isvc-canary-promote-predictor-cb75746f8-wj88p\" (UID: \"c0051a6a-812d-406c-8292-97ff6440ee0e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-wj88p"
Sep 09 20:27:28.415538 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.415426    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shdt4\" (UniqueName: \"kubernetes.io/projected/c0051a6a-812d-406c-8292-97ff6440ee0e-kube-api-access-shdt4\") pod \"isvc-canary-promote-predictor-cb75746f8-wj88p\" (UID: \"c0051a6a-812d-406c-8292-97ff6440ee0e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-wj88p"
Sep 09 20:27:28.415538 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.415472    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0051a6a-812d-406c-8292-97ff6440ee0e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-cb75746f8-wj88p\" (UID: \"c0051a6a-812d-406c-8292-97ff6440ee0e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-wj88p"
Sep 09 20:27:28.415538 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:27:28.415513    2523 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 09 20:27:28.415718 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:27:28.415576    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0051a6a-812d-406c-8292-97ff6440ee0e-proxy-tls podName:c0051a6a-812d-406c-8292-97ff6440ee0e nodeName:}" failed. No retries permitted until 2026-09-09 20:27:28.915555712 +0000 UTC m=+1953.587574688 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c0051a6a-812d-406c-8292-97ff6440ee0e-proxy-tls") pod "isvc-canary-promote-predictor-cb75746f8-wj88p" (UID: "c0051a6a-812d-406c-8292-97ff6440ee0e") : secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 09 20:27:28.415854 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.415837    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0051a6a-812d-406c-8292-97ff6440ee0e-kserve-provision-location\") pod \"isvc-canary-promote-predictor-cb75746f8-wj88p\" (UID: \"c0051a6a-812d-406c-8292-97ff6440ee0e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-wj88p"
Sep 09 20:27:28.416065 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.416047    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0051a6a-812d-406c-8292-97ff6440ee0e-cabundle-cert\") pod \"isvc-canary-promote-predictor-cb75746f8-wj88p\" (UID: \"c0051a6a-812d-406c-8292-97ff6440ee0e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-wj88p"
Sep 09 20:27:28.416098 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.416064    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0051a6a-812d-406c-8292-97ff6440ee0e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-cb75746f8-wj88p\" (UID: \"c0051a6a-812d-406c-8292-97ff6440ee0e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-wj88p"
Sep 09 20:27:28.424523 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.424501    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shdt4\" (UniqueName: \"kubernetes.io/projected/c0051a6a-812d-406c-8292-97ff6440ee0e-kube-api-access-shdt4\") pod \"isvc-canary-promote-predictor-cb75746f8-wj88p\" (UID: \"c0051a6a-812d-406c-8292-97ff6440ee0e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-wj88p"
Sep 09 20:27:28.617435 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.617415    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dab4510d-a522-4a64-b70d-518cd56be114-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-j2mt4\" (UID: \"dab4510d-a522-4a64-b70d-518cd56be114\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-j2mt4"
Sep 09 20:27:28.619697 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.619681    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dab4510d-a522-4a64-b70d-518cd56be114-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-j2mt4\" (UID: \"dab4510d-a522-4a64-b70d-518cd56be114\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-j2mt4"
Sep 09 20:27:28.878765 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.878738    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-j2mt4"
Sep 09 20:27:28.920104 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.920066    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0051a6a-812d-406c-8292-97ff6440ee0e-proxy-tls\") pod \"isvc-canary-promote-predictor-cb75746f8-wj88p\" (UID: \"c0051a6a-812d-406c-8292-97ff6440ee0e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-wj88p"
Sep 09 20:27:28.922497 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:28.922475    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0051a6a-812d-406c-8292-97ff6440ee0e-proxy-tls\") pod \"isvc-canary-promote-predictor-cb75746f8-wj88p\" (UID: \"c0051a6a-812d-406c-8292-97ff6440ee0e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-wj88p"
Sep 09 20:27:29.148779 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:29.148757    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-j2mt4"]
Sep 09 20:27:29.151494 ip-10-0-132-226 kubenswrapper[2523]: W0909 20:27:29.151468    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddab4510d_a522_4a64_b70d_518cd56be114.slice/crio-ccccc1933704609e2c4619006e2b2925e78a15127343ef7e4e5fdd1bc1a5a897 WatchSource:0}: Error finding container ccccc1933704609e2c4619006e2b2925e78a15127343ef7e4e5fdd1bc1a5a897: Status 404 returned error can't find the container with id ccccc1933704609e2c4619006e2b2925e78a15127343ef7e4e5fdd1bc1a5a897
Sep 09 20:27:29.172851 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:29.172831    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-wj88p"
Sep 09 20:27:29.445764 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:29.445690    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-wj88p"]
Sep 09 20:27:29.449129 ip-10-0-132-226 kubenswrapper[2523]: W0909 20:27:29.449090    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0051a6a_812d_406c_8292_97ff6440ee0e.slice/crio-34276df5f0aa551740287e47a51fc397626ea5d3f0cd20700bd5252bf5cbbd3e WatchSource:0}: Error finding container 34276df5f0aa551740287e47a51fc397626ea5d3f0cd20700bd5252bf5cbbd3e: Status 404 returned error can't find the container with id 34276df5f0aa551740287e47a51fc397626ea5d3f0cd20700bd5252bf5cbbd3e
Sep 09 20:27:29.807932 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:29.807850    2523 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45cc6423-62bf-4b6b-bb9b-2f3a1cde1639" path="/var/lib/kubelet/pods/45cc6423-62bf-4b6b-bb9b-2f3a1cde1639/volumes"
Sep 09 20:27:29.808352 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:29.808337    2523 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e36dc0c-fb8a-4970-9d19-1a52f717bedc" path="/var/lib/kubelet/pods/6e36dc0c-fb8a-4970-9d19-1a52f717bedc/volumes"
Sep 09 20:27:29.809048 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:29.809032    2523 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a359bd28-730b-4de8-91fd-48338328ac99" path="/var/lib/kubelet/pods/a359bd28-730b-4de8-91fd-48338328ac99/volumes"
Sep 09 20:27:30.115373 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:30.115337    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-j2mt4" event={"ID":"dab4510d-a522-4a64-b70d-518cd56be114","Type":"ContainerStarted","Data":"1cd59aca5ea3c1594b7a496efb18e3280803483e81ccbf204386ae1a288c61f0"}
Sep 09 20:27:30.115798 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:30.115429    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-j2mt4" event={"ID":"dab4510d-a522-4a64-b70d-518cd56be114","Type":"ContainerStarted","Data":"ccccc1933704609e2c4619006e2b2925e78a15127343ef7e4e5fdd1bc1a5a897"}
Sep 09 20:27:30.117042 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:30.117016    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-wj88p" event={"ID":"c0051a6a-812d-406c-8292-97ff6440ee0e","Type":"ContainerStarted","Data":"e1591cf8e2df719da6bba6b75be7e3da427991d84eec50d4b1be69fc387bab7d"}
Sep 09 20:27:30.117185 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:30.117076    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-wj88p" event={"ID":"c0051a6a-812d-406c-8292-97ff6440ee0e","Type":"ContainerStarted","Data":"34276df5f0aa551740287e47a51fc397626ea5d3f0cd20700bd5252bf5cbbd3e"}
Sep 09 20:27:34.136093 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:34.136061    2523 generic.go:356] "Generic (PLEG): container finished" podID="dab4510d-a522-4a64-b70d-518cd56be114" containerID="1cd59aca5ea3c1594b7a496efb18e3280803483e81ccbf204386ae1a288c61f0" exitCode=0
Sep 09 20:27:34.136502 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:34.136134    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-j2mt4" event={"ID":"dab4510d-a522-4a64-b70d-518cd56be114","Type":"ContainerDied","Data":"1cd59aca5ea3c1594b7a496efb18e3280803483e81ccbf204386ae1a288c61f0"}
Sep 09 20:27:34.137568 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:34.137552    2523 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:27:34.137856 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:34.137834    2523 generic.go:356] "Generic (PLEG): container finished" podID="c0051a6a-812d-406c-8292-97ff6440ee0e" containerID="e1591cf8e2df719da6bba6b75be7e3da427991d84eec50d4b1be69fc387bab7d" exitCode=0
Sep 09 20:27:34.137965 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:34.137943    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-wj88p" event={"ID":"c0051a6a-812d-406c-8292-97ff6440ee0e","Type":"ContainerDied","Data":"e1591cf8e2df719da6bba6b75be7e3da427991d84eec50d4b1be69fc387bab7d"}
Sep 09 20:27:35.148180 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:35.147818    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-wj88p" event={"ID":"c0051a6a-812d-406c-8292-97ff6440ee0e","Type":"ContainerStarted","Data":"fc567103b776f4ff57b03c991e9b0a322db629c346fd9cc77277944c97bc0f91"}
Sep 09 20:27:35.148180 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:35.147934    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-wj88p" event={"ID":"c0051a6a-812d-406c-8292-97ff6440ee0e","Type":"ContainerStarted","Data":"2847a6b65ae1a59eb3addd880d1300699bdff419259631db9fe8346b22616a5a"}
Sep 09 20:27:35.148711 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:35.148544    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-wj88p"
Sep 09 20:27:35.148711 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:35.148590    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-wj88p"
Sep 09 20:27:35.150206 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:35.150144    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-wj88p" podUID="c0051a6a-812d-406c-8292-97ff6440ee0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 09 20:27:35.169360 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:35.169285    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-wj88p" podStartSLOduration=3.286520334 podStartE2EDuration="7.169264925s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.882744591s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:27:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:27:35.167851852 +0000 UTC m=+1959.839870850" watchObservedRunningTime="2026-09-09 20:27:35.169264925 +0000 UTC m=+1959.841283924"
Sep 09 20:27:36.154438 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:36.154395    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-wj88p" podUID="c0051a6a-812d-406c-8292-97ff6440ee0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 09 20:27:38.169335 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:38.169305    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-j2mt4" event={"ID":"dab4510d-a522-4a64-b70d-518cd56be114","Type":"ContainerStarted","Data":"097245d99b7adaaf04ac68ebd61cc7e4b4cc97ad4ad6fc9496d49226e4cb8402"}
Sep 09 20:27:38.169656 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:38.169362    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-j2mt4" event={"ID":"dab4510d-a522-4a64-b70d-518cd56be114","Type":"ContainerStarted","Data":"486133314daa15448f733987686dd03c36f9d345567bad38bf7f4aacd47f87a1"}
Sep 09 20:27:38.169656 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:38.169606    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-j2mt4"
Sep 09 20:27:38.188747 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:38.188698    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-j2mt4" podStartSLOduration=3.06871664 podStartE2EDuration="11.188679654s" totalImagesPullingTime="3.802492811s" totalInitContainerRuntime="4.317470203s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:27:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:27:38.188103421 +0000 UTC m=+1962.860122419" watchObservedRunningTime="2026-09-09 20:27:38.188679654 +0000 UTC m=+1962.860698654"
Sep 09 20:27:39.173569 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:39.173533    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-j2mt4"
Sep 09 20:27:39.174745 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:39.174722    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-j2mt4" podUID="dab4510d-a522-4a64-b70d-518cd56be114" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 09 20:27:40.177681 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:40.177639    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-j2mt4" podUID="dab4510d-a522-4a64-b70d-518cd56be114" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 09 20:27:41.158874 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:41.158847    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-wj88p"
Sep 09 20:27:41.159356 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:41.159329    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-wj88p" podUID="c0051a6a-812d-406c-8292-97ff6440ee0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 09 20:27:45.181764 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:45.181734    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-j2mt4"
Sep 09 20:27:45.182222 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:45.182198    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-j2mt4" podUID="dab4510d-a522-4a64-b70d-518cd56be114" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 09 20:27:51.159855 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:51.159764    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-wj88p" podUID="c0051a6a-812d-406c-8292-97ff6440ee0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 09 20:27:55.183071 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:27:55.183041    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-j2mt4"
Sep 09 20:28:01.159922 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:01.159882    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-wj88p" podUID="c0051a6a-812d-406c-8292-97ff6440ee0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 09 20:28:07.561042 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:07.561009    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-j2mt4"]
Sep 09 20:28:07.561491 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:07.561276    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-j2mt4" podUID="dab4510d-a522-4a64-b70d-518cd56be114" containerName="kserve-container" containerID="cri-o://486133314daa15448f733987686dd03c36f9d345567bad38bf7f4aacd47f87a1" gracePeriod=30
Sep 09 20:28:07.561491 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:07.561335    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-j2mt4" podUID="dab4510d-a522-4a64-b70d-518cd56be114" containerName="kube-rbac-proxy" containerID="cri-o://097245d99b7adaaf04ac68ebd61cc7e4b4cc97ad4ad6fc9496d49226e4cb8402" gracePeriod=30
Sep 09 20:28:08.289433 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:08.289400    2523 generic.go:356] "Generic (PLEG): container finished" podID="dab4510d-a522-4a64-b70d-518cd56be114" containerID="097245d99b7adaaf04ac68ebd61cc7e4b4cc97ad4ad6fc9496d49226e4cb8402" exitCode=2
Sep 09 20:28:08.289591 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:08.289470    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-j2mt4" event={"ID":"dab4510d-a522-4a64-b70d-518cd56be114","Type":"ContainerDied","Data":"097245d99b7adaaf04ac68ebd61cc7e4b4cc97ad4ad6fc9496d49226e4cb8402"}
Sep 09 20:28:10.178291 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:10.178253    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-j2mt4" podUID="dab4510d-a522-4a64-b70d-518cd56be114" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Sep 09 20:28:11.160311 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:11.160274    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-wj88p" podUID="c0051a6a-812d-406c-8292-97ff6440ee0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 09 20:28:12.785997 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:12.785962    2523 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c458x"]
Sep 09 20:28:12.788487 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:12.788469    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c458x"
Sep 09 20:28:12.793494 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:12.793464    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
Sep 09 20:28:12.793628 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:12.793469    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
Sep 09 20:28:12.805558 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:12.805532    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c458x"]
Sep 09 20:28:12.930453 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:12.930427    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvkb9\" (UniqueName: \"kubernetes.io/projected/5baf6a52-56ae-4ceb-a826-6f67d1aae7c7-kube-api-access-nvkb9\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-c458x\" (UID: \"5baf6a52-56ae-4ceb-a826-6f67d1aae7c7\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c458x"
Sep 09 20:28:12.930584 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:12.930455    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5baf6a52-56ae-4ceb-a826-6f67d1aae7c7-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-c458x\" (UID: \"5baf6a52-56ae-4ceb-a826-6f67d1aae7c7\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c458x"
Sep 09 20:28:12.930584 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:12.930478    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5baf6a52-56ae-4ceb-a826-6f67d1aae7c7-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-c458x\" (UID: \"5baf6a52-56ae-4ceb-a826-6f67d1aae7c7\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c458x"
Sep 09 20:28:12.930584 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:12.930549    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5baf6a52-56ae-4ceb-a826-6f67d1aae7c7-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-c458x\" (UID: \"5baf6a52-56ae-4ceb-a826-6f67d1aae7c7\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c458x"
Sep 09 20:28:12.930710 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:12.930590    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5baf6a52-56ae-4ceb-a826-6f67d1aae7c7-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-c458x\" (UID: \"5baf6a52-56ae-4ceb-a826-6f67d1aae7c7\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c458x"
Sep 09 20:28:13.031345 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:13.031314    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5baf6a52-56ae-4ceb-a826-6f67d1aae7c7-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-c458x\" (UID: \"5baf6a52-56ae-4ceb-a826-6f67d1aae7c7\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c458x"
Sep 09 20:28:13.031468 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:13.031365    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5baf6a52-56ae-4ceb-a826-6f67d1aae7c7-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-c458x\" (UID: \"5baf6a52-56ae-4ceb-a826-6f67d1aae7c7\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c458x"
Sep 09 20:28:13.031468 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:13.031393    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5baf6a52-56ae-4ceb-a826-6f67d1aae7c7-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-c458x\" (UID: \"5baf6a52-56ae-4ceb-a826-6f67d1aae7c7\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c458x"
Sep 09 20:28:13.031468 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:13.031453    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvkb9\" (UniqueName: \"kubernetes.io/projected/5baf6a52-56ae-4ceb-a826-6f67d1aae7c7-kube-api-access-nvkb9\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-c458x\" (UID: \"5baf6a52-56ae-4ceb-a826-6f67d1aae7c7\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c458x"
Sep 09 20:28:13.031651 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:13.031624    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5baf6a52-56ae-4ceb-a826-6f67d1aae7c7-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-c458x\" (UID: \"5baf6a52-56ae-4ceb-a826-6f67d1aae7c7\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c458x"
Sep 09 20:28:13.031988 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:13.031969    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5baf6a52-56ae-4ceb-a826-6f67d1aae7c7-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-c458x\" (UID: \"5baf6a52-56ae-4ceb-a826-6f67d1aae7c7\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c458x"
Sep 09 20:28:13.032201 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:13.032184    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5baf6a52-56ae-4ceb-a826-6f67d1aae7c7-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-c458x\" (UID: \"5baf6a52-56ae-4ceb-a826-6f67d1aae7c7\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c458x"
Sep 09 20:28:13.032251 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:13.032207    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5baf6a52-56ae-4ceb-a826-6f67d1aae7c7-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-c458x\" (UID: \"5baf6a52-56ae-4ceb-a826-6f67d1aae7c7\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c458x"
Sep 09 20:28:13.033808 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:13.033789    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5baf6a52-56ae-4ceb-a826-6f67d1aae7c7-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-c458x\" (UID: \"5baf6a52-56ae-4ceb-a826-6f67d1aae7c7\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c458x"
Sep 09 20:28:13.040467 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:13.040422    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvkb9\" (UniqueName: \"kubernetes.io/projected/5baf6a52-56ae-4ceb-a826-6f67d1aae7c7-kube-api-access-nvkb9\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-c458x\" (UID: \"5baf6a52-56ae-4ceb-a826-6f67d1aae7c7\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c458x"
Sep 09 20:28:13.098680 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:13.098645    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c458x"
Sep 09 20:28:13.371211 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:13.371184    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c458x"]
Sep 09 20:28:13.373233 ip-10-0-132-226 kubenswrapper[2523]: W0909 20:28:13.373201    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5baf6a52_56ae_4ceb_a826_6f67d1aae7c7.slice/crio-460e7628b82f64a026c938423e52c1e0458051505e39d7a9808de3573f645da9 WatchSource:0}: Error finding container 460e7628b82f64a026c938423e52c1e0458051505e39d7a9808de3573f645da9: Status 404 returned error can't find the container with id 460e7628b82f64a026c938423e52c1e0458051505e39d7a9808de3573f645da9
Sep 09 20:28:14.313389 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:14.313348    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c458x" event={"ID":"5baf6a52-56ae-4ceb-a826-6f67d1aae7c7","Type":"ContainerStarted","Data":"c8262d7857aba15d7de3f4a5979924444820fc3cb736b485673f9f364ec48b35"}
Sep 09 20:28:14.313758 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:14.313420    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c458x" event={"ID":"5baf6a52-56ae-4ceb-a826-6f67d1aae7c7","Type":"ContainerStarted","Data":"460e7628b82f64a026c938423e52c1e0458051505e39d7a9808de3573f645da9"}
Sep 09 20:28:15.178622 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:15.178582    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-j2mt4" podUID="dab4510d-a522-4a64-b70d-518cd56be114" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Sep 09 20:28:18.331431 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:18.331397    2523 generic.go:356] "Generic (PLEG): container finished" podID="5baf6a52-56ae-4ceb-a826-6f67d1aae7c7" containerID="c8262d7857aba15d7de3f4a5979924444820fc3cb736b485673f9f364ec48b35" exitCode=0
Sep 09 20:28:18.331858 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:18.331475    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c458x" event={"ID":"5baf6a52-56ae-4ceb-a826-6f67d1aae7c7","Type":"ContainerDied","Data":"c8262d7857aba15d7de3f4a5979924444820fc3cb736b485673f9f364ec48b35"}
Sep 09 20:28:19.337630 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:19.337593    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c458x" event={"ID":"5baf6a52-56ae-4ceb-a826-6f67d1aae7c7","Type":"ContainerStarted","Data":"d9b06b7446f58ced57421616ed2469b94daad2a7aa34bf7d855ed76555e97caf"}
Sep 09 20:28:19.337991 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:19.337651    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c458x" event={"ID":"5baf6a52-56ae-4ceb-a826-6f67d1aae7c7","Type":"ContainerStarted","Data":"0fd25b54ae8a636b44d830dbaef7a9a43983124e14d19fba12595817246a97ad"}
Sep 09 20:28:19.337991 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:19.337912    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c458x"
Sep 09 20:28:19.338085 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:19.338045    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c458x"
Sep 09 20:28:19.339394 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:19.339368    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c458x" podUID="5baf6a52-56ae-4ceb-a826-6f67d1aae7c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 09 20:28:19.357243 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:19.357194    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c458x" podStartSLOduration=2.905107271 podStartE2EDuration="7.357179261s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.45207199s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:28:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:28:19.35698848 +0000 UTC m=+2004.029007477" watchObservedRunningTime="2026-09-09 20:28:19.357179261 +0000 UTC m=+2004.029198257"
Sep 09 20:28:20.178025 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:20.177985    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-j2mt4" podUID="dab4510d-a522-4a64-b70d-518cd56be114" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Sep 09 20:28:20.178220 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:20.178089    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-j2mt4"
Sep 09 20:28:20.341553 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:20.341517    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c458x" podUID="5baf6a52-56ae-4ceb-a826-6f67d1aae7c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 09 20:28:21.159831 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:21.159789    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-wj88p" podUID="c0051a6a-812d-406c-8292-97ff6440ee0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 09 20:28:25.177859 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:25.177817    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-j2mt4" podUID="dab4510d-a522-4a64-b70d-518cd56be114" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Sep 09 20:28:25.346392 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:25.346345    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c458x"
Sep 09 20:28:25.347007 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:25.346978    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c458x" podUID="5baf6a52-56ae-4ceb-a826-6f67d1aae7c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 09 20:28:30.177751 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:30.177708    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-j2mt4" podUID="dab4510d-a522-4a64-b70d-518cd56be114" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Sep 09 20:28:31.160245 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:31.160207    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-wj88p" podUID="c0051a6a-812d-406c-8292-97ff6440ee0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 09 20:28:35.178428 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:35.178388    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-j2mt4" podUID="dab4510d-a522-4a64-b70d-518cd56be114" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Sep 09 20:28:35.347789 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:35.347755    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c458x"
Sep 09 20:28:38.197701 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:38.197675    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-j2mt4"
Sep 09 20:28:38.211496 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:38.211462    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dab4510d-a522-4a64-b70d-518cd56be114-kube-rbac-proxy-sar-config\") pod \"dab4510d-a522-4a64-b70d-518cd56be114\" (UID: \"dab4510d-a522-4a64-b70d-518cd56be114\") "
Sep 09 20:28:38.211496 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:38.211492    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dab4510d-a522-4a64-b70d-518cd56be114-cabundle-cert\") pod \"dab4510d-a522-4a64-b70d-518cd56be114\" (UID: \"dab4510d-a522-4a64-b70d-518cd56be114\") "
Sep 09 20:28:38.211672 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:38.211515    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dab4510d-a522-4a64-b70d-518cd56be114-kserve-provision-location\") pod \"dab4510d-a522-4a64-b70d-518cd56be114\" (UID: \"dab4510d-a522-4a64-b70d-518cd56be114\") "
Sep 09 20:28:38.211672 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:38.211551    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dab4510d-a522-4a64-b70d-518cd56be114-proxy-tls\") pod \"dab4510d-a522-4a64-b70d-518cd56be114\" (UID: \"dab4510d-a522-4a64-b70d-518cd56be114\") "
Sep 09 20:28:38.211672 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:38.211571    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nm946\" (UniqueName: \"kubernetes.io/projected/dab4510d-a522-4a64-b70d-518cd56be114-kube-api-access-nm946\") pod \"dab4510d-a522-4a64-b70d-518cd56be114\" (UID: \"dab4510d-a522-4a64-b70d-518cd56be114\") "
Sep 09 20:28:38.211894 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:38.211871    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dab4510d-a522-4a64-b70d-518cd56be114-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dab4510d-a522-4a64-b70d-518cd56be114" (UID: "dab4510d-a522-4a64-b70d-518cd56be114"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:28:38.211965 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:38.211868    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dab4510d-a522-4a64-b70d-518cd56be114-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dab4510d-a522-4a64-b70d-518cd56be114" (UID: "dab4510d-a522-4a64-b70d-518cd56be114"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:28:38.213859 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:38.213833    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dab4510d-a522-4a64-b70d-518cd56be114-kube-api-access-nm946" (OuterVolumeSpecName: "kube-api-access-nm946") pod "dab4510d-a522-4a64-b70d-518cd56be114" (UID: "dab4510d-a522-4a64-b70d-518cd56be114"). InnerVolumeSpecName "kube-api-access-nm946". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:28:38.213859 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:38.213843    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dab4510d-a522-4a64-b70d-518cd56be114-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dab4510d-a522-4a64-b70d-518cd56be114" (UID: "dab4510d-a522-4a64-b70d-518cd56be114"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:28:38.223650 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:38.223622    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dab4510d-a522-4a64-b70d-518cd56be114-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dab4510d-a522-4a64-b70d-518cd56be114" (UID: "dab4510d-a522-4a64-b70d-518cd56be114"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:28:38.312215 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:38.312194    2523 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dab4510d-a522-4a64-b70d-518cd56be114-proxy-tls\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:28:38.312215 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:38.312215    2523 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm946\" (UniqueName: \"kubernetes.io/projected/dab4510d-a522-4a64-b70d-518cd56be114-kube-api-access-nm946\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:28:38.312337 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:38.312226    2523 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dab4510d-a522-4a64-b70d-518cd56be114-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:28:38.312337 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:38.312234    2523 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dab4510d-a522-4a64-b70d-518cd56be114-cabundle-cert\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:28:38.312337 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:38.312244    2523 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dab4510d-a522-4a64-b70d-518cd56be114-kserve-provision-location\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:28:38.422542 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:38.422476    2523 generic.go:356] "Generic (PLEG): container finished" podID="dab4510d-a522-4a64-b70d-518cd56be114" containerID="486133314daa15448f733987686dd03c36f9d345567bad38bf7f4aacd47f87a1" exitCode=137
Sep 09 20:28:38.422542 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:38.422533    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-j2mt4"
Sep 09 20:28:38.422672 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:38.422534    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-j2mt4" event={"ID":"dab4510d-a522-4a64-b70d-518cd56be114","Type":"ContainerDied","Data":"486133314daa15448f733987686dd03c36f9d345567bad38bf7f4aacd47f87a1"}
Sep 09 20:28:38.422672 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:38.422654    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-j2mt4" event={"ID":"dab4510d-a522-4a64-b70d-518cd56be114","Type":"ContainerDied","Data":"ccccc1933704609e2c4619006e2b2925e78a15127343ef7e4e5fdd1bc1a5a897"}
Sep 09 20:28:38.422738 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:38.422681    2523 scope.go:117] "RemoveContainer" containerID="097245d99b7adaaf04ac68ebd61cc7e4b4cc97ad4ad6fc9496d49226e4cb8402"
Sep 09 20:28:38.431292 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:38.431275    2523 scope.go:117] "RemoveContainer" containerID="486133314daa15448f733987686dd03c36f9d345567bad38bf7f4aacd47f87a1"
Sep 09 20:28:38.438454 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:38.438436    2523 scope.go:117] "RemoveContainer" containerID="1cd59aca5ea3c1594b7a496efb18e3280803483e81ccbf204386ae1a288c61f0"
Sep 09 20:28:38.444216 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:38.444192    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-j2mt4"]
Sep 09 20:28:38.446632 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:38.446616    2523 scope.go:117] "RemoveContainer" containerID="097245d99b7adaaf04ac68ebd61cc7e4b4cc97ad4ad6fc9496d49226e4cb8402"
Sep 09 20:28:38.446916 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:28:38.446899    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"097245d99b7adaaf04ac68ebd61cc7e4b4cc97ad4ad6fc9496d49226e4cb8402\": container with ID starting with 097245d99b7adaaf04ac68ebd61cc7e4b4cc97ad4ad6fc9496d49226e4cb8402 not found: ID does not exist" containerID="097245d99b7adaaf04ac68ebd61cc7e4b4cc97ad4ad6fc9496d49226e4cb8402"
Sep 09 20:28:38.446974 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:38.446923    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"097245d99b7adaaf04ac68ebd61cc7e4b4cc97ad4ad6fc9496d49226e4cb8402"} err="failed to get container status \"097245d99b7adaaf04ac68ebd61cc7e4b4cc97ad4ad6fc9496d49226e4cb8402\": rpc error: code = NotFound desc = could not find container \"097245d99b7adaaf04ac68ebd61cc7e4b4cc97ad4ad6fc9496d49226e4cb8402\": container with ID starting with 097245d99b7adaaf04ac68ebd61cc7e4b4cc97ad4ad6fc9496d49226e4cb8402 not found: ID does not exist"
Sep 09 20:28:38.446974 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:38.446939    2523 scope.go:117] "RemoveContainer" containerID="486133314daa15448f733987686dd03c36f9d345567bad38bf7f4aacd47f87a1"
Sep 09 20:28:38.447219 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:28:38.447204    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"486133314daa15448f733987686dd03c36f9d345567bad38bf7f4aacd47f87a1\": container with ID starting with 486133314daa15448f733987686dd03c36f9d345567bad38bf7f4aacd47f87a1 not found: ID does not exist" containerID="486133314daa15448f733987686dd03c36f9d345567bad38bf7f4aacd47f87a1"
Sep 09 20:28:38.447268 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:38.447222    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"486133314daa15448f733987686dd03c36f9d345567bad38bf7f4aacd47f87a1"} err="failed to get container status \"486133314daa15448f733987686dd03c36f9d345567bad38bf7f4aacd47f87a1\": rpc error: code = NotFound desc = could not find container \"486133314daa15448f733987686dd03c36f9d345567bad38bf7f4aacd47f87a1\": container with ID starting with 486133314daa15448f733987686dd03c36f9d345567bad38bf7f4aacd47f87a1 not found: ID does not exist"
Sep 09 20:28:38.447268 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:38.447234    2523 scope.go:117] "RemoveContainer" containerID="1cd59aca5ea3c1594b7a496efb18e3280803483e81ccbf204386ae1a288c61f0"
Sep 09 20:28:38.447482 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:28:38.447465    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cd59aca5ea3c1594b7a496efb18e3280803483e81ccbf204386ae1a288c61f0\": container with ID starting with 1cd59aca5ea3c1594b7a496efb18e3280803483e81ccbf204386ae1a288c61f0 not found: ID does not exist" containerID="1cd59aca5ea3c1594b7a496efb18e3280803483e81ccbf204386ae1a288c61f0"
Sep 09 20:28:38.447525 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:38.447492    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cd59aca5ea3c1594b7a496efb18e3280803483e81ccbf204386ae1a288c61f0"} err="failed to get container status \"1cd59aca5ea3c1594b7a496efb18e3280803483e81ccbf204386ae1a288c61f0\": rpc error: code = NotFound desc = could not find container \"1cd59aca5ea3c1594b7a496efb18e3280803483e81ccbf204386ae1a288c61f0\": container with ID starting with 1cd59aca5ea3c1594b7a496efb18e3280803483e81ccbf204386ae1a288c61f0 not found: ID does not exist"
Sep 09 20:28:38.449968 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:38.449948    2523 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-j2mt4"]
Sep 09 20:28:39.807550 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:39.807518    2523 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dab4510d-a522-4a64-b70d-518cd56be114" path="/var/lib/kubelet/pods/dab4510d-a522-4a64-b70d-518cd56be114/volumes"
Sep 09 20:28:41.159820 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:41.159787    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-wj88p" podUID="c0051a6a-812d-406c-8292-97ff6440ee0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 09 20:28:49.248383 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:49.248335    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-wj88p"]
Sep 09 20:28:49.248841 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:49.248752    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-wj88p" podUID="c0051a6a-812d-406c-8292-97ff6440ee0e" containerName="kserve-container" containerID="cri-o://2847a6b65ae1a59eb3addd880d1300699bdff419259631db9fe8346b22616a5a" gracePeriod=30
Sep 09 20:28:49.249240 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:49.249198    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-wj88p" podUID="c0051a6a-812d-406c-8292-97ff6440ee0e" containerName="kube-rbac-proxy" containerID="cri-o://fc567103b776f4ff57b03c991e9b0a322db629c346fd9cc77277944c97bc0f91" gracePeriod=30
Sep 09 20:28:49.252962 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:49.252938    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-wj88p"
Sep 09 20:28:49.469621 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:49.469586    2523 generic.go:356] "Generic (PLEG): container finished" podID="c0051a6a-812d-406c-8292-97ff6440ee0e" containerID="fc567103b776f4ff57b03c991e9b0a322db629c346fd9cc77277944c97bc0f91" exitCode=2
Sep 09 20:28:49.469755 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:49.469660    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-wj88p" event={"ID":"c0051a6a-812d-406c-8292-97ff6440ee0e","Type":"ContainerDied","Data":"fc567103b776f4ff57b03c991e9b0a322db629c346fd9cc77277944c97bc0f91"}
Sep 09 20:28:51.155730 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:51.155683    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-wj88p" podUID="c0051a6a-812d-406c-8292-97ff6440ee0e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Sep 09 20:28:52.058468 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:52.058434    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c458x"]
Sep 09 20:28:52.058732 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:52.058699    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c458x" podUID="5baf6a52-56ae-4ceb-a826-6f67d1aae7c7" containerName="kserve-container" containerID="cri-o://0fd25b54ae8a636b44d830dbaef7a9a43983124e14d19fba12595817246a97ad" gracePeriod=30
Sep 09 20:28:52.058904 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:52.058750    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c458x" podUID="5baf6a52-56ae-4ceb-a826-6f67d1aae7c7" containerName="kube-rbac-proxy" containerID="cri-o://d9b06b7446f58ced57421616ed2469b94daad2a7aa34bf7d855ed76555e97caf" gracePeriod=30
Sep 09 20:28:52.483016 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:52.482987    2523 generic.go:356] "Generic (PLEG): container finished" podID="5baf6a52-56ae-4ceb-a826-6f67d1aae7c7" containerID="d9b06b7446f58ced57421616ed2469b94daad2a7aa34bf7d855ed76555e97caf" exitCode=2
Sep 09 20:28:52.483391 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:52.483057    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c458x" event={"ID":"5baf6a52-56ae-4ceb-a826-6f67d1aae7c7","Type":"ContainerDied","Data":"d9b06b7446f58ced57421616ed2469b94daad2a7aa34bf7d855ed76555e97caf"}
Sep 09 20:28:53.289121 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:53.289086    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-wj88p"
Sep 09 20:28:53.314761 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:53.314739    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0051a6a-812d-406c-8292-97ff6440ee0e-proxy-tls\") pod \"c0051a6a-812d-406c-8292-97ff6440ee0e\" (UID: \"c0051a6a-812d-406c-8292-97ff6440ee0e\") "
Sep 09 20:28:53.314890 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:53.314777    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shdt4\" (UniqueName: \"kubernetes.io/projected/c0051a6a-812d-406c-8292-97ff6440ee0e-kube-api-access-shdt4\") pod \"c0051a6a-812d-406c-8292-97ff6440ee0e\" (UID: \"c0051a6a-812d-406c-8292-97ff6440ee0e\") "
Sep 09 20:28:53.314890 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:53.314803    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0051a6a-812d-406c-8292-97ff6440ee0e-kserve-provision-location\") pod \"c0051a6a-812d-406c-8292-97ff6440ee0e\" (UID: \"c0051a6a-812d-406c-8292-97ff6440ee0e\") "
Sep 09 20:28:53.314890 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:53.314859    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0051a6a-812d-406c-8292-97ff6440ee0e-cabundle-cert\") pod \"c0051a6a-812d-406c-8292-97ff6440ee0e\" (UID: \"c0051a6a-812d-406c-8292-97ff6440ee0e\") "
Sep 09 20:28:53.315048 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:53.314908    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0051a6a-812d-406c-8292-97ff6440ee0e-kube-rbac-proxy-sar-config\") pod \"c0051a6a-812d-406c-8292-97ff6440ee0e\" (UID: \"c0051a6a-812d-406c-8292-97ff6440ee0e\") "
Sep 09 20:28:53.315231 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:53.315195    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0051a6a-812d-406c-8292-97ff6440ee0e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c0051a6a-812d-406c-8292-97ff6440ee0e" (UID: "c0051a6a-812d-406c-8292-97ff6440ee0e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:28:53.315330 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:53.315315    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0051a6a-812d-406c-8292-97ff6440ee0e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c0051a6a-812d-406c-8292-97ff6440ee0e" (UID: "c0051a6a-812d-406c-8292-97ff6440ee0e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:28:53.315466 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:53.315315    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0051a6a-812d-406c-8292-97ff6440ee0e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c0051a6a-812d-406c-8292-97ff6440ee0e" (UID: "c0051a6a-812d-406c-8292-97ff6440ee0e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:28:53.316932 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:53.316903    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0051a6a-812d-406c-8292-97ff6440ee0e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c0051a6a-812d-406c-8292-97ff6440ee0e" (UID: "c0051a6a-812d-406c-8292-97ff6440ee0e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:28:53.317023 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:53.316956    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0051a6a-812d-406c-8292-97ff6440ee0e-kube-api-access-shdt4" (OuterVolumeSpecName: "kube-api-access-shdt4") pod "c0051a6a-812d-406c-8292-97ff6440ee0e" (UID: "c0051a6a-812d-406c-8292-97ff6440ee0e"). InnerVolumeSpecName "kube-api-access-shdt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:28:53.415996 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:53.415942    2523 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0051a6a-812d-406c-8292-97ff6440ee0e-cabundle-cert\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:28:53.415996 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:53.415962    2523 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0051a6a-812d-406c-8292-97ff6440ee0e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:28:53.415996 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:53.415971    2523 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0051a6a-812d-406c-8292-97ff6440ee0e-proxy-tls\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:28:53.415996 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:53.415979    2523 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shdt4\" (UniqueName: \"kubernetes.io/projected/c0051a6a-812d-406c-8292-97ff6440ee0e-kube-api-access-shdt4\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:28:53.415996 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:53.415987    2523 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0051a6a-812d-406c-8292-97ff6440ee0e-kserve-provision-location\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:28:53.489207 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:53.489182    2523 generic.go:356] "Generic (PLEG): container finished" podID="c0051a6a-812d-406c-8292-97ff6440ee0e" containerID="2847a6b65ae1a59eb3addd880d1300699bdff419259631db9fe8346b22616a5a" exitCode=0
Sep 09 20:28:53.489470 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:53.489231    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-wj88p"
Sep 09 20:28:53.489470 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:53.489256    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-wj88p" event={"ID":"c0051a6a-812d-406c-8292-97ff6440ee0e","Type":"ContainerDied","Data":"2847a6b65ae1a59eb3addd880d1300699bdff419259631db9fe8346b22616a5a"}
Sep 09 20:28:53.489470 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:53.489308    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-wj88p" event={"ID":"c0051a6a-812d-406c-8292-97ff6440ee0e","Type":"ContainerDied","Data":"34276df5f0aa551740287e47a51fc397626ea5d3f0cd20700bd5252bf5cbbd3e"}
Sep 09 20:28:53.489470 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:53.489337    2523 scope.go:117] "RemoveContainer" containerID="fc567103b776f4ff57b03c991e9b0a322db629c346fd9cc77277944c97bc0f91"
Sep 09 20:28:53.498137 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:53.498053    2523 scope.go:117] "RemoveContainer" containerID="2847a6b65ae1a59eb3addd880d1300699bdff419259631db9fe8346b22616a5a"
Sep 09 20:28:53.507133 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:53.507112    2523 scope.go:117] "RemoveContainer" containerID="e1591cf8e2df719da6bba6b75be7e3da427991d84eec50d4b1be69fc387bab7d"
Sep 09 20:28:53.509687 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:53.509667    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-wj88p"]
Sep 09 20:28:53.513165 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:53.513133    2523 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-wj88p"]
Sep 09 20:28:53.515134 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:53.515117    2523 scope.go:117] "RemoveContainer" containerID="fc567103b776f4ff57b03c991e9b0a322db629c346fd9cc77277944c97bc0f91"
Sep 09 20:28:53.515415 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:28:53.515397    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc567103b776f4ff57b03c991e9b0a322db629c346fd9cc77277944c97bc0f91\": container with ID starting with fc567103b776f4ff57b03c991e9b0a322db629c346fd9cc77277944c97bc0f91 not found: ID does not exist" containerID="fc567103b776f4ff57b03c991e9b0a322db629c346fd9cc77277944c97bc0f91"
Sep 09 20:28:53.515466 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:53.515424    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc567103b776f4ff57b03c991e9b0a322db629c346fd9cc77277944c97bc0f91"} err="failed to get container status \"fc567103b776f4ff57b03c991e9b0a322db629c346fd9cc77277944c97bc0f91\": rpc error: code = NotFound desc = could not find container \"fc567103b776f4ff57b03c991e9b0a322db629c346fd9cc77277944c97bc0f91\": container with ID starting with fc567103b776f4ff57b03c991e9b0a322db629c346fd9cc77277944c97bc0f91 not found: ID does not exist"
Sep 09 20:28:53.515466 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:53.515440    2523 scope.go:117] "RemoveContainer" containerID="2847a6b65ae1a59eb3addd880d1300699bdff419259631db9fe8346b22616a5a"
Sep 09 20:28:53.515708 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:28:53.515693    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2847a6b65ae1a59eb3addd880d1300699bdff419259631db9fe8346b22616a5a\": container with ID starting with 2847a6b65ae1a59eb3addd880d1300699bdff419259631db9fe8346b22616a5a not found: ID does not exist" containerID="2847a6b65ae1a59eb3addd880d1300699bdff419259631db9fe8346b22616a5a"
Sep 09 20:28:53.515763 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:53.515710    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2847a6b65ae1a59eb3addd880d1300699bdff419259631db9fe8346b22616a5a"} err="failed to get container status \"2847a6b65ae1a59eb3addd880d1300699bdff419259631db9fe8346b22616a5a\": rpc error: code = NotFound desc = could not find container \"2847a6b65ae1a59eb3addd880d1300699bdff419259631db9fe8346b22616a5a\": container with ID starting with 2847a6b65ae1a59eb3addd880d1300699bdff419259631db9fe8346b22616a5a not found: ID does not exist"
Sep 09 20:28:53.515763 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:53.515722    2523 scope.go:117] "RemoveContainer" containerID="e1591cf8e2df719da6bba6b75be7e3da427991d84eec50d4b1be69fc387bab7d"
Sep 09 20:28:53.515930 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:28:53.515917    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1591cf8e2df719da6bba6b75be7e3da427991d84eec50d4b1be69fc387bab7d\": container with ID starting with e1591cf8e2df719da6bba6b75be7e3da427991d84eec50d4b1be69fc387bab7d not found: ID does not exist" containerID="e1591cf8e2df719da6bba6b75be7e3da427991d84eec50d4b1be69fc387bab7d"
Sep 09 20:28:53.515972 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:53.515932    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1591cf8e2df719da6bba6b75be7e3da427991d84eec50d4b1be69fc387bab7d"} err="failed to get container status \"e1591cf8e2df719da6bba6b75be7e3da427991d84eec50d4b1be69fc387bab7d\": rpc error: code = NotFound desc = could not find container \"e1591cf8e2df719da6bba6b75be7e3da427991d84eec50d4b1be69fc387bab7d\": container with ID starting with e1591cf8e2df719da6bba6b75be7e3da427991d84eec50d4b1be69fc387bab7d not found: ID does not exist"
Sep 09 20:28:53.807164 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:53.807085    2523 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0051a6a-812d-406c-8292-97ff6440ee0e" path="/var/lib/kubelet/pods/c0051a6a-812d-406c-8292-97ff6440ee0e/volumes"
Sep 09 20:28:55.342349 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:55.342311    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c458x" podUID="5baf6a52-56ae-4ceb-a826-6f67d1aae7c7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Sep 09 20:28:55.940391 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:55.940360    2523 scope.go:117] "RemoveContainer" containerID="ffcd89d543e406758788b28350fdd3f89f938c83d0d64bbdeb07e2da7694b13b"
Sep 09 20:28:55.947894 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:55.947873    2523 scope.go:117] "RemoveContainer" containerID="85cf2fc3a48f43060a92539d75bd7a02e1586a29f86cc772bc75d227317a066e"
Sep 09 20:28:55.954673 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:55.954656    2523 scope.go:117] "RemoveContainer" containerID="879c47088e52f399e05f35438b3854c6e8744aebbd2b193b3f7022e833a64beb"
Sep 09 20:28:57.264628 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:57.264601    2523 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2pk9q"]
Sep 09 20:28:57.264987 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:57.264862    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dab4510d-a522-4a64-b70d-518cd56be114" containerName="kserve-container"
Sep 09 20:28:57.264987 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:57.264872    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="dab4510d-a522-4a64-b70d-518cd56be114" containerName="kserve-container"
Sep 09 20:28:57.264987 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:57.264889    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dab4510d-a522-4a64-b70d-518cd56be114" containerName="kube-rbac-proxy"
Sep 09 20:28:57.264987 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:57.264894    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="dab4510d-a522-4a64-b70d-518cd56be114" containerName="kube-rbac-proxy"
Sep 09 20:28:57.264987 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:57.264902    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0051a6a-812d-406c-8292-97ff6440ee0e" containerName="storage-initializer"
Sep 09 20:28:57.264987 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:57.264907    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0051a6a-812d-406c-8292-97ff6440ee0e" containerName="storage-initializer"
Sep 09 20:28:57.264987 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:57.264915    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0051a6a-812d-406c-8292-97ff6440ee0e" containerName="kserve-container"
Sep 09 20:28:57.264987 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:57.264920    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0051a6a-812d-406c-8292-97ff6440ee0e" containerName="kserve-container"
Sep 09 20:28:57.264987 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:57.264925    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0051a6a-812d-406c-8292-97ff6440ee0e" containerName="kube-rbac-proxy"
Sep 09 20:28:57.264987 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:57.264929    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0051a6a-812d-406c-8292-97ff6440ee0e" containerName="kube-rbac-proxy"
Sep 09 20:28:57.264987 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:57.264937    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dab4510d-a522-4a64-b70d-518cd56be114" containerName="storage-initializer"
Sep 09 20:28:57.264987 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:57.264942    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="dab4510d-a522-4a64-b70d-518cd56be114" containerName="storage-initializer"
Sep 09 20:28:57.264987 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:57.264984    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="dab4510d-a522-4a64-b70d-518cd56be114" containerName="kube-rbac-proxy"
Sep 09 20:28:57.264987 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:57.264992    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0051a6a-812d-406c-8292-97ff6440ee0e" containerName="kube-rbac-proxy"
Sep 09 20:28:57.265446 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:57.264997    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0051a6a-812d-406c-8292-97ff6440ee0e" containerName="kserve-container"
Sep 09 20:28:57.265446 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:57.265003    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="dab4510d-a522-4a64-b70d-518cd56be114" containerName="kserve-container"
Sep 09 20:28:57.268324 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:57.268308    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2pk9q"
Sep 09 20:28:57.270913 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:57.270892    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-predictor-serving-cert\""
Sep 09 20:28:57.271068 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:57.270924    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-kube-rbac-proxy-sar-config\""
Sep 09 20:28:57.277556 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:57.277533    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2pk9q"]
Sep 09 20:28:57.344899 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:57.344874    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f67ebb18-149f-478f-a333-0e0d981cf237-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-2pk9q\" (UID: \"f67ebb18-149f-478f-a333-0e0d981cf237\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2pk9q"
Sep 09 20:28:57.344998 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:57.344902    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z2sx\" (UniqueName: \"kubernetes.io/projected/f67ebb18-149f-478f-a333-0e0d981cf237-kube-api-access-6z2sx\") pod \"isvc-triton-predictor-5d7db58676-2pk9q\" (UID: \"f67ebb18-149f-478f-a333-0e0d981cf237\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2pk9q"
Sep 09 20:28:57.344998 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:57.344924    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f67ebb18-149f-478f-a333-0e0d981cf237-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-2pk9q\" (UID: \"f67ebb18-149f-478f-a333-0e0d981cf237\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2pk9q"
Sep 09 20:28:57.344998 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:57.344943    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f67ebb18-149f-478f-a333-0e0d981cf237-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-2pk9q\" (UID: \"f67ebb18-149f-478f-a333-0e0d981cf237\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2pk9q"
Sep 09 20:28:57.345118 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:57.345008    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f67ebb18-149f-478f-a333-0e0d981cf237-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-2pk9q\" (UID: \"f67ebb18-149f-478f-a333-0e0d981cf237\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2pk9q"
Sep 09 20:28:57.446360 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:57.446331    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f67ebb18-149f-478f-a333-0e0d981cf237-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-2pk9q\" (UID: \"f67ebb18-149f-478f-a333-0e0d981cf237\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2pk9q"
Sep 09 20:28:57.446478 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:57.446366    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z2sx\" (UniqueName: \"kubernetes.io/projected/f67ebb18-149f-478f-a333-0e0d981cf237-kube-api-access-6z2sx\") pod \"isvc-triton-predictor-5d7db58676-2pk9q\" (UID: \"f67ebb18-149f-478f-a333-0e0d981cf237\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2pk9q"
Sep 09 20:28:57.446478 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:57.446384    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f67ebb18-149f-478f-a333-0e0d981cf237-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-2pk9q\" (UID: \"f67ebb18-149f-478f-a333-0e0d981cf237\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2pk9q"
Sep 09 20:28:57.446478 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:57.446403    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f67ebb18-149f-478f-a333-0e0d981cf237-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-2pk9q\" (UID: \"f67ebb18-149f-478f-a333-0e0d981cf237\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2pk9q"
Sep 09 20:28:57.446478 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:57.446438    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f67ebb18-149f-478f-a333-0e0d981cf237-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-2pk9q\" (UID: \"f67ebb18-149f-478f-a333-0e0d981cf237\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2pk9q"
Sep 09 20:28:57.446690 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:28:57.446488    2523 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-triton-predictor-serving-cert: secret "isvc-triton-predictor-serving-cert" not found
Sep 09 20:28:57.446690 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:28:57.446560    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f67ebb18-149f-478f-a333-0e0d981cf237-proxy-tls podName:f67ebb18-149f-478f-a333-0e0d981cf237 nodeName:}" failed. No retries permitted until 2026-09-09 20:28:57.946541069 +0000 UTC m=+2042.618560045 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f67ebb18-149f-478f-a333-0e0d981cf237-proxy-tls") pod "isvc-triton-predictor-5d7db58676-2pk9q" (UID: "f67ebb18-149f-478f-a333-0e0d981cf237") : secret "isvc-triton-predictor-serving-cert" not found
Sep 09 20:28:57.446844 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:57.446828    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f67ebb18-149f-478f-a333-0e0d981cf237-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-2pk9q\" (UID: \"f67ebb18-149f-478f-a333-0e0d981cf237\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2pk9q"
Sep 09 20:28:57.447047 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:57.447031    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f67ebb18-149f-478f-a333-0e0d981cf237-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-2pk9q\" (UID: \"f67ebb18-149f-478f-a333-0e0d981cf237\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2pk9q"
Sep 09 20:28:57.447089 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:57.447037    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f67ebb18-149f-478f-a333-0e0d981cf237-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-2pk9q\" (UID: \"f67ebb18-149f-478f-a333-0e0d981cf237\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2pk9q"
Sep 09 20:28:57.457883 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:57.457858    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z2sx\" (UniqueName: \"kubernetes.io/projected/f67ebb18-149f-478f-a333-0e0d981cf237-kube-api-access-6z2sx\") pod \"isvc-triton-predictor-5d7db58676-2pk9q\" (UID: \"f67ebb18-149f-478f-a333-0e0d981cf237\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2pk9q"
Sep 09 20:28:57.949974 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:57.949944    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f67ebb18-149f-478f-a333-0e0d981cf237-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-2pk9q\" (UID: \"f67ebb18-149f-478f-a333-0e0d981cf237\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2pk9q"
Sep 09 20:28:57.952359 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:57.952342    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f67ebb18-149f-478f-a333-0e0d981cf237-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-2pk9q\" (UID: \"f67ebb18-149f-478f-a333-0e0d981cf237\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2pk9q"
Sep 09 20:28:58.179231 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:58.179207    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2pk9q"
Sep 09 20:28:58.330588 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:58.330558    2523 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z"]
Sep 09 20:28:58.333502 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:58.333482    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z"
Sep 09 20:28:58.336162 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:58.336133    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 09 20:28:58.336277 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:58.336258    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 09 20:28:58.343846 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:58.343827    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z"]
Sep 09 20:28:58.454693 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:58.454671    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6faa194-d15e-456c-8ea8-d8d5fe28e31c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z\" (UID: \"e6faa194-d15e-456c-8ea8-d8d5fe28e31c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z"
Sep 09 20:28:58.454815 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:58.454702    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6faa194-d15e-456c-8ea8-d8d5fe28e31c-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z\" (UID: \"e6faa194-d15e-456c-8ea8-d8d5fe28e31c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z"
Sep 09 20:28:58.454815 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:58.454728    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgknt\" (UniqueName: \"kubernetes.io/projected/e6faa194-d15e-456c-8ea8-d8d5fe28e31c-kube-api-access-kgknt\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z\" (UID: \"e6faa194-d15e-456c-8ea8-d8d5fe28e31c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z"
Sep 09 20:28:58.454933 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:58.454879    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6faa194-d15e-456c-8ea8-d8d5fe28e31c-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z\" (UID: \"e6faa194-d15e-456c-8ea8-d8d5fe28e31c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z"
Sep 09 20:28:58.454989 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:58.454927    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6faa194-d15e-456c-8ea8-d8d5fe28e31c-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z\" (UID: \"e6faa194-d15e-456c-8ea8-d8d5fe28e31c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z"
Sep 09 20:28:58.455593 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:58.455575    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2pk9q"]
Sep 09 20:28:58.457807 ip-10-0-132-226 kubenswrapper[2523]: W0909 20:28:58.457784    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf67ebb18_149f_478f_a333_0e0d981cf237.slice/crio-82ca3ddec9ebb7a06c1df6b094f896fa390cae533ad51fa3fb68760abe1f5e14 WatchSource:0}: Error finding container 82ca3ddec9ebb7a06c1df6b094f896fa390cae533ad51fa3fb68760abe1f5e14: Status 404 returned error can't find the container with id 82ca3ddec9ebb7a06c1df6b094f896fa390cae533ad51fa3fb68760abe1f5e14
Sep 09 20:28:58.509760 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:58.509735    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2pk9q" event={"ID":"f67ebb18-149f-478f-a333-0e0d981cf237","Type":"ContainerStarted","Data":"a053344ecb3b2f7dfe024c3565f81191e5d1bee580b1a6ca4322d4677e64d179"}
Sep 09 20:28:58.509860 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:58.509798    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2pk9q" event={"ID":"f67ebb18-149f-478f-a333-0e0d981cf237","Type":"ContainerStarted","Data":"82ca3ddec9ebb7a06c1df6b094f896fa390cae533ad51fa3fb68760abe1f5e14"}
Sep 09 20:28:58.556061 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:58.556033    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgknt\" (UniqueName: \"kubernetes.io/projected/e6faa194-d15e-456c-8ea8-d8d5fe28e31c-kube-api-access-kgknt\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z\" (UID: \"e6faa194-d15e-456c-8ea8-d8d5fe28e31c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z"
Sep 09 20:28:58.556184 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:58.556094    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6faa194-d15e-456c-8ea8-d8d5fe28e31c-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z\" (UID: \"e6faa194-d15e-456c-8ea8-d8d5fe28e31c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z"
Sep 09 20:28:58.556261 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:58.556235    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6faa194-d15e-456c-8ea8-d8d5fe28e31c-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z\" (UID: \"e6faa194-d15e-456c-8ea8-d8d5fe28e31c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z"
Sep 09 20:28:58.556430 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:58.556319    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6faa194-d15e-456c-8ea8-d8d5fe28e31c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z\" (UID: \"e6faa194-d15e-456c-8ea8-d8d5fe28e31c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z"
Sep 09 20:28:58.556430 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:58.556353    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6faa194-d15e-456c-8ea8-d8d5fe28e31c-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z\" (UID: \"e6faa194-d15e-456c-8ea8-d8d5fe28e31c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z"
Sep 09 20:28:58.556609 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:58.556469    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6faa194-d15e-456c-8ea8-d8d5fe28e31c-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z\" (UID: \"e6faa194-d15e-456c-8ea8-d8d5fe28e31c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z"
Sep 09 20:28:58.557032 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:58.557009    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6faa194-d15e-456c-8ea8-d8d5fe28e31c-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z\" (UID: \"e6faa194-d15e-456c-8ea8-d8d5fe28e31c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z"
Sep 09 20:28:58.557117 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:58.557039    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6faa194-d15e-456c-8ea8-d8d5fe28e31c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z\" (UID: \"e6faa194-d15e-456c-8ea8-d8d5fe28e31c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z"
Sep 09 20:28:58.558794 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:58.558771    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6faa194-d15e-456c-8ea8-d8d5fe28e31c-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z\" (UID: \"e6faa194-d15e-456c-8ea8-d8d5fe28e31c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z"
Sep 09 20:28:58.564179 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:58.564136    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgknt\" (UniqueName: \"kubernetes.io/projected/e6faa194-d15e-456c-8ea8-d8d5fe28e31c-kube-api-access-kgknt\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z\" (UID: \"e6faa194-d15e-456c-8ea8-d8d5fe28e31c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z"
Sep 09 20:28:58.644239 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:58.644210    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z"
Sep 09 20:28:58.952804 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:58.952771    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z"]
Sep 09 20:28:58.963578 ip-10-0-132-226 kubenswrapper[2523]: W0909 20:28:58.963549    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6faa194_d15e_456c_8ea8_d8d5fe28e31c.slice/crio-e82695e384677c6437ce6f193bb201d29744d21b55af8fa92d2132a9f57a1aff WatchSource:0}: Error finding container e82695e384677c6437ce6f193bb201d29744d21b55af8fa92d2132a9f57a1aff: Status 404 returned error can't find the container with id e82695e384677c6437ce6f193bb201d29744d21b55af8fa92d2132a9f57a1aff
Sep 09 20:28:59.521796 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:59.521639    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z" event={"ID":"e6faa194-d15e-456c-8ea8-d8d5fe28e31c","Type":"ContainerStarted","Data":"aac7f41afa77ae2170bf3d0f4cdda183bf652ac56f6eb69f7f4cf5bdfbc331bb"}
Sep 09 20:28:59.522277 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:28:59.521809    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z" event={"ID":"e6faa194-d15e-456c-8ea8-d8d5fe28e31c","Type":"ContainerStarted","Data":"e82695e384677c6437ce6f193bb201d29744d21b55af8fa92d2132a9f57a1aff"}
Sep 09 20:29:00.341929 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:29:00.341885    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c458x" podUID="5baf6a52-56ae-4ceb-a826-6f67d1aae7c7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Sep 09 20:29:02.533826 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:29:02.533796    2523 generic.go:356] "Generic (PLEG): container finished" podID="f67ebb18-149f-478f-a333-0e0d981cf237" containerID="a053344ecb3b2f7dfe024c3565f81191e5d1bee580b1a6ca4322d4677e64d179" exitCode=0
Sep 09 20:29:02.534124 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:29:02.533873    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2pk9q" event={"ID":"f67ebb18-149f-478f-a333-0e0d981cf237","Type":"ContainerDied","Data":"a053344ecb3b2f7dfe024c3565f81191e5d1bee580b1a6ca4322d4677e64d179"}
Sep 09 20:29:03.543511 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:29:03.543041    2523 generic.go:356] "Generic (PLEG): container finished" podID="e6faa194-d15e-456c-8ea8-d8d5fe28e31c" containerID="aac7f41afa77ae2170bf3d0f4cdda183bf652ac56f6eb69f7f4cf5bdfbc331bb" exitCode=0
Sep 09 20:29:03.543511 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:29:03.543090    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z" event={"ID":"e6faa194-d15e-456c-8ea8-d8d5fe28e31c","Type":"ContainerDied","Data":"aac7f41afa77ae2170bf3d0f4cdda183bf652ac56f6eb69f7f4cf5bdfbc331bb"}
Sep 09 20:29:04.554706 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:29:04.553803    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z" event={"ID":"e6faa194-d15e-456c-8ea8-d8d5fe28e31c","Type":"ContainerStarted","Data":"32cb9d0a3ee7be0e88fb3d8b06f7a93ea805357d354c5a4993ac4abac1ea84ec"}
Sep 09 20:29:04.554706 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:29:04.553876    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z" event={"ID":"e6faa194-d15e-456c-8ea8-d8d5fe28e31c","Type":"ContainerStarted","Data":"03a45d853064cc96fa10393cf95e4991620ac389272adea8379a7e5f0e7542c7"}
Sep 09 20:29:04.554706 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:29:04.554438    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z"
Sep 09 20:29:04.554706 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:29:04.554507    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z"
Sep 09 20:29:04.557085 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:29:04.556595    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z" podUID="e6faa194-d15e-456c-8ea8-d8d5fe28e31c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 09 20:29:04.574212 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:29:04.574143    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z" podStartSLOduration=2.764757308 podStartE2EDuration="6.574122903s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.809365595s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:28:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:29:04.573310897 +0000 UTC m=+2049.245329895" watchObservedRunningTime="2026-09-09 20:29:04.574122903 +0000 UTC m=+2049.246141937"
Sep 09 20:29:05.342579 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:29:05.342528    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c458x" podUID="5baf6a52-56ae-4ceb-a826-6f67d1aae7c7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Sep 09 20:29:05.342774 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:29:05.342729    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c458x"
Sep 09 20:29:05.562765 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:29:05.562269    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z" podUID="e6faa194-d15e-456c-8ea8-d8d5fe28e31c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 09 20:29:10.342519 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:29:10.342473    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c458x" podUID="5baf6a52-56ae-4ceb-a826-6f67d1aae7c7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Sep 09 20:29:10.570023 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:29:10.569699    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z"
Sep 09 20:29:10.571322 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:29:10.570961    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z" podUID="e6faa194-d15e-456c-8ea8-d8d5fe28e31c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 09 20:29:15.342271 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:29:15.342199    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c458x" podUID="5baf6a52-56ae-4ceb-a826-6f67d1aae7c7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Sep 09 20:29:20.342480 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:29:20.342439    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c458x" podUID="5baf6a52-56ae-4ceb-a826-6f67d1aae7c7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Sep 09 20:29:20.571447 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:29:20.571400    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z" podUID="e6faa194-d15e-456c-8ea8-d8d5fe28e31c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 09 20:29:22.668805 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:29:22.668774    2523 generic.go:356] "Generic (PLEG): container finished" podID="5baf6a52-56ae-4ceb-a826-6f67d1aae7c7" containerID="0fd25b54ae8a636b44d830dbaef7a9a43983124e14d19fba12595817246a97ad" exitCode=137
Sep 09 20:29:22.669287 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:29:22.668852    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c458x" event={"ID":"5baf6a52-56ae-4ceb-a826-6f67d1aae7c7","Type":"ContainerDied","Data":"0fd25b54ae8a636b44d830dbaef7a9a43983124e14d19fba12595817246a97ad"}
Sep 09 20:29:22.777779 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:29:22.777746    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c458x"
Sep 09 20:29:22.878780 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:29:22.878750    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5baf6a52-56ae-4ceb-a826-6f67d1aae7c7-kube-rbac-proxy-sar-config\") pod \"5baf6a52-56ae-4ceb-a826-6f67d1aae7c7\" (UID: \"5baf6a52-56ae-4ceb-a826-6f67d1aae7c7\") "
Sep 09 20:29:22.878956 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:29:22.878863    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5baf6a52-56ae-4ceb-a826-6f67d1aae7c7-cabundle-cert\") pod \"5baf6a52-56ae-4ceb-a826-6f67d1aae7c7\" (UID: \"5baf6a52-56ae-4ceb-a826-6f67d1aae7c7\") "
Sep 09 20:29:22.878956 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:29:22.878897    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nvkb9\" (UniqueName: \"kubernetes.io/projected/5baf6a52-56ae-4ceb-a826-6f67d1aae7c7-kube-api-access-nvkb9\") pod \"5baf6a52-56ae-4ceb-a826-6f67d1aae7c7\" (UID: \"5baf6a52-56ae-4ceb-a826-6f67d1aae7c7\") "
Sep 09 20:29:22.878956 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:29:22.878932    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5baf6a52-56ae-4ceb-a826-6f67d1aae7c7-kserve-provision-location\") pod \"5baf6a52-56ae-4ceb-a826-6f67d1aae7c7\" (UID: \"5baf6a52-56ae-4ceb-a826-6f67d1aae7c7\") "
Sep 09 20:29:22.879141 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:29:22.878976    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5baf6a52-56ae-4ceb-a826-6f67d1aae7c7-proxy-tls\") pod \"5baf6a52-56ae-4ceb-a826-6f67d1aae7c7\" (UID: \"5baf6a52-56ae-4ceb-a826-6f67d1aae7c7\") "
Sep 09 20:29:22.879235 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:29:22.879205    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5baf6a52-56ae-4ceb-a826-6f67d1aae7c7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5baf6a52-56ae-4ceb-a826-6f67d1aae7c7" (UID: "5baf6a52-56ae-4ceb-a826-6f67d1aae7c7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:29:22.879546 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:29:22.879510    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5baf6a52-56ae-4ceb-a826-6f67d1aae7c7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5baf6a52-56ae-4ceb-a826-6f67d1aae7c7" (UID: "5baf6a52-56ae-4ceb-a826-6f67d1aae7c7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:29:22.882330 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:29:22.882290    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5baf6a52-56ae-4ceb-a826-6f67d1aae7c7-kube-api-access-nvkb9" (OuterVolumeSpecName: "kube-api-access-nvkb9") pod "5baf6a52-56ae-4ceb-a826-6f67d1aae7c7" (UID: "5baf6a52-56ae-4ceb-a826-6f67d1aae7c7"). InnerVolumeSpecName "kube-api-access-nvkb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:29:22.882437 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:29:22.882368    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5baf6a52-56ae-4ceb-a826-6f67d1aae7c7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5baf6a52-56ae-4ceb-a826-6f67d1aae7c7" (UID: "5baf6a52-56ae-4ceb-a826-6f67d1aae7c7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:29:22.891386 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:29:22.891359    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5baf6a52-56ae-4ceb-a826-6f67d1aae7c7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5baf6a52-56ae-4ceb-a826-6f67d1aae7c7" (UID: "5baf6a52-56ae-4ceb-a826-6f67d1aae7c7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:29:22.979736 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:29:22.979708    2523 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5baf6a52-56ae-4ceb-a826-6f67d1aae7c7-cabundle-cert\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:29:22.979736 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:29:22.979737    2523 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvkb9\" (UniqueName: \"kubernetes.io/projected/5baf6a52-56ae-4ceb-a826-6f67d1aae7c7-kube-api-access-nvkb9\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:29:22.979939 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:29:22.979748    2523 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5baf6a52-56ae-4ceb-a826-6f67d1aae7c7-kserve-provision-location\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:29:22.979939 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:29:22.979759    2523 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5baf6a52-56ae-4ceb-a826-6f67d1aae7c7-proxy-tls\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:29:22.979939 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:29:22.979768    2523 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5baf6a52-56ae-4ceb-a826-6f67d1aae7c7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:29:23.677987 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:29:23.677756    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c458x" event={"ID":"5baf6a52-56ae-4ceb-a826-6f67d1aae7c7","Type":"ContainerDied","Data":"460e7628b82f64a026c938423e52c1e0458051505e39d7a9808de3573f645da9"}
Sep 09 20:29:23.677987 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:29:23.677817    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c458x"
Sep 09 20:29:23.677987 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:29:23.677848    2523 scope.go:117] "RemoveContainer" containerID="d9b06b7446f58ced57421616ed2469b94daad2a7aa34bf7d855ed76555e97caf"
Sep 09 20:29:23.690649 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:29:23.690626    2523 scope.go:117] "RemoveContainer" containerID="0fd25b54ae8a636b44d830dbaef7a9a43983124e14d19fba12595817246a97ad"
Sep 09 20:29:23.700202 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:29:23.700181    2523 scope.go:117] "RemoveContainer" containerID="c8262d7857aba15d7de3f4a5979924444820fc3cb736b485673f9f364ec48b35"
Sep 09 20:29:23.702726 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:29:23.702703    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c458x"]
Sep 09 20:29:23.709177 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:29:23.709128    2523 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c458x"]
Sep 09 20:29:23.810537 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:29:23.810496    2523 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5baf6a52-56ae-4ceb-a826-6f67d1aae7c7" path="/var/lib/kubelet/pods/5baf6a52-56ae-4ceb-a826-6f67d1aae7c7/volumes"
Sep 09 20:29:30.570877 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:29:30.570818    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z" podUID="e6faa194-d15e-456c-8ea8-d8d5fe28e31c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 09 20:29:40.571051 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:29:40.570811    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z" podUID="e6faa194-d15e-456c-8ea8-d8d5fe28e31c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 09 20:29:50.571679 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:29:50.571640    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z" podUID="e6faa194-d15e-456c-8ea8-d8d5fe28e31c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 09 20:30:00.570939 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:00.570891    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z" podUID="e6faa194-d15e-456c-8ea8-d8d5fe28e31c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 09 20:30:10.570985 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:10.570944    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z" podUID="e6faa194-d15e-456c-8ea8-d8d5fe28e31c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 09 20:30:20.571482 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:20.571443    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z" podUID="e6faa194-d15e-456c-8ea8-d8d5fe28e31c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 09 20:30:27.802882 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:27.802822    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z" podUID="e6faa194-d15e-456c-8ea8-d8d5fe28e31c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 09 20:30:37.808406 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:37.808376    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z"
Sep 09 20:30:38.507447 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:38.507406    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z"]
Sep 09 20:30:38.507899 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:38.507736    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z" podUID="e6faa194-d15e-456c-8ea8-d8d5fe28e31c" containerName="kserve-container" containerID="cri-o://03a45d853064cc96fa10393cf95e4991620ac389272adea8379a7e5f0e7542c7" gracePeriod=30
Sep 09 20:30:38.507899 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:38.507811    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z" podUID="e6faa194-d15e-456c-8ea8-d8d5fe28e31c" containerName="kube-rbac-proxy" containerID="cri-o://32cb9d0a3ee7be0e88fb3d8b06f7a93ea805357d354c5a4993ac4abac1ea84ec" gracePeriod=30
Sep 09 20:30:40.562639 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:40.562594    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z" podUID="e6faa194-d15e-456c-8ea8-d8d5fe28e31c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Sep 09 20:30:43.722241 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:43.722145    2523 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9nvcp"]
Sep 09 20:30:43.722727 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:43.722584    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5baf6a52-56ae-4ceb-a826-6f67d1aae7c7" containerName="storage-initializer"
Sep 09 20:30:43.722727 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:43.722603    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="5baf6a52-56ae-4ceb-a826-6f67d1aae7c7" containerName="storage-initializer"
Sep 09 20:30:43.722727 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:43.722618    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5baf6a52-56ae-4ceb-a826-6f67d1aae7c7" containerName="kube-rbac-proxy"
Sep 09 20:30:43.722727 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:43.722627    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="5baf6a52-56ae-4ceb-a826-6f67d1aae7c7" containerName="kube-rbac-proxy"
Sep 09 20:30:43.722727 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:43.722642    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5baf6a52-56ae-4ceb-a826-6f67d1aae7c7" containerName="kserve-container"
Sep 09 20:30:43.722727 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:43.722651    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="5baf6a52-56ae-4ceb-a826-6f67d1aae7c7" containerName="kserve-container"
Sep 09 20:30:43.723056 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:43.722758    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="5baf6a52-56ae-4ceb-a826-6f67d1aae7c7" containerName="kserve-container"
Sep 09 20:30:43.723056 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:43.722771    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="5baf6a52-56ae-4ceb-a826-6f67d1aae7c7" containerName="kube-rbac-proxy"
Sep 09 20:30:43.736500 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:43.736470    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9nvcp"
Sep 09 20:30:43.741900 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:43.741834    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9nvcp"]
Sep 09 20:30:43.742354 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:43.742328    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 09 20:30:43.742473 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:43.742362    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 09 20:30:43.858020 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:43.857980    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvg5l\" (UniqueName: \"kubernetes.io/projected/9b2bd21a-9cad-4786-b3eb-fe3b1f07b821-kube-api-access-xvg5l\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-9nvcp\" (UID: \"9b2bd21a-9cad-4786-b3eb-fe3b1f07b821\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9nvcp"
Sep 09 20:30:43.858020 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:43.858022    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b2bd21a-9cad-4786-b3eb-fe3b1f07b821-cabundle-cert\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-9nvcp\" (UID: \"9b2bd21a-9cad-4786-b3eb-fe3b1f07b821\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9nvcp"
Sep 09 20:30:43.858288 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:43.858078    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b2bd21a-9cad-4786-b3eb-fe3b1f07b821-proxy-tls\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-9nvcp\" (UID: \"9b2bd21a-9cad-4786-b3eb-fe3b1f07b821\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9nvcp"
Sep 09 20:30:43.858288 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:43.858130    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b2bd21a-9cad-4786-b3eb-fe3b1f07b821-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-9nvcp\" (UID: \"9b2bd21a-9cad-4786-b3eb-fe3b1f07b821\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9nvcp"
Sep 09 20:30:43.858288 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:43.858203    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b2bd21a-9cad-4786-b3eb-fe3b1f07b821-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-9nvcp\" (UID: \"9b2bd21a-9cad-4786-b3eb-fe3b1f07b821\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9nvcp"
Sep 09 20:30:43.958722 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:43.958689    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b2bd21a-9cad-4786-b3eb-fe3b1f07b821-proxy-tls\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-9nvcp\" (UID: \"9b2bd21a-9cad-4786-b3eb-fe3b1f07b821\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9nvcp"
Sep 09 20:30:43.958882 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:43.958741    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b2bd21a-9cad-4786-b3eb-fe3b1f07b821-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-9nvcp\" (UID: \"9b2bd21a-9cad-4786-b3eb-fe3b1f07b821\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9nvcp"
Sep 09 20:30:43.958882 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:43.958783    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b2bd21a-9cad-4786-b3eb-fe3b1f07b821-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-9nvcp\" (UID: \"9b2bd21a-9cad-4786-b3eb-fe3b1f07b821\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9nvcp"
Sep 09 20:30:43.958882 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:43.958828    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvg5l\" (UniqueName: \"kubernetes.io/projected/9b2bd21a-9cad-4786-b3eb-fe3b1f07b821-kube-api-access-xvg5l\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-9nvcp\" (UID: \"9b2bd21a-9cad-4786-b3eb-fe3b1f07b821\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9nvcp"
Sep 09 20:30:43.958882 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:43.958859    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b2bd21a-9cad-4786-b3eb-fe3b1f07b821-cabundle-cert\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-9nvcp\" (UID: \"9b2bd21a-9cad-4786-b3eb-fe3b1f07b821\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9nvcp"
Sep 09 20:30:43.959067 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:30:43.958891    2523 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 09 20:30:43.959067 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:30:43.958988    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b2bd21a-9cad-4786-b3eb-fe3b1f07b821-proxy-tls podName:9b2bd21a-9cad-4786-b3eb-fe3b1f07b821 nodeName:}" failed. No retries permitted until 2026-09-09 20:30:44.458967103 +0000 UTC m=+2149.130986086 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9b2bd21a-9cad-4786-b3eb-fe3b1f07b821-proxy-tls") pod "isvc-canary-rollback-predictor-585bd9bcc8-9nvcp" (UID: "9b2bd21a-9cad-4786-b3eb-fe3b1f07b821") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 09 20:30:43.959303 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:43.959279    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b2bd21a-9cad-4786-b3eb-fe3b1f07b821-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-9nvcp\" (UID: \"9b2bd21a-9cad-4786-b3eb-fe3b1f07b821\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9nvcp"
Sep 09 20:30:43.959557 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:43.959538    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b2bd21a-9cad-4786-b3eb-fe3b1f07b821-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-9nvcp\" (UID: \"9b2bd21a-9cad-4786-b3eb-fe3b1f07b821\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9nvcp"
Sep 09 20:30:43.959627 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:43.959555    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b2bd21a-9cad-4786-b3eb-fe3b1f07b821-cabundle-cert\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-9nvcp\" (UID: \"9b2bd21a-9cad-4786-b3eb-fe3b1f07b821\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9nvcp"
Sep 09 20:30:43.969680 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:43.969658    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvg5l\" (UniqueName: \"kubernetes.io/projected/9b2bd21a-9cad-4786-b3eb-fe3b1f07b821-kube-api-access-xvg5l\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-9nvcp\" (UID: \"9b2bd21a-9cad-4786-b3eb-fe3b1f07b821\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9nvcp"
Sep 09 20:30:44.462919 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:44.462878    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b2bd21a-9cad-4786-b3eb-fe3b1f07b821-proxy-tls\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-9nvcp\" (UID: \"9b2bd21a-9cad-4786-b3eb-fe3b1f07b821\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9nvcp"
Sep 09 20:30:44.465685 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:44.465652    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b2bd21a-9cad-4786-b3eb-fe3b1f07b821-proxy-tls\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-9nvcp\" (UID: \"9b2bd21a-9cad-4786-b3eb-fe3b1f07b821\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9nvcp"
Sep 09 20:30:44.649248 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:44.649207    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9nvcp"
Sep 09 20:30:45.562796 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:45.562750    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z" podUID="e6faa194-d15e-456c-8ea8-d8d5fe28e31c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Sep 09 20:30:47.803020 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:47.802930    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z" podUID="e6faa194-d15e-456c-8ea8-d8d5fe28e31c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 09 20:30:50.562466 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:50.562421    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z" podUID="e6faa194-d15e-456c-8ea8-d8d5fe28e31c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Sep 09 20:30:50.562999 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:50.562561    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z"
Sep 09 20:30:55.562892 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:55.562853    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z" podUID="e6faa194-d15e-456c-8ea8-d8d5fe28e31c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Sep 09 20:30:55.983773 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:55.983744    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z"
Sep 09 20:30:56.054826 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:56.054804    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kgknt\" (UniqueName: \"kubernetes.io/projected/e6faa194-d15e-456c-8ea8-d8d5fe28e31c-kube-api-access-kgknt\") pod \"e6faa194-d15e-456c-8ea8-d8d5fe28e31c\" (UID: \"e6faa194-d15e-456c-8ea8-d8d5fe28e31c\") "
Sep 09 20:30:56.054980 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:56.054840    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6faa194-d15e-456c-8ea8-d8d5fe28e31c-kserve-provision-location\") pod \"e6faa194-d15e-456c-8ea8-d8d5fe28e31c\" (UID: \"e6faa194-d15e-456c-8ea8-d8d5fe28e31c\") "
Sep 09 20:30:56.054980 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:56.054882    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6faa194-d15e-456c-8ea8-d8d5fe28e31c-cabundle-cert\") pod \"e6faa194-d15e-456c-8ea8-d8d5fe28e31c\" (UID: \"e6faa194-d15e-456c-8ea8-d8d5fe28e31c\") "
Sep 09 20:30:56.054980 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:56.054933    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6faa194-d15e-456c-8ea8-d8d5fe28e31c-kube-rbac-proxy-sar-config\") pod \"e6faa194-d15e-456c-8ea8-d8d5fe28e31c\" (UID: \"e6faa194-d15e-456c-8ea8-d8d5fe28e31c\") "
Sep 09 20:30:56.054980 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:56.054953    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6faa194-d15e-456c-8ea8-d8d5fe28e31c-proxy-tls\") pod \"e6faa194-d15e-456c-8ea8-d8d5fe28e31c\" (UID: \"e6faa194-d15e-456c-8ea8-d8d5fe28e31c\") "
Sep 09 20:30:56.055244 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:56.055216    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6faa194-d15e-456c-8ea8-d8d5fe28e31c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e6faa194-d15e-456c-8ea8-d8d5fe28e31c" (UID: "e6faa194-d15e-456c-8ea8-d8d5fe28e31c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:30:56.055304 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:56.055268    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6faa194-d15e-456c-8ea8-d8d5fe28e31c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e6faa194-d15e-456c-8ea8-d8d5fe28e31c" (UID: "e6faa194-d15e-456c-8ea8-d8d5fe28e31c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:30:56.055361 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:56.055317    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6faa194-d15e-456c-8ea8-d8d5fe28e31c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e6faa194-d15e-456c-8ea8-d8d5fe28e31c" (UID: "e6faa194-d15e-456c-8ea8-d8d5fe28e31c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:30:56.057025 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:56.057005    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6faa194-d15e-456c-8ea8-d8d5fe28e31c-kube-api-access-kgknt" (OuterVolumeSpecName: "kube-api-access-kgknt") pod "e6faa194-d15e-456c-8ea8-d8d5fe28e31c" (UID: "e6faa194-d15e-456c-8ea8-d8d5fe28e31c"). InnerVolumeSpecName "kube-api-access-kgknt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:30:56.057025 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:56.057013    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6faa194-d15e-456c-8ea8-d8d5fe28e31c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e6faa194-d15e-456c-8ea8-d8d5fe28e31c" (UID: "e6faa194-d15e-456c-8ea8-d8d5fe28e31c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:30:56.092574 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:56.092551    2523 generic.go:356] "Generic (PLEG): container finished" podID="e6faa194-d15e-456c-8ea8-d8d5fe28e31c" containerID="32cb9d0a3ee7be0e88fb3d8b06f7a93ea805357d354c5a4993ac4abac1ea84ec" exitCode=2
Sep 09 20:30:56.092574 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:56.092571    2523 generic.go:356] "Generic (PLEG): container finished" podID="e6faa194-d15e-456c-8ea8-d8d5fe28e31c" containerID="03a45d853064cc96fa10393cf95e4991620ac389272adea8379a7e5f0e7542c7" exitCode=0
Sep 09 20:30:56.092756 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:56.092609    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z"
Sep 09 20:30:56.092756 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:56.092602    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z" event={"ID":"e6faa194-d15e-456c-8ea8-d8d5fe28e31c","Type":"ContainerDied","Data":"32cb9d0a3ee7be0e88fb3d8b06f7a93ea805357d354c5a4993ac4abac1ea84ec"}
Sep 09 20:30:56.092756 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:56.092742    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z" event={"ID":"e6faa194-d15e-456c-8ea8-d8d5fe28e31c","Type":"ContainerDied","Data":"03a45d853064cc96fa10393cf95e4991620ac389272adea8379a7e5f0e7542c7"}
Sep 09 20:30:56.092919 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:56.092774    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z" event={"ID":"e6faa194-d15e-456c-8ea8-d8d5fe28e31c","Type":"ContainerDied","Data":"e82695e384677c6437ce6f193bb201d29744d21b55af8fa92d2132a9f57a1aff"}
Sep 09 20:30:56.092919 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:56.092808    2523 scope.go:117] "RemoveContainer" containerID="32cb9d0a3ee7be0e88fb3d8b06f7a93ea805357d354c5a4993ac4abac1ea84ec"
Sep 09 20:30:56.102260 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:56.102046    2523 scope.go:117] "RemoveContainer" containerID="03a45d853064cc96fa10393cf95e4991620ac389272adea8379a7e5f0e7542c7"
Sep 09 20:30:56.111053 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:56.111036    2523 scope.go:117] "RemoveContainer" containerID="aac7f41afa77ae2170bf3d0f4cdda183bf652ac56f6eb69f7f4cf5bdfbc331bb"
Sep 09 20:30:56.115336 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:56.115315    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z"]
Sep 09 20:30:56.118825 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:56.118804    2523 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-8ng6z"]
Sep 09 20:30:56.119999 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:56.119987    2523 scope.go:117] "RemoveContainer" containerID="32cb9d0a3ee7be0e88fb3d8b06f7a93ea805357d354c5a4993ac4abac1ea84ec"
Sep 09 20:30:56.120287 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:30:56.120268    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32cb9d0a3ee7be0e88fb3d8b06f7a93ea805357d354c5a4993ac4abac1ea84ec\": container with ID starting with 32cb9d0a3ee7be0e88fb3d8b06f7a93ea805357d354c5a4993ac4abac1ea84ec not found: ID does not exist" containerID="32cb9d0a3ee7be0e88fb3d8b06f7a93ea805357d354c5a4993ac4abac1ea84ec"
Sep 09 20:30:56.120346 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:56.120296    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32cb9d0a3ee7be0e88fb3d8b06f7a93ea805357d354c5a4993ac4abac1ea84ec"} err="failed to get container status \"32cb9d0a3ee7be0e88fb3d8b06f7a93ea805357d354c5a4993ac4abac1ea84ec\": rpc error: code = NotFound desc = could not find container \"32cb9d0a3ee7be0e88fb3d8b06f7a93ea805357d354c5a4993ac4abac1ea84ec\": container with ID starting with 32cb9d0a3ee7be0e88fb3d8b06f7a93ea805357d354c5a4993ac4abac1ea84ec not found: ID does not exist"
Sep 09 20:30:56.120346 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:56.120312    2523 scope.go:117] "RemoveContainer" containerID="03a45d853064cc96fa10393cf95e4991620ac389272adea8379a7e5f0e7542c7"
Sep 09 20:30:56.120583 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:30:56.120565    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03a45d853064cc96fa10393cf95e4991620ac389272adea8379a7e5f0e7542c7\": container with ID starting with 03a45d853064cc96fa10393cf95e4991620ac389272adea8379a7e5f0e7542c7 not found: ID does not exist" containerID="03a45d853064cc96fa10393cf95e4991620ac389272adea8379a7e5f0e7542c7"
Sep 09 20:30:56.120653 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:56.120591    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03a45d853064cc96fa10393cf95e4991620ac389272adea8379a7e5f0e7542c7"} err="failed to get container status \"03a45d853064cc96fa10393cf95e4991620ac389272adea8379a7e5f0e7542c7\": rpc error: code = NotFound desc = could not find container \"03a45d853064cc96fa10393cf95e4991620ac389272adea8379a7e5f0e7542c7\": container with ID starting with 03a45d853064cc96fa10393cf95e4991620ac389272adea8379a7e5f0e7542c7 not found: ID does not exist"
Sep 09 20:30:56.120653 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:56.120612    2523 scope.go:117] "RemoveContainer" containerID="aac7f41afa77ae2170bf3d0f4cdda183bf652ac56f6eb69f7f4cf5bdfbc331bb"
Sep 09 20:30:56.120917 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:30:56.120902    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aac7f41afa77ae2170bf3d0f4cdda183bf652ac56f6eb69f7f4cf5bdfbc331bb\": container with ID starting with aac7f41afa77ae2170bf3d0f4cdda183bf652ac56f6eb69f7f4cf5bdfbc331bb not found: ID does not exist" containerID="aac7f41afa77ae2170bf3d0f4cdda183bf652ac56f6eb69f7f4cf5bdfbc331bb"
Sep 09 20:30:56.120917 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:56.120921    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aac7f41afa77ae2170bf3d0f4cdda183bf652ac56f6eb69f7f4cf5bdfbc331bb"} err="failed to get container status \"aac7f41afa77ae2170bf3d0f4cdda183bf652ac56f6eb69f7f4cf5bdfbc331bb\": rpc error: code = NotFound desc = could not find container \"aac7f41afa77ae2170bf3d0f4cdda183bf652ac56f6eb69f7f4cf5bdfbc331bb\": container with ID starting with aac7f41afa77ae2170bf3d0f4cdda183bf652ac56f6eb69f7f4cf5bdfbc331bb not found: ID does not exist"
Sep 09 20:30:56.121020 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:56.120934    2523 scope.go:117] "RemoveContainer" containerID="32cb9d0a3ee7be0e88fb3d8b06f7a93ea805357d354c5a4993ac4abac1ea84ec"
Sep 09 20:30:56.121206 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:56.121187    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32cb9d0a3ee7be0e88fb3d8b06f7a93ea805357d354c5a4993ac4abac1ea84ec"} err="failed to get container status \"32cb9d0a3ee7be0e88fb3d8b06f7a93ea805357d354c5a4993ac4abac1ea84ec\": rpc error: code = NotFound desc = could not find container \"32cb9d0a3ee7be0e88fb3d8b06f7a93ea805357d354c5a4993ac4abac1ea84ec\": container with ID starting with 32cb9d0a3ee7be0e88fb3d8b06f7a93ea805357d354c5a4993ac4abac1ea84ec not found: ID does not exist"
Sep 09 20:30:56.121251 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:56.121207    2523 scope.go:117] "RemoveContainer" containerID="03a45d853064cc96fa10393cf95e4991620ac389272adea8379a7e5f0e7542c7"
Sep 09 20:30:56.121442 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:56.121425    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03a45d853064cc96fa10393cf95e4991620ac389272adea8379a7e5f0e7542c7"} err="failed to get container status \"03a45d853064cc96fa10393cf95e4991620ac389272adea8379a7e5f0e7542c7\": rpc error: code = NotFound desc = could not find container \"03a45d853064cc96fa10393cf95e4991620ac389272adea8379a7e5f0e7542c7\": container with ID starting with 03a45d853064cc96fa10393cf95e4991620ac389272adea8379a7e5f0e7542c7 not found: ID does not exist"
Sep 09 20:30:56.121500 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:56.121443    2523 scope.go:117] "RemoveContainer" containerID="aac7f41afa77ae2170bf3d0f4cdda183bf652ac56f6eb69f7f4cf5bdfbc331bb"
Sep 09 20:30:56.121706 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:56.121689    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aac7f41afa77ae2170bf3d0f4cdda183bf652ac56f6eb69f7f4cf5bdfbc331bb"} err="failed to get container status \"aac7f41afa77ae2170bf3d0f4cdda183bf652ac56f6eb69f7f4cf5bdfbc331bb\": rpc error: code = NotFound desc = could not find container \"aac7f41afa77ae2170bf3d0f4cdda183bf652ac56f6eb69f7f4cf5bdfbc331bb\": container with ID starting with aac7f41afa77ae2170bf3d0f4cdda183bf652ac56f6eb69f7f4cf5bdfbc331bb not found: ID does not exist"
Sep 09 20:30:56.156375 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:56.156356    2523 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6faa194-d15e-456c-8ea8-d8d5fe28e31c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:30:56.156489 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:56.156376    2523 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6faa194-d15e-456c-8ea8-d8d5fe28e31c-proxy-tls\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:30:56.156489 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:56.156388    2523 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgknt\" (UniqueName: \"kubernetes.io/projected/e6faa194-d15e-456c-8ea8-d8d5fe28e31c-kube-api-access-kgknt\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:30:56.156489 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:56.156396    2523 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6faa194-d15e-456c-8ea8-d8d5fe28e31c-kserve-provision-location\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:30:56.156489 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:56.156405    2523 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6faa194-d15e-456c-8ea8-d8d5fe28e31c-cabundle-cert\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:30:56.180561 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:56.180542    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9nvcp"]
Sep 09 20:30:56.182998 ip-10-0-132-226 kubenswrapper[2523]: W0909 20:30:56.182974    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b2bd21a_9cad_4786_b3eb_fe3b1f07b821.slice/crio-0df10328dc4ee530f2b55b94d381a4335f0e794332c85b9d0d789856b7806aac WatchSource:0}: Error finding container 0df10328dc4ee530f2b55b94d381a4335f0e794332c85b9d0d789856b7806aac: Status 404 returned error can't find the container with id 0df10328dc4ee530f2b55b94d381a4335f0e794332c85b9d0d789856b7806aac
Sep 09 20:30:57.098468 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:57.098428    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2pk9q" event={"ID":"f67ebb18-149f-478f-a333-0e0d981cf237","Type":"ContainerStarted","Data":"01b8063d5c674a95819ded4ac05436f981563303860d09caa6e735b1428dbd97"}
Sep 09 20:30:57.098941 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:57.098533    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2pk9q" event={"ID":"f67ebb18-149f-478f-a333-0e0d981cf237","Type":"ContainerStarted","Data":"1b7176e1530e3f8b24712752258b0ed0187a3f74f832ff79a6c69d3637c80468"}
Sep 09 20:30:57.098941 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:57.098569    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2pk9q"
Sep 09 20:30:57.100131 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:57.100105    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9nvcp" event={"ID":"9b2bd21a-9cad-4786-b3eb-fe3b1f07b821","Type":"ContainerStarted","Data":"c80ddaf7a08420e54abfcaf3cd8189b197f8c5cee653e1aa2b0cb04f72b61270"}
Sep 09 20:30:57.100276 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:57.100179    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9nvcp" event={"ID":"9b2bd21a-9cad-4786-b3eb-fe3b1f07b821","Type":"ContainerStarted","Data":"0df10328dc4ee530f2b55b94d381a4335f0e794332c85b9d0d789856b7806aac"}
Sep 09 20:30:57.117294 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:57.117247    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2pk9q" podStartSLOduration=2.227718744 podStartE2EDuration="2m0.117233708s" totalImagesPullingTime="1m54.058546046s" totalInitContainerRuntime="3.830968918s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:28:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:30:57.115513063 +0000 UTC m=+2161.787532061" watchObservedRunningTime="2026-09-09 20:30:57.117233708 +0000 UTC m=+2161.789252708"
Sep 09 20:30:57.807249 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:57.807219    2523 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6faa194-d15e-456c-8ea8-d8d5fe28e31c" path="/var/lib/kubelet/pods/e6faa194-d15e-456c-8ea8-d8d5fe28e31c/volumes"
Sep 09 20:30:58.104971 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:58.104928    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2pk9q"
Sep 09 20:30:58.105950 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:58.105922    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2pk9q" podUID="f67ebb18-149f-478f-a333-0e0d981cf237" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 09 20:30:59.108873 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:30:59.108836    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2pk9q" podUID="f67ebb18-149f-478f-a333-0e0d981cf237" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 09 20:31:01.116966 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:01.116937    2523 generic.go:356] "Generic (PLEG): container finished" podID="9b2bd21a-9cad-4786-b3eb-fe3b1f07b821" containerID="c80ddaf7a08420e54abfcaf3cd8189b197f8c5cee653e1aa2b0cb04f72b61270" exitCode=0
Sep 09 20:31:01.117413 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:01.117014    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9nvcp" event={"ID":"9b2bd21a-9cad-4786-b3eb-fe3b1f07b821","Type":"ContainerDied","Data":"c80ddaf7a08420e54abfcaf3cd8189b197f8c5cee653e1aa2b0cb04f72b61270"}
Sep 09 20:31:02.123934 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:02.123900    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9nvcp" event={"ID":"9b2bd21a-9cad-4786-b3eb-fe3b1f07b821","Type":"ContainerStarted","Data":"b710236df183ba70273c91bb2d0e14a819591a1466c03e20df488dde9acfff1d"}
Sep 09 20:31:02.124464 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:02.123956    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9nvcp" event={"ID":"9b2bd21a-9cad-4786-b3eb-fe3b1f07b821","Type":"ContainerStarted","Data":"078477b5fd2a88c4cd927cef83c20233e544159364a4c38a2e4f6bc2ba88a74e"}
Sep 09 20:31:02.124464 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:02.124186    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9nvcp"
Sep 09 20:31:02.124464 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:02.124231    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9nvcp"
Sep 09 20:31:02.125679 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:02.125650    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9nvcp" podUID="9b2bd21a-9cad-4786-b3eb-fe3b1f07b821" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 09 20:31:02.143494 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:02.143451    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9nvcp" podStartSLOduration=14.784939133 podStartE2EDuration="19.143439954s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.358500821s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:30:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:31:02.142420166 +0000 UTC m=+2166.814439160" watchObservedRunningTime="2026-09-09 20:31:02.143439954 +0000 UTC m=+2166.815458950"
Sep 09 20:31:03.127732 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:03.127695    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9nvcp" podUID="9b2bd21a-9cad-4786-b3eb-fe3b1f07b821" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 09 20:31:04.113902 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:04.113875    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2pk9q"
Sep 09 20:31:04.114824 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:04.114802    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2pk9q"
Sep 09 20:31:08.133096 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:08.133059    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9nvcp"
Sep 09 20:31:08.133676 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:08.133652    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9nvcp" podUID="9b2bd21a-9cad-4786-b3eb-fe3b1f07b821" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 09 20:31:11.163027 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:11.162995    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2pk9q"]
Sep 09 20:31:11.163504 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:11.163268    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2pk9q" podUID="f67ebb18-149f-478f-a333-0e0d981cf237" containerName="kserve-container" containerID="cri-o://1b7176e1530e3f8b24712752258b0ed0187a3f74f832ff79a6c69d3637c80468" gracePeriod=30
Sep 09 20:31:11.163504 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:11.163328    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2pk9q" podUID="f67ebb18-149f-478f-a333-0e0d981cf237" containerName="kube-rbac-proxy" containerID="cri-o://01b8063d5c674a95819ded4ac05436f981563303860d09caa6e735b1428dbd97" gracePeriod=30
Sep 09 20:31:12.164624 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:12.164594    2523 generic.go:356] "Generic (PLEG): container finished" podID="f67ebb18-149f-478f-a333-0e0d981cf237" containerID="01b8063d5c674a95819ded4ac05436f981563303860d09caa6e735b1428dbd97" exitCode=2
Sep 09 20:31:12.164971 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:12.164637    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2pk9q" event={"ID":"f67ebb18-149f-478f-a333-0e0d981cf237","Type":"ContainerDied","Data":"01b8063d5c674a95819ded4ac05436f981563303860d09caa6e735b1428dbd97"}
Sep 09 20:31:13.296049 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:13.296024    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2pk9q"
Sep 09 20:31:13.380380 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:13.380356    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6z2sx\" (UniqueName: \"kubernetes.io/projected/f67ebb18-149f-478f-a333-0e0d981cf237-kube-api-access-6z2sx\") pod \"f67ebb18-149f-478f-a333-0e0d981cf237\" (UID: \"f67ebb18-149f-478f-a333-0e0d981cf237\") "
Sep 09 20:31:13.380506 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:13.380393    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f67ebb18-149f-478f-a333-0e0d981cf237-cabundle-cert\") pod \"f67ebb18-149f-478f-a333-0e0d981cf237\" (UID: \"f67ebb18-149f-478f-a333-0e0d981cf237\") "
Sep 09 20:31:13.380506 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:13.380437    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f67ebb18-149f-478f-a333-0e0d981cf237-proxy-tls\") pod \"f67ebb18-149f-478f-a333-0e0d981cf237\" (UID: \"f67ebb18-149f-478f-a333-0e0d981cf237\") "
Sep 09 20:31:13.380506 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:13.380477    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f67ebb18-149f-478f-a333-0e0d981cf237-kserve-provision-location\") pod \"f67ebb18-149f-478f-a333-0e0d981cf237\" (UID: \"f67ebb18-149f-478f-a333-0e0d981cf237\") "
Sep 09 20:31:13.380506 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:13.380494    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f67ebb18-149f-478f-a333-0e0d981cf237-kube-rbac-proxy-sar-config\") pod \"f67ebb18-149f-478f-a333-0e0d981cf237\" (UID: \"f67ebb18-149f-478f-a333-0e0d981cf237\") "
Sep 09 20:31:13.380873 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:13.380849    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f67ebb18-149f-478f-a333-0e0d981cf237-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f67ebb18-149f-478f-a333-0e0d981cf237" (UID: "f67ebb18-149f-478f-a333-0e0d981cf237"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:31:13.380955 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:13.380860    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f67ebb18-149f-478f-a333-0e0d981cf237-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f67ebb18-149f-478f-a333-0e0d981cf237" (UID: "f67ebb18-149f-478f-a333-0e0d981cf237"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:31:13.380955 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:13.380892    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f67ebb18-149f-478f-a333-0e0d981cf237-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f67ebb18-149f-478f-a333-0e0d981cf237" (UID: "f67ebb18-149f-478f-a333-0e0d981cf237"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:31:13.382522 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:13.382496    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f67ebb18-149f-478f-a333-0e0d981cf237-kube-api-access-6z2sx" (OuterVolumeSpecName: "kube-api-access-6z2sx") pod "f67ebb18-149f-478f-a333-0e0d981cf237" (UID: "f67ebb18-149f-478f-a333-0e0d981cf237"). InnerVolumeSpecName "kube-api-access-6z2sx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:31:13.382648 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:13.382627    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f67ebb18-149f-478f-a333-0e0d981cf237-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f67ebb18-149f-478f-a333-0e0d981cf237" (UID: "f67ebb18-149f-478f-a333-0e0d981cf237"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:31:13.481117 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:13.481059    2523 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f67ebb18-149f-478f-a333-0e0d981cf237-kserve-provision-location\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:31:13.481117 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:13.481080    2523 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f67ebb18-149f-478f-a333-0e0d981cf237-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:31:13.481117 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:13.481098    2523 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z2sx\" (UniqueName: \"kubernetes.io/projected/f67ebb18-149f-478f-a333-0e0d981cf237-kube-api-access-6z2sx\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:31:13.481117 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:13.481107    2523 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f67ebb18-149f-478f-a333-0e0d981cf237-cabundle-cert\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:31:13.481117 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:13.481116    2523 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f67ebb18-149f-478f-a333-0e0d981cf237-proxy-tls\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:31:14.178128 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:14.178094    2523 generic.go:356] "Generic (PLEG): container finished" podID="f67ebb18-149f-478f-a333-0e0d981cf237" containerID="1b7176e1530e3f8b24712752258b0ed0187a3f74f832ff79a6c69d3637c80468" exitCode=0
Sep 09 20:31:14.178334 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:14.178176    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2pk9q"
Sep 09 20:31:14.178334 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:14.178182    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2pk9q" event={"ID":"f67ebb18-149f-478f-a333-0e0d981cf237","Type":"ContainerDied","Data":"1b7176e1530e3f8b24712752258b0ed0187a3f74f832ff79a6c69d3637c80468"}
Sep 09 20:31:14.178334 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:14.178238    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2pk9q" event={"ID":"f67ebb18-149f-478f-a333-0e0d981cf237","Type":"ContainerDied","Data":"82ca3ddec9ebb7a06c1df6b094f896fa390cae533ad51fa3fb68760abe1f5e14"}
Sep 09 20:31:14.178334 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:14.178272    2523 scope.go:117] "RemoveContainer" containerID="01b8063d5c674a95819ded4ac05436f981563303860d09caa6e735b1428dbd97"
Sep 09 20:31:14.186503 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:14.186484    2523 scope.go:117] "RemoveContainer" containerID="1b7176e1530e3f8b24712752258b0ed0187a3f74f832ff79a6c69d3637c80468"
Sep 09 20:31:14.193350 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:14.193334    2523 scope.go:117] "RemoveContainer" containerID="a053344ecb3b2f7dfe024c3565f81191e5d1bee580b1a6ca4322d4677e64d179"
Sep 09 20:31:14.196571 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:14.196552    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2pk9q"]
Sep 09 20:31:14.199755 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:14.199737    2523 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2pk9q"]
Sep 09 20:31:14.200744 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:14.200725    2523 scope.go:117] "RemoveContainer" containerID="01b8063d5c674a95819ded4ac05436f981563303860d09caa6e735b1428dbd97"
Sep 09 20:31:14.201031 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:31:14.201013    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01b8063d5c674a95819ded4ac05436f981563303860d09caa6e735b1428dbd97\": container with ID starting with 01b8063d5c674a95819ded4ac05436f981563303860d09caa6e735b1428dbd97 not found: ID does not exist" containerID="01b8063d5c674a95819ded4ac05436f981563303860d09caa6e735b1428dbd97"
Sep 09 20:31:14.201082 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:14.201038    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01b8063d5c674a95819ded4ac05436f981563303860d09caa6e735b1428dbd97"} err="failed to get container status \"01b8063d5c674a95819ded4ac05436f981563303860d09caa6e735b1428dbd97\": rpc error: code = NotFound desc = could not find container \"01b8063d5c674a95819ded4ac05436f981563303860d09caa6e735b1428dbd97\": container with ID starting with 01b8063d5c674a95819ded4ac05436f981563303860d09caa6e735b1428dbd97 not found: ID does not exist"
Sep 09 20:31:14.201082 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:14.201055    2523 scope.go:117] "RemoveContainer" containerID="1b7176e1530e3f8b24712752258b0ed0187a3f74f832ff79a6c69d3637c80468"
Sep 09 20:31:14.201326 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:31:14.201309    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b7176e1530e3f8b24712752258b0ed0187a3f74f832ff79a6c69d3637c80468\": container with ID starting with 1b7176e1530e3f8b24712752258b0ed0187a3f74f832ff79a6c69d3637c80468 not found: ID does not exist" containerID="1b7176e1530e3f8b24712752258b0ed0187a3f74f832ff79a6c69d3637c80468"
Sep 09 20:31:14.201372 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:14.201332    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b7176e1530e3f8b24712752258b0ed0187a3f74f832ff79a6c69d3637c80468"} err="failed to get container status \"1b7176e1530e3f8b24712752258b0ed0187a3f74f832ff79a6c69d3637c80468\": rpc error: code = NotFound desc = could not find container \"1b7176e1530e3f8b24712752258b0ed0187a3f74f832ff79a6c69d3637c80468\": container with ID starting with 1b7176e1530e3f8b24712752258b0ed0187a3f74f832ff79a6c69d3637c80468 not found: ID does not exist"
Sep 09 20:31:14.201372 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:14.201346    2523 scope.go:117] "RemoveContainer" containerID="a053344ecb3b2f7dfe024c3565f81191e5d1bee580b1a6ca4322d4677e64d179"
Sep 09 20:31:14.201602 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:31:14.201586    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a053344ecb3b2f7dfe024c3565f81191e5d1bee580b1a6ca4322d4677e64d179\": container with ID starting with a053344ecb3b2f7dfe024c3565f81191e5d1bee580b1a6ca4322d4677e64d179 not found: ID does not exist" containerID="a053344ecb3b2f7dfe024c3565f81191e5d1bee580b1a6ca4322d4677e64d179"
Sep 09 20:31:14.201655 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:14.201610    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a053344ecb3b2f7dfe024c3565f81191e5d1bee580b1a6ca4322d4677e64d179"} err="failed to get container status \"a053344ecb3b2f7dfe024c3565f81191e5d1bee580b1a6ca4322d4677e64d179\": rpc error: code = NotFound desc = could not find container \"a053344ecb3b2f7dfe024c3565f81191e5d1bee580b1a6ca4322d4677e64d179\": container with ID starting with a053344ecb3b2f7dfe024c3565f81191e5d1bee580b1a6ca4322d4677e64d179 not found: ID does not exist"
Sep 09 20:31:15.807784 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:15.807758    2523 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f67ebb18-149f-478f-a333-0e0d981cf237" path="/var/lib/kubelet/pods/f67ebb18-149f-478f-a333-0e0d981cf237/volumes"
Sep 09 20:31:16.389643 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:16.389609    2523 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6nkv"]
Sep 09 20:31:16.389883 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:16.389871    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f67ebb18-149f-478f-a333-0e0d981cf237" containerName="kserve-container"
Sep 09 20:31:16.389927 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:16.389885    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="f67ebb18-149f-478f-a333-0e0d981cf237" containerName="kserve-container"
Sep 09 20:31:16.389927 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:16.389893    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f67ebb18-149f-478f-a333-0e0d981cf237" containerName="kube-rbac-proxy"
Sep 09 20:31:16.389927 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:16.389899    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="f67ebb18-149f-478f-a333-0e0d981cf237" containerName="kube-rbac-proxy"
Sep 09 20:31:16.389927 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:16.389914    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6faa194-d15e-456c-8ea8-d8d5fe28e31c" containerName="storage-initializer"
Sep 09 20:31:16.389927 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:16.389920    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6faa194-d15e-456c-8ea8-d8d5fe28e31c" containerName="storage-initializer"
Sep 09 20:31:16.390079 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:16.389932    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f67ebb18-149f-478f-a333-0e0d981cf237" containerName="storage-initializer"
Sep 09 20:31:16.390079 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:16.389937    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="f67ebb18-149f-478f-a333-0e0d981cf237" containerName="storage-initializer"
Sep 09 20:31:16.390079 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:16.389943    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6faa194-d15e-456c-8ea8-d8d5fe28e31c" containerName="kserve-container"
Sep 09 20:31:16.390079 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:16.389948    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6faa194-d15e-456c-8ea8-d8d5fe28e31c" containerName="kserve-container"
Sep 09 20:31:16.390079 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:16.389955    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6faa194-d15e-456c-8ea8-d8d5fe28e31c" containerName="kube-rbac-proxy"
Sep 09 20:31:16.390079 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:16.389960    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6faa194-d15e-456c-8ea8-d8d5fe28e31c" containerName="kube-rbac-proxy"
Sep 09 20:31:16.390079 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:16.389998    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6faa194-d15e-456c-8ea8-d8d5fe28e31c" containerName="kserve-container"
Sep 09 20:31:16.390079 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:16.390004    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6faa194-d15e-456c-8ea8-d8d5fe28e31c" containerName="kube-rbac-proxy"
Sep 09 20:31:16.390079 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:16.390010    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="f67ebb18-149f-478f-a333-0e0d981cf237" containerName="kserve-container"
Sep 09 20:31:16.390079 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:16.390015    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="f67ebb18-149f-478f-a333-0e0d981cf237" containerName="kube-rbac-proxy"
Sep 09 20:31:16.392805 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:16.392790    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6nkv"
Sep 09 20:31:16.395599 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:16.395575    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\""
Sep 09 20:31:16.395734 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:16.395635    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
Sep 09 20:31:16.404390 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:16.404367    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6nkv"]
Sep 09 20:31:16.503386 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:16.503361    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/522f62bf-c873-4531-9d67-c2b6c0de3e09-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-r6nkv\" (UID: \"522f62bf-c873-4531-9d67-c2b6c0de3e09\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6nkv"
Sep 09 20:31:16.503490 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:16.503390    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/522f62bf-c873-4531-9d67-c2b6c0de3e09-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-r6nkv\" (UID: \"522f62bf-c873-4531-9d67-c2b6c0de3e09\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6nkv"
Sep 09 20:31:16.503490 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:16.503409    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmcdw\" (UniqueName: \"kubernetes.io/projected/522f62bf-c873-4531-9d67-c2b6c0de3e09-kube-api-access-lmcdw\") pod \"isvc-xgboost-predictor-7566d65bf7-r6nkv\" (UID: \"522f62bf-c873-4531-9d67-c2b6c0de3e09\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6nkv"
Sep 09 20:31:16.503490 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:16.503475    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/522f62bf-c873-4531-9d67-c2b6c0de3e09-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-r6nkv\" (UID: \"522f62bf-c873-4531-9d67-c2b6c0de3e09\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6nkv"
Sep 09 20:31:16.503595 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:16.503521    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/522f62bf-c873-4531-9d67-c2b6c0de3e09-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-r6nkv\" (UID: \"522f62bf-c873-4531-9d67-c2b6c0de3e09\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6nkv"
Sep 09 20:31:16.603992 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:16.603956    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/522f62bf-c873-4531-9d67-c2b6c0de3e09-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-r6nkv\" (UID: \"522f62bf-c873-4531-9d67-c2b6c0de3e09\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6nkv"
Sep 09 20:31:16.604120 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:16.603999    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/522f62bf-c873-4531-9d67-c2b6c0de3e09-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-r6nkv\" (UID: \"522f62bf-c873-4531-9d67-c2b6c0de3e09\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6nkv"
Sep 09 20:31:16.604120 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:16.604023    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmcdw\" (UniqueName: \"kubernetes.io/projected/522f62bf-c873-4531-9d67-c2b6c0de3e09-kube-api-access-lmcdw\") pod \"isvc-xgboost-predictor-7566d65bf7-r6nkv\" (UID: \"522f62bf-c873-4531-9d67-c2b6c0de3e09\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6nkv"
Sep 09 20:31:16.604120 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:16.604068    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/522f62bf-c873-4531-9d67-c2b6c0de3e09-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-r6nkv\" (UID: \"522f62bf-c873-4531-9d67-c2b6c0de3e09\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6nkv"
Sep 09 20:31:16.604120 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:16.604110    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/522f62bf-c873-4531-9d67-c2b6c0de3e09-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-r6nkv\" (UID: \"522f62bf-c873-4531-9d67-c2b6c0de3e09\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6nkv"
Sep 09 20:31:16.604554 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:16.604531    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/522f62bf-c873-4531-9d67-c2b6c0de3e09-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-r6nkv\" (UID: \"522f62bf-c873-4531-9d67-c2b6c0de3e09\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6nkv"
Sep 09 20:31:16.604791 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:16.604736    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/522f62bf-c873-4531-9d67-c2b6c0de3e09-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-r6nkv\" (UID: \"522f62bf-c873-4531-9d67-c2b6c0de3e09\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6nkv"
Sep 09 20:31:16.604791 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:16.604781    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/522f62bf-c873-4531-9d67-c2b6c0de3e09-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-r6nkv\" (UID: \"522f62bf-c873-4531-9d67-c2b6c0de3e09\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6nkv"
Sep 09 20:31:16.606626 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:16.606604    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/522f62bf-c873-4531-9d67-c2b6c0de3e09-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-r6nkv\" (UID: \"522f62bf-c873-4531-9d67-c2b6c0de3e09\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6nkv"
Sep 09 20:31:16.612605 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:16.612583    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmcdw\" (UniqueName: \"kubernetes.io/projected/522f62bf-c873-4531-9d67-c2b6c0de3e09-kube-api-access-lmcdw\") pod \"isvc-xgboost-predictor-7566d65bf7-r6nkv\" (UID: \"522f62bf-c873-4531-9d67-c2b6c0de3e09\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6nkv"
Sep 09 20:31:16.703210 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:16.703144    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6nkv"
Sep 09 20:31:16.978875 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:16.978841    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6nkv"]
Sep 09 20:31:16.981169 ip-10-0-132-226 kubenswrapper[2523]: W0909 20:31:16.981123    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod522f62bf_c873_4531_9d67_c2b6c0de3e09.slice/crio-f4d93174bf0b937796fe2bcd45330b9f3c282f1d295c622647c6f27cb59f6ea6 WatchSource:0}: Error finding container f4d93174bf0b937796fe2bcd45330b9f3c282f1d295c622647c6f27cb59f6ea6: Status 404 returned error can't find the container with id f4d93174bf0b937796fe2bcd45330b9f3c282f1d295c622647c6f27cb59f6ea6
Sep 09 20:31:17.195420 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:17.195379    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6nkv" event={"ID":"522f62bf-c873-4531-9d67-c2b6c0de3e09","Type":"ContainerStarted","Data":"ebe73bf76987be17180270767f8fb6b43834eb587c778cfc0231b01d90c0abdd"}
Sep 09 20:31:17.195580 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:17.195470    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6nkv" event={"ID":"522f62bf-c873-4531-9d67-c2b6c0de3e09","Type":"ContainerStarted","Data":"f4d93174bf0b937796fe2bcd45330b9f3c282f1d295c622647c6f27cb59f6ea6"}
Sep 09 20:31:18.134561 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:18.134527    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9nvcp" podUID="9b2bd21a-9cad-4786-b3eb-fe3b1f07b821" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 09 20:31:21.210991 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:21.210959    2523 generic.go:356] "Generic (PLEG): container finished" podID="522f62bf-c873-4531-9d67-c2b6c0de3e09" containerID="ebe73bf76987be17180270767f8fb6b43834eb587c778cfc0231b01d90c0abdd" exitCode=0
Sep 09 20:31:21.211382 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:21.211038    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6nkv" event={"ID":"522f62bf-c873-4531-9d67-c2b6c0de3e09","Type":"ContainerDied","Data":"ebe73bf76987be17180270767f8fb6b43834eb587c778cfc0231b01d90c0abdd"}
Sep 09 20:31:28.134128 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:28.134087    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9nvcp" podUID="9b2bd21a-9cad-4786-b3eb-fe3b1f07b821" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 09 20:31:38.133774 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:38.133727    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9nvcp" podUID="9b2bd21a-9cad-4786-b3eb-fe3b1f07b821" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 09 20:31:41.306790 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:41.306703    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6nkv" event={"ID":"522f62bf-c873-4531-9d67-c2b6c0de3e09","Type":"ContainerStarted","Data":"521a137d71e323eee425100e9c47e86dde0c6d599fb4cbf8400a6447ab259621"}
Sep 09 20:31:41.306790 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:41.306764    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6nkv" event={"ID":"522f62bf-c873-4531-9d67-c2b6c0de3e09","Type":"ContainerStarted","Data":"f0a81139107ae586944f93a13804b2f1be56dfc49662f25a49732ce5c9556418"}
Sep 09 20:31:41.307215 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:41.307044    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6nkv"
Sep 09 20:31:41.325366 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:41.325324    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6nkv" podStartSLOduration=1.8524246359999998 podStartE2EDuration="25.325312765s" totalImagesPullingTime="19.767267024s" totalInitContainerRuntime="3.705621105s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:31:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:31:41.323959056 +0000 UTC m=+2205.995978053" watchObservedRunningTime="2026-09-09 20:31:41.325312765 +0000 UTC m=+2205.997331762"
Sep 09 20:31:42.310098 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:42.310066    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6nkv"
Sep 09 20:31:42.311066 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:42.311039    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6nkv" podUID="522f62bf-c873-4531-9d67-c2b6c0de3e09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 09 20:31:43.313967 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:43.313930    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6nkv" podUID="522f62bf-c873-4531-9d67-c2b6c0de3e09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 09 20:31:48.133628 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:48.133579    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9nvcp" podUID="9b2bd21a-9cad-4786-b3eb-fe3b1f07b821" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 09 20:31:48.318430 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:48.318406    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6nkv"
Sep 09 20:31:48.318892 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:48.318862    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6nkv" podUID="522f62bf-c873-4531-9d67-c2b6c0de3e09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 09 20:31:58.134222 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:58.134186    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9nvcp" podUID="9b2bd21a-9cad-4786-b3eb-fe3b1f07b821" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 09 20:31:58.319181 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:31:58.319127    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6nkv" podUID="522f62bf-c873-4531-9d67-c2b6c0de3e09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 09 20:32:08.133559 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:32:08.133519    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9nvcp" podUID="9b2bd21a-9cad-4786-b3eb-fe3b1f07b821" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 09 20:32:08.318949 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:32:08.318908    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6nkv" podUID="522f62bf-c873-4531-9d67-c2b6c0de3e09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 09 20:32:18.134377 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:32:18.134292    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9nvcp"
Sep 09 20:32:18.319328 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:32:18.319295    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6nkv" podUID="522f62bf-c873-4531-9d67-c2b6c0de3e09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 09 20:32:28.319749 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:32:28.319707    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6nkv" podUID="522f62bf-c873-4531-9d67-c2b6c0de3e09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 09 20:32:38.319171 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:32:38.319126    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6nkv"
Sep 09 20:32:46.437261 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:32:46.437223    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6nkv"]
Sep 09 20:32:46.437666 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:32:46.437455    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6nkv" podUID="522f62bf-c873-4531-9d67-c2b6c0de3e09" containerName="kserve-container" containerID="cri-o://f0a81139107ae586944f93a13804b2f1be56dfc49662f25a49732ce5c9556418" gracePeriod=30
Sep 09 20:32:46.437666 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:32:46.437574    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6nkv" podUID="522f62bf-c873-4531-9d67-c2b6c0de3e09" containerName="kube-rbac-proxy" containerID="cri-o://521a137d71e323eee425100e9c47e86dde0c6d599fb4cbf8400a6447ab259621" gracePeriod=30
Sep 09 20:32:46.555305 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:32:46.555144    2523 generic.go:356] "Generic (PLEG): container finished" podID="522f62bf-c873-4531-9d67-c2b6c0de3e09" containerID="521a137d71e323eee425100e9c47e86dde0c6d599fb4cbf8400a6447ab259621" exitCode=2
Sep 09 20:32:46.555305 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:32:46.555179    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6nkv" event={"ID":"522f62bf-c873-4531-9d67-c2b6c0de3e09","Type":"ContainerDied","Data":"521a137d71e323eee425100e9c47e86dde0c6d599fb4cbf8400a6447ab259621"}
Sep 09 20:32:48.314355 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:32:48.314314    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6nkv" podUID="522f62bf-c873-4531-9d67-c2b6c0de3e09" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": dial tcp 10.132.0.41:8643: connect: connection refused"
Sep 09 20:32:48.319653 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:32:48.319630    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6nkv" podUID="522f62bf-c873-4531-9d67-c2b6c0de3e09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 09 20:32:49.374860 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:32:49.374837    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6nkv"
Sep 09 20:32:49.471041 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:32:49.470958    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/522f62bf-c873-4531-9d67-c2b6c0de3e09-cabundle-cert\") pod \"522f62bf-c873-4531-9d67-c2b6c0de3e09\" (UID: \"522f62bf-c873-4531-9d67-c2b6c0de3e09\") "
Sep 09 20:32:49.471041 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:32:49.471001    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/522f62bf-c873-4531-9d67-c2b6c0de3e09-kube-rbac-proxy-sar-config\") pod \"522f62bf-c873-4531-9d67-c2b6c0de3e09\" (UID: \"522f62bf-c873-4531-9d67-c2b6c0de3e09\") "
Sep 09 20:32:49.471304 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:32:49.471044    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/522f62bf-c873-4531-9d67-c2b6c0de3e09-kserve-provision-location\") pod \"522f62bf-c873-4531-9d67-c2b6c0de3e09\" (UID: \"522f62bf-c873-4531-9d67-c2b6c0de3e09\") "
Sep 09 20:32:49.471304 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:32:49.471093    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/522f62bf-c873-4531-9d67-c2b6c0de3e09-proxy-tls\") pod \"522f62bf-c873-4531-9d67-c2b6c0de3e09\" (UID: \"522f62bf-c873-4531-9d67-c2b6c0de3e09\") "
Sep 09 20:32:49.471304 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:32:49.471119    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lmcdw\" (UniqueName: \"kubernetes.io/projected/522f62bf-c873-4531-9d67-c2b6c0de3e09-kube-api-access-lmcdw\") pod \"522f62bf-c873-4531-9d67-c2b6c0de3e09\" (UID: \"522f62bf-c873-4531-9d67-c2b6c0de3e09\") "
Sep 09 20:32:49.471429 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:32:49.471399    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/522f62bf-c873-4531-9d67-c2b6c0de3e09-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "522f62bf-c873-4531-9d67-c2b6c0de3e09" (UID: "522f62bf-c873-4531-9d67-c2b6c0de3e09"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:32:49.471481 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:32:49.471416    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/522f62bf-c873-4531-9d67-c2b6c0de3e09-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "522f62bf-c873-4531-9d67-c2b6c0de3e09" (UID: "522f62bf-c873-4531-9d67-c2b6c0de3e09"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:32:49.471481 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:32:49.471434    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/522f62bf-c873-4531-9d67-c2b6c0de3e09-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "522f62bf-c873-4531-9d67-c2b6c0de3e09" (UID: "522f62bf-c873-4531-9d67-c2b6c0de3e09"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:32:49.473382 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:32:49.473360    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/522f62bf-c873-4531-9d67-c2b6c0de3e09-kube-api-access-lmcdw" (OuterVolumeSpecName: "kube-api-access-lmcdw") pod "522f62bf-c873-4531-9d67-c2b6c0de3e09" (UID: "522f62bf-c873-4531-9d67-c2b6c0de3e09"). InnerVolumeSpecName "kube-api-access-lmcdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:32:49.473519 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:32:49.473500    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/522f62bf-c873-4531-9d67-c2b6c0de3e09-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "522f62bf-c873-4531-9d67-c2b6c0de3e09" (UID: "522f62bf-c873-4531-9d67-c2b6c0de3e09"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:32:49.567697 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:32:49.567663    2523 generic.go:356] "Generic (PLEG): container finished" podID="522f62bf-c873-4531-9d67-c2b6c0de3e09" containerID="f0a81139107ae586944f93a13804b2f1be56dfc49662f25a49732ce5c9556418" exitCode=0
Sep 09 20:32:49.567803 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:32:49.567716    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6nkv"
Sep 09 20:32:49.567803 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:32:49.567741    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6nkv" event={"ID":"522f62bf-c873-4531-9d67-c2b6c0de3e09","Type":"ContainerDied","Data":"f0a81139107ae586944f93a13804b2f1be56dfc49662f25a49732ce5c9556418"}
Sep 09 20:32:49.567890 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:32:49.567805    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6nkv" event={"ID":"522f62bf-c873-4531-9d67-c2b6c0de3e09","Type":"ContainerDied","Data":"f4d93174bf0b937796fe2bcd45330b9f3c282f1d295c622647c6f27cb59f6ea6"}
Sep 09 20:32:49.567890 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:32:49.567836    2523 scope.go:117] "RemoveContainer" containerID="521a137d71e323eee425100e9c47e86dde0c6d599fb4cbf8400a6447ab259621"
Sep 09 20:32:49.572039 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:32:49.572022    2523 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/522f62bf-c873-4531-9d67-c2b6c0de3e09-cabundle-cert\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:32:49.572175 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:32:49.572042    2523 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/522f62bf-c873-4531-9d67-c2b6c0de3e09-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:32:49.572175 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:32:49.572051    2523 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/522f62bf-c873-4531-9d67-c2b6c0de3e09-kserve-provision-location\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:32:49.572175 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:32:49.572060    2523 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/522f62bf-c873-4531-9d67-c2b6c0de3e09-proxy-tls\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:32:49.572175 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:32:49.572068    2523 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmcdw\" (UniqueName: \"kubernetes.io/projected/522f62bf-c873-4531-9d67-c2b6c0de3e09-kube-api-access-lmcdw\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:32:49.576672 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:32:49.576657    2523 scope.go:117] "RemoveContainer" containerID="f0a81139107ae586944f93a13804b2f1be56dfc49662f25a49732ce5c9556418"
Sep 09 20:32:49.583755 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:32:49.583737    2523 scope.go:117] "RemoveContainer" containerID="ebe73bf76987be17180270767f8fb6b43834eb587c778cfc0231b01d90c0abdd"
Sep 09 20:32:49.591049 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:32:49.591030    2523 scope.go:117] "RemoveContainer" containerID="521a137d71e323eee425100e9c47e86dde0c6d599fb4cbf8400a6447ab259621"
Sep 09 20:32:49.591163 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:32:49.591135    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6nkv"]
Sep 09 20:32:49.591372 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:32:49.591335    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"521a137d71e323eee425100e9c47e86dde0c6d599fb4cbf8400a6447ab259621\": container with ID starting with 521a137d71e323eee425100e9c47e86dde0c6d599fb4cbf8400a6447ab259621 not found: ID does not exist" containerID="521a137d71e323eee425100e9c47e86dde0c6d599fb4cbf8400a6447ab259621"
Sep 09 20:32:49.591415 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:32:49.591380    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"521a137d71e323eee425100e9c47e86dde0c6d599fb4cbf8400a6447ab259621"} err="failed to get container status \"521a137d71e323eee425100e9c47e86dde0c6d599fb4cbf8400a6447ab259621\": rpc error: code = NotFound desc = could not find container \"521a137d71e323eee425100e9c47e86dde0c6d599fb4cbf8400a6447ab259621\": container with ID starting with 521a137d71e323eee425100e9c47e86dde0c6d599fb4cbf8400a6447ab259621 not found: ID does not exist"
Sep 09 20:32:49.591415 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:32:49.591395    2523 scope.go:117] "RemoveContainer" containerID="f0a81139107ae586944f93a13804b2f1be56dfc49662f25a49732ce5c9556418"
Sep 09 20:32:49.591642 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:32:49.591618    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0a81139107ae586944f93a13804b2f1be56dfc49662f25a49732ce5c9556418\": container with ID starting with f0a81139107ae586944f93a13804b2f1be56dfc49662f25a49732ce5c9556418 not found: ID does not exist" containerID="f0a81139107ae586944f93a13804b2f1be56dfc49662f25a49732ce5c9556418"
Sep 09 20:32:49.591689 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:32:49.591648    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0a81139107ae586944f93a13804b2f1be56dfc49662f25a49732ce5c9556418"} err="failed to get container status \"f0a81139107ae586944f93a13804b2f1be56dfc49662f25a49732ce5c9556418\": rpc error: code = NotFound desc = could not find container \"f0a81139107ae586944f93a13804b2f1be56dfc49662f25a49732ce5c9556418\": container with ID starting with f0a81139107ae586944f93a13804b2f1be56dfc49662f25a49732ce5c9556418 not found: ID does not exist"
Sep 09 20:32:49.591689 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:32:49.591665    2523 scope.go:117] "RemoveContainer" containerID="ebe73bf76987be17180270767f8fb6b43834eb587c778cfc0231b01d90c0abdd"
Sep 09 20:32:49.591948 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:32:49.591912    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebe73bf76987be17180270767f8fb6b43834eb587c778cfc0231b01d90c0abdd\": container with ID starting with ebe73bf76987be17180270767f8fb6b43834eb587c778cfc0231b01d90c0abdd not found: ID does not exist" containerID="ebe73bf76987be17180270767f8fb6b43834eb587c778cfc0231b01d90c0abdd"
Sep 09 20:32:49.591995 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:32:49.591944    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebe73bf76987be17180270767f8fb6b43834eb587c778cfc0231b01d90c0abdd"} err="failed to get container status \"ebe73bf76987be17180270767f8fb6b43834eb587c778cfc0231b01d90c0abdd\": rpc error: code = NotFound desc = could not find container \"ebe73bf76987be17180270767f8fb6b43834eb587c778cfc0231b01d90c0abdd\": container with ID starting with ebe73bf76987be17180270767f8fb6b43834eb587c778cfc0231b01d90c0abdd not found: ID does not exist"
Sep 09 20:32:49.594636 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:32:49.594616    2523 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-r6nkv"]
Sep 09 20:32:49.807400 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:32:49.807347    2523 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="522f62bf-c873-4531-9d67-c2b6c0de3e09" path="/var/lib/kubelet/pods/522f62bf-c873-4531-9d67-c2b6c0de3e09/volumes"
Sep 09 20:33:53.789759 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:53.789686    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9nvcp"]
Sep 09 20:33:53.790287 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:53.789968    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9nvcp" podUID="9b2bd21a-9cad-4786-b3eb-fe3b1f07b821" containerName="kserve-container" containerID="cri-o://078477b5fd2a88c4cd927cef83c20233e544159364a4c38a2e4f6bc2ba88a74e" gracePeriod=30
Sep 09 20:33:53.790287 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:53.790043    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9nvcp" podUID="9b2bd21a-9cad-4786-b3eb-fe3b1f07b821" containerName="kube-rbac-proxy" containerID="cri-o://b710236df183ba70273c91bb2d0e14a819591a1466c03e20df488dde9acfff1d" gracePeriod=30
Sep 09 20:33:54.806797 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:54.806766    2523 generic.go:356] "Generic (PLEG): container finished" podID="9b2bd21a-9cad-4786-b3eb-fe3b1f07b821" containerID="b710236df183ba70273c91bb2d0e14a819591a1466c03e20df488dde9acfff1d" exitCode=2
Sep 09 20:33:54.807204 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:54.806807    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9nvcp" event={"ID":"9b2bd21a-9cad-4786-b3eb-fe3b1f07b821","Type":"ContainerDied","Data":"b710236df183ba70273c91bb2d0e14a819591a1466c03e20df488dde9acfff1d"}
Sep 09 20:33:57.326463 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:57.326440    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9nvcp"
Sep 09 20:33:57.418999 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:57.418930    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b2bd21a-9cad-4786-b3eb-fe3b1f07b821-kube-rbac-proxy-sar-config\") pod \"9b2bd21a-9cad-4786-b3eb-fe3b1f07b821\" (UID: \"9b2bd21a-9cad-4786-b3eb-fe3b1f07b821\") "
Sep 09 20:33:57.418999 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:57.418974    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b2bd21a-9cad-4786-b3eb-fe3b1f07b821-cabundle-cert\") pod \"9b2bd21a-9cad-4786-b3eb-fe3b1f07b821\" (UID: \"9b2bd21a-9cad-4786-b3eb-fe3b1f07b821\") "
Sep 09 20:33:57.419262 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:57.419025    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b2bd21a-9cad-4786-b3eb-fe3b1f07b821-proxy-tls\") pod \"9b2bd21a-9cad-4786-b3eb-fe3b1f07b821\" (UID: \"9b2bd21a-9cad-4786-b3eb-fe3b1f07b821\") "
Sep 09 20:33:57.419262 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:57.419046    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvg5l\" (UniqueName: \"kubernetes.io/projected/9b2bd21a-9cad-4786-b3eb-fe3b1f07b821-kube-api-access-xvg5l\") pod \"9b2bd21a-9cad-4786-b3eb-fe3b1f07b821\" (UID: \"9b2bd21a-9cad-4786-b3eb-fe3b1f07b821\") "
Sep 09 20:33:57.419262 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:57.419062    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b2bd21a-9cad-4786-b3eb-fe3b1f07b821-kserve-provision-location\") pod \"9b2bd21a-9cad-4786-b3eb-fe3b1f07b821\" (UID: \"9b2bd21a-9cad-4786-b3eb-fe3b1f07b821\") "
Sep 09 20:33:57.419431 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:57.419377    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b2bd21a-9cad-4786-b3eb-fe3b1f07b821-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9b2bd21a-9cad-4786-b3eb-fe3b1f07b821" (UID: "9b2bd21a-9cad-4786-b3eb-fe3b1f07b821"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:33:57.419431 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:57.419410    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b2bd21a-9cad-4786-b3eb-fe3b1f07b821-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9b2bd21a-9cad-4786-b3eb-fe3b1f07b821" (UID: "9b2bd21a-9cad-4786-b3eb-fe3b1f07b821"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:33:57.419510 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:57.419424    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b2bd21a-9cad-4786-b3eb-fe3b1f07b821-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9b2bd21a-9cad-4786-b3eb-fe3b1f07b821" (UID: "9b2bd21a-9cad-4786-b3eb-fe3b1f07b821"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:33:57.421070 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:57.421050    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b2bd21a-9cad-4786-b3eb-fe3b1f07b821-kube-api-access-xvg5l" (OuterVolumeSpecName: "kube-api-access-xvg5l") pod "9b2bd21a-9cad-4786-b3eb-fe3b1f07b821" (UID: "9b2bd21a-9cad-4786-b3eb-fe3b1f07b821"). InnerVolumeSpecName "kube-api-access-xvg5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:33:57.421214 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:57.421192    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b2bd21a-9cad-4786-b3eb-fe3b1f07b821-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9b2bd21a-9cad-4786-b3eb-fe3b1f07b821" (UID: "9b2bd21a-9cad-4786-b3eb-fe3b1f07b821"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:33:57.520035 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:57.520012    2523 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b2bd21a-9cad-4786-b3eb-fe3b1f07b821-proxy-tls\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:33:57.520035 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:57.520032    2523 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvg5l\" (UniqueName: \"kubernetes.io/projected/9b2bd21a-9cad-4786-b3eb-fe3b1f07b821-kube-api-access-xvg5l\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:33:57.520188 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:57.520042    2523 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b2bd21a-9cad-4786-b3eb-fe3b1f07b821-kserve-provision-location\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:33:57.520188 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:57.520050    2523 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b2bd21a-9cad-4786-b3eb-fe3b1f07b821-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:33:57.520188 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:57.520059    2523 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b2bd21a-9cad-4786-b3eb-fe3b1f07b821-cabundle-cert\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:33:57.820064 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:57.820039    2523 generic.go:356] "Generic (PLEG): container finished" podID="9b2bd21a-9cad-4786-b3eb-fe3b1f07b821" containerID="078477b5fd2a88c4cd927cef83c20233e544159364a4c38a2e4f6bc2ba88a74e" exitCode=0
Sep 09 20:33:57.820188 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:57.820101    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9nvcp"
Sep 09 20:33:57.820188 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:57.820102    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9nvcp" event={"ID":"9b2bd21a-9cad-4786-b3eb-fe3b1f07b821","Type":"ContainerDied","Data":"078477b5fd2a88c4cd927cef83c20233e544159364a4c38a2e4f6bc2ba88a74e"}
Sep 09 20:33:57.820285 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:57.820209    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9nvcp" event={"ID":"9b2bd21a-9cad-4786-b3eb-fe3b1f07b821","Type":"ContainerDied","Data":"0df10328dc4ee530f2b55b94d381a4335f0e794332c85b9d0d789856b7806aac"}
Sep 09 20:33:57.820285 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:57.820234    2523 scope.go:117] "RemoveContainer" containerID="b710236df183ba70273c91bb2d0e14a819591a1466c03e20df488dde9acfff1d"
Sep 09 20:33:57.828041 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:57.828023    2523 scope.go:117] "RemoveContainer" containerID="078477b5fd2a88c4cd927cef83c20233e544159364a4c38a2e4f6bc2ba88a74e"
Sep 09 20:33:57.834917 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:57.834898    2523 scope.go:117] "RemoveContainer" containerID="c80ddaf7a08420e54abfcaf3cd8189b197f8c5cee653e1aa2b0cb04f72b61270"
Sep 09 20:33:57.841666 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:57.841649    2523 scope.go:117] "RemoveContainer" containerID="b710236df183ba70273c91bb2d0e14a819591a1466c03e20df488dde9acfff1d"
Sep 09 20:33:57.841994 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:33:57.841963    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b710236df183ba70273c91bb2d0e14a819591a1466c03e20df488dde9acfff1d\": container with ID starting with b710236df183ba70273c91bb2d0e14a819591a1466c03e20df488dde9acfff1d not found: ID does not exist" containerID="b710236df183ba70273c91bb2d0e14a819591a1466c03e20df488dde9acfff1d"
Sep 09 20:33:57.842105 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:57.842000    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b710236df183ba70273c91bb2d0e14a819591a1466c03e20df488dde9acfff1d"} err="failed to get container status \"b710236df183ba70273c91bb2d0e14a819591a1466c03e20df488dde9acfff1d\": rpc error: code = NotFound desc = could not find container \"b710236df183ba70273c91bb2d0e14a819591a1466c03e20df488dde9acfff1d\": container with ID starting with b710236df183ba70273c91bb2d0e14a819591a1466c03e20df488dde9acfff1d not found: ID does not exist"
Sep 09 20:33:57.842105 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:57.842032    2523 scope.go:117] "RemoveContainer" containerID="078477b5fd2a88c4cd927cef83c20233e544159364a4c38a2e4f6bc2ba88a74e"
Sep 09 20:33:57.842439 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:33:57.842416    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"078477b5fd2a88c4cd927cef83c20233e544159364a4c38a2e4f6bc2ba88a74e\": container with ID starting with 078477b5fd2a88c4cd927cef83c20233e544159364a4c38a2e4f6bc2ba88a74e not found: ID does not exist" containerID="078477b5fd2a88c4cd927cef83c20233e544159364a4c38a2e4f6bc2ba88a74e"
Sep 09 20:33:57.842523 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:57.842448    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"078477b5fd2a88c4cd927cef83c20233e544159364a4c38a2e4f6bc2ba88a74e"} err="failed to get container status \"078477b5fd2a88c4cd927cef83c20233e544159364a4c38a2e4f6bc2ba88a74e\": rpc error: code = NotFound desc = could not find container \"078477b5fd2a88c4cd927cef83c20233e544159364a4c38a2e4f6bc2ba88a74e\": container with ID starting with 078477b5fd2a88c4cd927cef83c20233e544159364a4c38a2e4f6bc2ba88a74e not found: ID does not exist"
Sep 09 20:33:57.842523 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:57.842468    2523 scope.go:117] "RemoveContainer" containerID="c80ddaf7a08420e54abfcaf3cd8189b197f8c5cee653e1aa2b0cb04f72b61270"
Sep 09 20:33:57.842754 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:33:57.842736    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c80ddaf7a08420e54abfcaf3cd8189b197f8c5cee653e1aa2b0cb04f72b61270\": container with ID starting with c80ddaf7a08420e54abfcaf3cd8189b197f8c5cee653e1aa2b0cb04f72b61270 not found: ID does not exist" containerID="c80ddaf7a08420e54abfcaf3cd8189b197f8c5cee653e1aa2b0cb04f72b61270"
Sep 09 20:33:57.842817 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:57.842759    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c80ddaf7a08420e54abfcaf3cd8189b197f8c5cee653e1aa2b0cb04f72b61270"} err="failed to get container status \"c80ddaf7a08420e54abfcaf3cd8189b197f8c5cee653e1aa2b0cb04f72b61270\": rpc error: code = NotFound desc = could not find container \"c80ddaf7a08420e54abfcaf3cd8189b197f8c5cee653e1aa2b0cb04f72b61270\": container with ID starting with c80ddaf7a08420e54abfcaf3cd8189b197f8c5cee653e1aa2b0cb04f72b61270 not found: ID does not exist"
Sep 09 20:33:57.843458 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:57.843441    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9nvcp"]
Sep 09 20:33:57.850874 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:57.850852    2523 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9nvcp"]
Sep 09 20:33:59.096989 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.096961    2523 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-psxj5"]
Sep 09 20:33:59.097409 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.097238    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="522f62bf-c873-4531-9d67-c2b6c0de3e09" containerName="kube-rbac-proxy"
Sep 09 20:33:59.097409 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.097249    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="522f62bf-c873-4531-9d67-c2b6c0de3e09" containerName="kube-rbac-proxy"
Sep 09 20:33:59.097409 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.097264    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b2bd21a-9cad-4786-b3eb-fe3b1f07b821" containerName="kserve-container"
Sep 09 20:33:59.097409 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.097269    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b2bd21a-9cad-4786-b3eb-fe3b1f07b821" containerName="kserve-container"
Sep 09 20:33:59.097409 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.097278    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b2bd21a-9cad-4786-b3eb-fe3b1f07b821" containerName="kube-rbac-proxy"
Sep 09 20:33:59.097409 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.097283    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b2bd21a-9cad-4786-b3eb-fe3b1f07b821" containerName="kube-rbac-proxy"
Sep 09 20:33:59.097409 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.097290    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="522f62bf-c873-4531-9d67-c2b6c0de3e09" containerName="storage-initializer"
Sep 09 20:33:59.097409 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.097295    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="522f62bf-c873-4531-9d67-c2b6c0de3e09" containerName="storage-initializer"
Sep 09 20:33:59.097409 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.097304    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="522f62bf-c873-4531-9d67-c2b6c0de3e09" containerName="kserve-container"
Sep 09 20:33:59.097409 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.097308    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="522f62bf-c873-4531-9d67-c2b6c0de3e09" containerName="kserve-container"
Sep 09 20:33:59.097409 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.097320    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b2bd21a-9cad-4786-b3eb-fe3b1f07b821" containerName="storage-initializer"
Sep 09 20:33:59.097409 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.097325    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b2bd21a-9cad-4786-b3eb-fe3b1f07b821" containerName="storage-initializer"
Sep 09 20:33:59.097409 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.097367    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="522f62bf-c873-4531-9d67-c2b6c0de3e09" containerName="kserve-container"
Sep 09 20:33:59.097409 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.097378    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b2bd21a-9cad-4786-b3eb-fe3b1f07b821" containerName="kserve-container"
Sep 09 20:33:59.097409 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.097383    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b2bd21a-9cad-4786-b3eb-fe3b1f07b821" containerName="kube-rbac-proxy"
Sep 09 20:33:59.097409 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.097389    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="522f62bf-c873-4531-9d67-c2b6c0de3e09" containerName="kube-rbac-proxy"
Sep 09 20:33:59.100259 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.100242    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-psxj5"
Sep 09 20:33:59.106461 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.106423    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Sep 09 20:33:59.107084 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.107066    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-ks7mb\""
Sep 09 20:33:59.107212 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.107067    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 20:33:59.107532 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.107518    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 20:33:59.107845 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.107824    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 20:33:59.107959 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.107877    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 09 20:33:59.108363 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.108345    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:33:59.140978 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.140956    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-psxj5"]
Sep 09 20:33:59.179532 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.179508    2523 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-8v9xl"]
Sep 09 20:33:59.181976 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.181962    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-8v9xl"
Sep 09 20:33:59.189189 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.189145    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 09 20:33:59.198540 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.198515    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-8v9xl"]
Sep 09 20:33:59.231334 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.231307    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e7656fc-2108-410e-8a39-749ba37b9c2c-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-psxj5\" (UID: \"3e7656fc-2108-410e-8a39-749ba37b9c2c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-psxj5"
Sep 09 20:33:59.231428 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.231343    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmf8p\" (UniqueName: \"kubernetes.io/projected/3e7656fc-2108-410e-8a39-749ba37b9c2c-kube-api-access-tmf8p\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-psxj5\" (UID: \"3e7656fc-2108-410e-8a39-749ba37b9c2c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-psxj5"
Sep 09 20:33:59.231428 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.231369    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e7656fc-2108-410e-8a39-749ba37b9c2c-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-psxj5\" (UID: \"3e7656fc-2108-410e-8a39-749ba37b9c2c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-psxj5"
Sep 09 20:33:59.231428 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.231417    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e7656fc-2108-410e-8a39-749ba37b9c2c-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-psxj5\" (UID: \"3e7656fc-2108-410e-8a39-749ba37b9c2c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-psxj5"
Sep 09 20:33:59.231567 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.231442    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e7656fc-2108-410e-8a39-749ba37b9c2c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-psxj5\" (UID: \"3e7656fc-2108-410e-8a39-749ba37b9c2c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-psxj5"
Sep 09 20:33:59.332639 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.332615    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxd9r\" (UniqueName: \"kubernetes.io/projected/b08bb0ac-24c9-4bf4-baf2-309acdd96f49-kube-api-access-wxd9r\") pod \"isvc-canary-stop-predictor-5d5b899946-8v9xl\" (UID: \"b08bb0ac-24c9-4bf4-baf2-309acdd96f49\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-8v9xl"
Sep 09 20:33:59.332783 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.332650    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmf8p\" (UniqueName: \"kubernetes.io/projected/3e7656fc-2108-410e-8a39-749ba37b9c2c-kube-api-access-tmf8p\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-psxj5\" (UID: \"3e7656fc-2108-410e-8a39-749ba37b9c2c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-psxj5"
Sep 09 20:33:59.332783 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.332671    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b08bb0ac-24c9-4bf4-baf2-309acdd96f49-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5d5b899946-8v9xl\" (UID: \"b08bb0ac-24c9-4bf4-baf2-309acdd96f49\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-8v9xl"
Sep 09 20:33:59.332783 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.332729    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e7656fc-2108-410e-8a39-749ba37b9c2c-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-psxj5\" (UID: \"3e7656fc-2108-410e-8a39-749ba37b9c2c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-psxj5"
Sep 09 20:33:59.332891 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.332794    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b08bb0ac-24c9-4bf4-baf2-309acdd96f49-cabundle-cert\") pod \"isvc-canary-stop-predictor-5d5b899946-8v9xl\" (UID: \"b08bb0ac-24c9-4bf4-baf2-309acdd96f49\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-8v9xl"
Sep 09 20:33:59.332891 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.332856    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e7656fc-2108-410e-8a39-749ba37b9c2c-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-psxj5\" (UID: \"3e7656fc-2108-410e-8a39-749ba37b9c2c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-psxj5"
Sep 09 20:33:59.332891 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.332884    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b08bb0ac-24c9-4bf4-baf2-309acdd96f49-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5d5b899946-8v9xl\" (UID: \"b08bb0ac-24c9-4bf4-baf2-309acdd96f49\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-8v9xl"
Sep 09 20:33:59.332990 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.332922    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b08bb0ac-24c9-4bf4-baf2-309acdd96f49-proxy-tls\") pod \"isvc-canary-stop-predictor-5d5b899946-8v9xl\" (UID: \"b08bb0ac-24c9-4bf4-baf2-309acdd96f49\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-8v9xl"
Sep 09 20:33:59.333030 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.333013    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e7656fc-2108-410e-8a39-749ba37b9c2c-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-psxj5\" (UID: \"3e7656fc-2108-410e-8a39-749ba37b9c2c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-psxj5"
Sep 09 20:33:59.333068 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.333024    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e7656fc-2108-410e-8a39-749ba37b9c2c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-psxj5\" (UID: \"3e7656fc-2108-410e-8a39-749ba37b9c2c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-psxj5"
Sep 09 20:33:59.333127 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.333067    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e7656fc-2108-410e-8a39-749ba37b9c2c-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-psxj5\" (UID: \"3e7656fc-2108-410e-8a39-749ba37b9c2c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-psxj5"
Sep 09 20:33:59.333499 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.333481    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e7656fc-2108-410e-8a39-749ba37b9c2c-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-psxj5\" (UID: \"3e7656fc-2108-410e-8a39-749ba37b9c2c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-psxj5"
Sep 09 20:33:59.333556 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.333544    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e7656fc-2108-410e-8a39-749ba37b9c2c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-psxj5\" (UID: \"3e7656fc-2108-410e-8a39-749ba37b9c2c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-psxj5"
Sep 09 20:33:59.335495 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.335480    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e7656fc-2108-410e-8a39-749ba37b9c2c-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-psxj5\" (UID: \"3e7656fc-2108-410e-8a39-749ba37b9c2c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-psxj5"
Sep 09 20:33:59.346831 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.346813    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmf8p\" (UniqueName: \"kubernetes.io/projected/3e7656fc-2108-410e-8a39-749ba37b9c2c-kube-api-access-tmf8p\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-psxj5\" (UID: \"3e7656fc-2108-410e-8a39-749ba37b9c2c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-psxj5"
Sep 09 20:33:59.409889 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.409838    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-psxj5"
Sep 09 20:33:59.433394 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.433373    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxd9r\" (UniqueName: \"kubernetes.io/projected/b08bb0ac-24c9-4bf4-baf2-309acdd96f49-kube-api-access-wxd9r\") pod \"isvc-canary-stop-predictor-5d5b899946-8v9xl\" (UID: \"b08bb0ac-24c9-4bf4-baf2-309acdd96f49\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-8v9xl"
Sep 09 20:33:59.433498 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.433405    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b08bb0ac-24c9-4bf4-baf2-309acdd96f49-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5d5b899946-8v9xl\" (UID: \"b08bb0ac-24c9-4bf4-baf2-309acdd96f49\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-8v9xl"
Sep 09 20:33:59.433498 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.433440    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b08bb0ac-24c9-4bf4-baf2-309acdd96f49-cabundle-cert\") pod \"isvc-canary-stop-predictor-5d5b899946-8v9xl\" (UID: \"b08bb0ac-24c9-4bf4-baf2-309acdd96f49\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-8v9xl"
Sep 09 20:33:59.433498 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.433479    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b08bb0ac-24c9-4bf4-baf2-309acdd96f49-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5d5b899946-8v9xl\" (UID: \"b08bb0ac-24c9-4bf4-baf2-309acdd96f49\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-8v9xl"
Sep 09 20:33:59.433662 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.433511    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b08bb0ac-24c9-4bf4-baf2-309acdd96f49-proxy-tls\") pod \"isvc-canary-stop-predictor-5d5b899946-8v9xl\" (UID: \"b08bb0ac-24c9-4bf4-baf2-309acdd96f49\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-8v9xl"
Sep 09 20:33:59.433824 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.433796    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b08bb0ac-24c9-4bf4-baf2-309acdd96f49-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5d5b899946-8v9xl\" (UID: \"b08bb0ac-24c9-4bf4-baf2-309acdd96f49\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-8v9xl"
Sep 09 20:33:59.434131 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.434111    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b08bb0ac-24c9-4bf4-baf2-309acdd96f49-cabundle-cert\") pod \"isvc-canary-stop-predictor-5d5b899946-8v9xl\" (UID: \"b08bb0ac-24c9-4bf4-baf2-309acdd96f49\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-8v9xl"
Sep 09 20:33:59.434283 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.434255    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b08bb0ac-24c9-4bf4-baf2-309acdd96f49-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5d5b899946-8v9xl\" (UID: \"b08bb0ac-24c9-4bf4-baf2-309acdd96f49\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-8v9xl"
Sep 09 20:33:59.436002 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.435976    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b08bb0ac-24c9-4bf4-baf2-309acdd96f49-proxy-tls\") pod \"isvc-canary-stop-predictor-5d5b899946-8v9xl\" (UID: \"b08bb0ac-24c9-4bf4-baf2-309acdd96f49\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-8v9xl"
Sep 09 20:33:59.449483 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.449459    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxd9r\" (UniqueName: \"kubernetes.io/projected/b08bb0ac-24c9-4bf4-baf2-309acdd96f49-kube-api-access-wxd9r\") pod \"isvc-canary-stop-predictor-5d5b899946-8v9xl\" (UID: \"b08bb0ac-24c9-4bf4-baf2-309acdd96f49\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-8v9xl"
Sep 09 20:33:59.490800 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.490776    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-8v9xl"
Sep 09 20:33:59.694707 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.694637    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-psxj5"]
Sep 09 20:33:59.697738 ip-10-0-132-226 kubenswrapper[2523]: W0909 20:33:59.697715    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e7656fc_2108_410e_8a39_749ba37b9c2c.slice/crio-12a81d737342ea14f66646b9016b6a0ed982c183de8ac3490b57661bbe730c41 WatchSource:0}: Error finding container 12a81d737342ea14f66646b9016b6a0ed982c183de8ac3490b57661bbe730c41: Status 404 returned error can't find the container with id 12a81d737342ea14f66646b9016b6a0ed982c183de8ac3490b57661bbe730c41
Sep 09 20:33:59.774632 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.774554    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-8v9xl"]
Sep 09 20:33:59.776405 ip-10-0-132-226 kubenswrapper[2523]: W0909 20:33:59.776387    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb08bb0ac_24c9_4bf4_baf2_309acdd96f49.slice/crio-47b1b125d4699bc1160cce82d8ee50626f3ddf77e63824236f37ba7766d4d585 WatchSource:0}: Error finding container 47b1b125d4699bc1160cce82d8ee50626f3ddf77e63824236f37ba7766d4d585: Status 404 returned error can't find the container with id 47b1b125d4699bc1160cce82d8ee50626f3ddf77e63824236f37ba7766d4d585
Sep 09 20:33:59.808005 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.807979    2523 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b2bd21a-9cad-4786-b3eb-fe3b1f07b821" path="/var/lib/kubelet/pods/9b2bd21a-9cad-4786-b3eb-fe3b1f07b821/volumes"
Sep 09 20:33:59.830093 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.830067    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-8v9xl" event={"ID":"b08bb0ac-24c9-4bf4-baf2-309acdd96f49","Type":"ContainerStarted","Data":"47b1b125d4699bc1160cce82d8ee50626f3ddf77e63824236f37ba7766d4d585"}
Sep 09 20:33:59.832065 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.832037    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-psxj5" event={"ID":"3e7656fc-2108-410e-8a39-749ba37b9c2c","Type":"ContainerStarted","Data":"4eeef90533b9daef8938c1fa173dfc26f46d693bfcfef23a8335ca0872f2a662"}
Sep 09 20:33:59.832142 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:33:59.832101    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-psxj5" event={"ID":"3e7656fc-2108-410e-8a39-749ba37b9c2c","Type":"ContainerStarted","Data":"12a81d737342ea14f66646b9016b6a0ed982c183de8ac3490b57661bbe730c41"}
Sep 09 20:34:00.837088 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:34:00.837049    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-8v9xl" event={"ID":"b08bb0ac-24c9-4bf4-baf2-309acdd96f49","Type":"ContainerStarted","Data":"308f88978915b8e8c56599ddd2adf09ca00ba65ddf85a2cab99e68af1e792839"}
Sep 09 20:34:03.849301 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:34:03.849269    2523 generic.go:356] "Generic (PLEG): container finished" podID="b08bb0ac-24c9-4bf4-baf2-309acdd96f49" containerID="308f88978915b8e8c56599ddd2adf09ca00ba65ddf85a2cab99e68af1e792839" exitCode=0
Sep 09 20:34:03.849685 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:34:03.849351    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-8v9xl" event={"ID":"b08bb0ac-24c9-4bf4-baf2-309acdd96f49","Type":"ContainerDied","Data":"308f88978915b8e8c56599ddd2adf09ca00ba65ddf85a2cab99e68af1e792839"}
Sep 09 20:34:03.850967 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:34:03.850908    2523 generic.go:356] "Generic (PLEG): container finished" podID="3e7656fc-2108-410e-8a39-749ba37b9c2c" containerID="4eeef90533b9daef8938c1fa173dfc26f46d693bfcfef23a8335ca0872f2a662" exitCode=0
Sep 09 20:34:03.851042 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:34:03.850983    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-psxj5" event={"ID":"3e7656fc-2108-410e-8a39-749ba37b9c2c","Type":"ContainerDied","Data":"4eeef90533b9daef8938c1fa173dfc26f46d693bfcfef23a8335ca0872f2a662"}
Sep 09 20:34:04.858379 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:34:04.858340    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-8v9xl" event={"ID":"b08bb0ac-24c9-4bf4-baf2-309acdd96f49","Type":"ContainerStarted","Data":"760069d5652d44039cb2ae64b9bbe86fe054609429d49c8ccd1901bbf748a120"}
Sep 09 20:34:04.858817 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:34:04.858418    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-8v9xl" event={"ID":"b08bb0ac-24c9-4bf4-baf2-309acdd96f49","Type":"ContainerStarted","Data":"2b8a2d5303c51a946db6e9324d227cbcac91924149049f31af028a15953be792"}
Sep 09 20:34:04.858817 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:34:04.858747    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-8v9xl"
Sep 09 20:34:04.860433 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:34:04.860412    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-psxj5" event={"ID":"3e7656fc-2108-410e-8a39-749ba37b9c2c","Type":"ContainerStarted","Data":"b1070d14e30b2426249b5f5bc6ace58cdcaa9dadd2c16f932fd3a4c547d27449"}
Sep 09 20:34:04.860546 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:34:04.860457    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-psxj5" event={"ID":"3e7656fc-2108-410e-8a39-749ba37b9c2c","Type":"ContainerStarted","Data":"d353f0d47d0be2c56bb7663c0f72ac049106acee9cad28cb6f1bf4159abeee2a"}
Sep 09 20:34:04.860763 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:34:04.860748    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-psxj5"
Sep 09 20:34:04.860826 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:34:04.860799    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-psxj5"
Sep 09 20:34:04.862194 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:34:04.862170    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-psxj5" podUID="3e7656fc-2108-410e-8a39-749ba37b9c2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 09 20:34:04.879260 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:34:04.879219    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-8v9xl" podStartSLOduration=2.141837264 podStartE2EDuration="5.879205685s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.737368421s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:33:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:34:04.877720249 +0000 UTC m=+2349.549739279" watchObservedRunningTime="2026-09-09 20:34:04.879205685 +0000 UTC m=+2349.551224682"
Sep 09 20:34:04.896910 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:34:04.896870    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-psxj5" podStartSLOduration=2.3225441350000002 podStartE2EDuration="5.896858555s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.57431442s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:33:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:34:04.895996789 +0000 UTC m=+2349.568015785" watchObservedRunningTime="2026-09-09 20:34:04.896858555 +0000 UTC m=+2349.568877552"
Sep 09 20:34:05.864476 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:34:05.864434    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-psxj5" podUID="3e7656fc-2108-410e-8a39-749ba37b9c2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 09 20:34:05.864924 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:34:05.864533    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-8v9xl"
Sep 09 20:34:05.865977 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:34:05.865954    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-8v9xl" podUID="b08bb0ac-24c9-4bf4-baf2-309acdd96f49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 09 20:34:06.868432 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:34:06.868393    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-8v9xl" podUID="b08bb0ac-24c9-4bf4-baf2-309acdd96f49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 09 20:34:10.868691 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:34:10.868664    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-psxj5"
Sep 09 20:34:10.869104 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:34:10.869065    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-psxj5" podUID="3e7656fc-2108-410e-8a39-749ba37b9c2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 09 20:34:11.873518 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:34:11.873487    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-8v9xl"
Sep 09 20:34:11.873987 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:34:11.873962    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-8v9xl" podUID="b08bb0ac-24c9-4bf4-baf2-309acdd96f49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 09 20:34:20.869290 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:34:20.869242    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-psxj5" podUID="3e7656fc-2108-410e-8a39-749ba37b9c2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 09 20:34:21.874542 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:34:21.874501    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-8v9xl" podUID="b08bb0ac-24c9-4bf4-baf2-309acdd96f49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 09 20:34:30.869067 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:34:30.869028    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-psxj5" podUID="3e7656fc-2108-410e-8a39-749ba37b9c2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 09 20:34:31.874609 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:34:31.874574    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-8v9xl" podUID="b08bb0ac-24c9-4bf4-baf2-309acdd96f49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 09 20:34:40.869688 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:34:40.869650    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-psxj5" podUID="3e7656fc-2108-410e-8a39-749ba37b9c2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 09 20:34:41.874596 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:34:41.874557    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-8v9xl" podUID="b08bb0ac-24c9-4bf4-baf2-309acdd96f49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 09 20:34:50.869200 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:34:50.869129    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-psxj5" podUID="3e7656fc-2108-410e-8a39-749ba37b9c2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 09 20:34:51.874707 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:34:51.874669    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-8v9xl" podUID="b08bb0ac-24c9-4bf4-baf2-309acdd96f49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 09 20:35:00.869065 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:00.869024    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-psxj5" podUID="3e7656fc-2108-410e-8a39-749ba37b9c2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 09 20:35:01.874291 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:01.874247    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-8v9xl" podUID="b08bb0ac-24c9-4bf4-baf2-309acdd96f49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 09 20:35:10.870043 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:10.870000    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-psxj5" podUID="3e7656fc-2108-410e-8a39-749ba37b9c2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 09 20:35:11.874837 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:11.874803    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-8v9xl"
Sep 09 20:35:20.870426 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:20.870395    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-psxj5"
Sep 09 20:35:29.011713 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:29.011679    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-psxj5"]
Sep 09 20:35:29.012210 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:29.011905    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-psxj5" podUID="3e7656fc-2108-410e-8a39-749ba37b9c2c" containerName="kserve-container" containerID="cri-o://d353f0d47d0be2c56bb7663c0f72ac049106acee9cad28cb6f1bf4159abeee2a" gracePeriod=30
Sep 09 20:35:29.012210 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:29.011950    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-psxj5" podUID="3e7656fc-2108-410e-8a39-749ba37b9c2c" containerName="kube-rbac-proxy" containerID="cri-o://b1070d14e30b2426249b5f5bc6ace58cdcaa9dadd2c16f932fd3a4c547d27449" gracePeriod=30
Sep 09 20:35:29.052056 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:29.052032    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-8v9xl"]
Sep 09 20:35:29.052341 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:29.052315    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-8v9xl" podUID="b08bb0ac-24c9-4bf4-baf2-309acdd96f49" containerName="kserve-container" containerID="cri-o://2b8a2d5303c51a946db6e9324d227cbcac91924149049f31af028a15953be792" gracePeriod=30
Sep 09 20:35:29.052467 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:29.052330    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-8v9xl" podUID="b08bb0ac-24c9-4bf4-baf2-309acdd96f49" containerName="kube-rbac-proxy" containerID="cri-o://760069d5652d44039cb2ae64b9bbe86fe054609429d49c8ccd1901bbf748a120" gracePeriod=30
Sep 09 20:35:29.181817 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:29.181792    2523 generic.go:356] "Generic (PLEG): container finished" podID="b08bb0ac-24c9-4bf4-baf2-309acdd96f49" containerID="760069d5652d44039cb2ae64b9bbe86fe054609429d49c8ccd1901bbf748a120" exitCode=2
Sep 09 20:35:29.181942 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:29.181868    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-8v9xl" event={"ID":"b08bb0ac-24c9-4bf4-baf2-309acdd96f49","Type":"ContainerDied","Data":"760069d5652d44039cb2ae64b9bbe86fe054609429d49c8ccd1901bbf748a120"}
Sep 09 20:35:29.184184 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:29.184138    2523 generic.go:356] "Generic (PLEG): container finished" podID="3e7656fc-2108-410e-8a39-749ba37b9c2c" containerID="b1070d14e30b2426249b5f5bc6ace58cdcaa9dadd2c16f932fd3a4c547d27449" exitCode=2
Sep 09 20:35:29.184272 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:29.184176    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-psxj5" event={"ID":"3e7656fc-2108-410e-8a39-749ba37b9c2c","Type":"ContainerDied","Data":"b1070d14e30b2426249b5f5bc6ace58cdcaa9dadd2c16f932fd3a4c547d27449"}
Sep 09 20:35:30.864571 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:30.864533    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-psxj5" podUID="3e7656fc-2108-410e-8a39-749ba37b9c2c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Sep 09 20:35:30.869062 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:30.869039    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-psxj5" podUID="3e7656fc-2108-410e-8a39-749ba37b9c2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 09 20:35:31.869626 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:31.869582    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-8v9xl" podUID="b08bb0ac-24c9-4bf4-baf2-309acdd96f49" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused"
Sep 09 20:35:31.873939 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:31.873915    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-8v9xl" podUID="b08bb0ac-24c9-4bf4-baf2-309acdd96f49" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 09 20:35:33.133666 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:33.133640    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-8v9xl"
Sep 09 20:35:33.151483 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:33.151458    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b08bb0ac-24c9-4bf4-baf2-309acdd96f49-proxy-tls\") pod \"b08bb0ac-24c9-4bf4-baf2-309acdd96f49\" (UID: \"b08bb0ac-24c9-4bf4-baf2-309acdd96f49\") "
Sep 09 20:35:33.151631 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:33.151541    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b08bb0ac-24c9-4bf4-baf2-309acdd96f49-kserve-provision-location\") pod \"b08bb0ac-24c9-4bf4-baf2-309acdd96f49\" (UID: \"b08bb0ac-24c9-4bf4-baf2-309acdd96f49\") "
Sep 09 20:35:33.151703 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:33.151633    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b08bb0ac-24c9-4bf4-baf2-309acdd96f49-kube-rbac-proxy-sar-config\") pod \"b08bb0ac-24c9-4bf4-baf2-309acdd96f49\" (UID: \"b08bb0ac-24c9-4bf4-baf2-309acdd96f49\") "
Sep 09 20:35:33.151703 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:33.151685    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b08bb0ac-24c9-4bf4-baf2-309acdd96f49-cabundle-cert\") pod \"b08bb0ac-24c9-4bf4-baf2-309acdd96f49\" (UID: \"b08bb0ac-24c9-4bf4-baf2-309acdd96f49\") "
Sep 09 20:35:33.151844 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:33.151717    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxd9r\" (UniqueName: \"kubernetes.io/projected/b08bb0ac-24c9-4bf4-baf2-309acdd96f49-kube-api-access-wxd9r\") pod \"b08bb0ac-24c9-4bf4-baf2-309acdd96f49\" (UID: \"b08bb0ac-24c9-4bf4-baf2-309acdd96f49\") "
Sep 09 20:35:33.151844 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:33.151815    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b08bb0ac-24c9-4bf4-baf2-309acdd96f49-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b08bb0ac-24c9-4bf4-baf2-309acdd96f49" (UID: "b08bb0ac-24c9-4bf4-baf2-309acdd96f49"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:35:33.151959 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:33.151942    2523 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b08bb0ac-24c9-4bf4-baf2-309acdd96f49-kserve-provision-location\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:35:33.152031 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:33.152005    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b08bb0ac-24c9-4bf4-baf2-309acdd96f49-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b08bb0ac-24c9-4bf4-baf2-309acdd96f49" (UID: "b08bb0ac-24c9-4bf4-baf2-309acdd96f49"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:35:33.152138 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:33.152114    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b08bb0ac-24c9-4bf4-baf2-309acdd96f49-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b08bb0ac-24c9-4bf4-baf2-309acdd96f49" (UID: "b08bb0ac-24c9-4bf4-baf2-309acdd96f49"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:35:33.153762 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:33.153733    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b08bb0ac-24c9-4bf4-baf2-309acdd96f49-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b08bb0ac-24c9-4bf4-baf2-309acdd96f49" (UID: "b08bb0ac-24c9-4bf4-baf2-309acdd96f49"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:35:33.154049 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:33.154032    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b08bb0ac-24c9-4bf4-baf2-309acdd96f49-kube-api-access-wxd9r" (OuterVolumeSpecName: "kube-api-access-wxd9r") pod "b08bb0ac-24c9-4bf4-baf2-309acdd96f49" (UID: "b08bb0ac-24c9-4bf4-baf2-309acdd96f49"). InnerVolumeSpecName "kube-api-access-wxd9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:35:33.203595 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:33.203563    2523 generic.go:356] "Generic (PLEG): container finished" podID="b08bb0ac-24c9-4bf4-baf2-309acdd96f49" containerID="2b8a2d5303c51a946db6e9324d227cbcac91924149049f31af028a15953be792" exitCode=0
Sep 09 20:35:33.203720 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:33.203614    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-8v9xl"
Sep 09 20:35:33.203720 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:33.203644    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-8v9xl" event={"ID":"b08bb0ac-24c9-4bf4-baf2-309acdd96f49","Type":"ContainerDied","Data":"2b8a2d5303c51a946db6e9324d227cbcac91924149049f31af028a15953be792"}
Sep 09 20:35:33.203720 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:33.203715    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-8v9xl" event={"ID":"b08bb0ac-24c9-4bf4-baf2-309acdd96f49","Type":"ContainerDied","Data":"47b1b125d4699bc1160cce82d8ee50626f3ddf77e63824236f37ba7766d4d585"}
Sep 09 20:35:33.203879 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:33.203755    2523 scope.go:117] "RemoveContainer" containerID="760069d5652d44039cb2ae64b9bbe86fe054609429d49c8ccd1901bbf748a120"
Sep 09 20:35:33.207326 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:33.207140    2523 generic.go:356] "Generic (PLEG): container finished" podID="3e7656fc-2108-410e-8a39-749ba37b9c2c" containerID="d353f0d47d0be2c56bb7663c0f72ac049106acee9cad28cb6f1bf4159abeee2a" exitCode=0
Sep 09 20:35:33.207326 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:33.207266    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-psxj5" event={"ID":"3e7656fc-2108-410e-8a39-749ba37b9c2c","Type":"ContainerDied","Data":"d353f0d47d0be2c56bb7663c0f72ac049106acee9cad28cb6f1bf4159abeee2a"}
Sep 09 20:35:33.214494 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:33.214475    2523 scope.go:117] "RemoveContainer" containerID="2b8a2d5303c51a946db6e9324d227cbcac91924149049f31af028a15953be792"
Sep 09 20:35:33.222329 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:33.222309    2523 scope.go:117] "RemoveContainer" containerID="308f88978915b8e8c56599ddd2adf09ca00ba65ddf85a2cab99e68af1e792839"
Sep 09 20:35:33.229879 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:33.229861    2523 scope.go:117] "RemoveContainer" containerID="760069d5652d44039cb2ae64b9bbe86fe054609429d49c8ccd1901bbf748a120"
Sep 09 20:35:33.230170 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:35:33.230133    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"760069d5652d44039cb2ae64b9bbe86fe054609429d49c8ccd1901bbf748a120\": container with ID starting with 760069d5652d44039cb2ae64b9bbe86fe054609429d49c8ccd1901bbf748a120 not found: ID does not exist" containerID="760069d5652d44039cb2ae64b9bbe86fe054609429d49c8ccd1901bbf748a120"
Sep 09 20:35:33.230286 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:33.230176    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"760069d5652d44039cb2ae64b9bbe86fe054609429d49c8ccd1901bbf748a120"} err="failed to get container status \"760069d5652d44039cb2ae64b9bbe86fe054609429d49c8ccd1901bbf748a120\": rpc error: code = NotFound desc = could not find container \"760069d5652d44039cb2ae64b9bbe86fe054609429d49c8ccd1901bbf748a120\": container with ID starting with 760069d5652d44039cb2ae64b9bbe86fe054609429d49c8ccd1901bbf748a120 not found: ID does not exist"
Sep 09 20:35:33.230286 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:33.230198    2523 scope.go:117] "RemoveContainer" containerID="2b8a2d5303c51a946db6e9324d227cbcac91924149049f31af028a15953be792"
Sep 09 20:35:33.230513 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:35:33.230486    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b8a2d5303c51a946db6e9324d227cbcac91924149049f31af028a15953be792\": container with ID starting with 2b8a2d5303c51a946db6e9324d227cbcac91924149049f31af028a15953be792 not found: ID does not exist" containerID="2b8a2d5303c51a946db6e9324d227cbcac91924149049f31af028a15953be792"
Sep 09 20:35:33.230591 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:33.230519    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b8a2d5303c51a946db6e9324d227cbcac91924149049f31af028a15953be792"} err="failed to get container status \"2b8a2d5303c51a946db6e9324d227cbcac91924149049f31af028a15953be792\": rpc error: code = NotFound desc = could not find container \"2b8a2d5303c51a946db6e9324d227cbcac91924149049f31af028a15953be792\": container with ID starting with 2b8a2d5303c51a946db6e9324d227cbcac91924149049f31af028a15953be792 not found: ID does not exist"
Sep 09 20:35:33.230591 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:33.230538    2523 scope.go:117] "RemoveContainer" containerID="308f88978915b8e8c56599ddd2adf09ca00ba65ddf85a2cab99e68af1e792839"
Sep 09 20:35:33.230811 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:35:33.230794    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"308f88978915b8e8c56599ddd2adf09ca00ba65ddf85a2cab99e68af1e792839\": container with ID starting with 308f88978915b8e8c56599ddd2adf09ca00ba65ddf85a2cab99e68af1e792839 not found: ID does not exist" containerID="308f88978915b8e8c56599ddd2adf09ca00ba65ddf85a2cab99e68af1e792839"
Sep 09 20:35:33.230872 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:33.230818    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"308f88978915b8e8c56599ddd2adf09ca00ba65ddf85a2cab99e68af1e792839"} err="failed to get container status \"308f88978915b8e8c56599ddd2adf09ca00ba65ddf85a2cab99e68af1e792839\": rpc error: code = NotFound desc = could not find container \"308f88978915b8e8c56599ddd2adf09ca00ba65ddf85a2cab99e68af1e792839\": container with ID starting with 308f88978915b8e8c56599ddd2adf09ca00ba65ddf85a2cab99e68af1e792839 not found: ID does not exist"
Sep 09 20:35:33.235753 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:33.235732    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-8v9xl"]
Sep 09 20:35:33.244568 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:33.244549    2523 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-8v9xl"]
Sep 09 20:35:33.252599 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:33.252579    2523 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b08bb0ac-24c9-4bf4-baf2-309acdd96f49-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:35:33.252679 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:33.252604    2523 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b08bb0ac-24c9-4bf4-baf2-309acdd96f49-cabundle-cert\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:35:33.252679 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:33.252627    2523 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxd9r\" (UniqueName: \"kubernetes.io/projected/b08bb0ac-24c9-4bf4-baf2-309acdd96f49-kube-api-access-wxd9r\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:35:33.252679 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:33.252640    2523 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b08bb0ac-24c9-4bf4-baf2-309acdd96f49-proxy-tls\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:35:33.257972 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:33.257955    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-psxj5"
Sep 09 20:35:33.353505 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:33.353478    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e7656fc-2108-410e-8a39-749ba37b9c2c-kserve-provision-location\") pod \"3e7656fc-2108-410e-8a39-749ba37b9c2c\" (UID: \"3e7656fc-2108-410e-8a39-749ba37b9c2c\") "
Sep 09 20:35:33.353606 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:33.353510    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmf8p\" (UniqueName: \"kubernetes.io/projected/3e7656fc-2108-410e-8a39-749ba37b9c2c-kube-api-access-tmf8p\") pod \"3e7656fc-2108-410e-8a39-749ba37b9c2c\" (UID: \"3e7656fc-2108-410e-8a39-749ba37b9c2c\") "
Sep 09 20:35:33.353606 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:33.353535    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e7656fc-2108-410e-8a39-749ba37b9c2c-proxy-tls\") pod \"3e7656fc-2108-410e-8a39-749ba37b9c2c\" (UID: \"3e7656fc-2108-410e-8a39-749ba37b9c2c\") "
Sep 09 20:35:33.353606 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:33.353569    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e7656fc-2108-410e-8a39-749ba37b9c2c-cabundle-cert\") pod \"3e7656fc-2108-410e-8a39-749ba37b9c2c\" (UID: \"3e7656fc-2108-410e-8a39-749ba37b9c2c\") "
Sep 09 20:35:33.353606 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:33.353590    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e7656fc-2108-410e-8a39-749ba37b9c2c-kube-rbac-proxy-sar-config\") pod \"3e7656fc-2108-410e-8a39-749ba37b9c2c\" (UID: \"3e7656fc-2108-410e-8a39-749ba37b9c2c\") "
Sep 09 20:35:33.353837 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:33.353811    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e7656fc-2108-410e-8a39-749ba37b9c2c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3e7656fc-2108-410e-8a39-749ba37b9c2c" (UID: "3e7656fc-2108-410e-8a39-749ba37b9c2c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:35:33.353941 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:33.353923    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e7656fc-2108-410e-8a39-749ba37b9c2c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3e7656fc-2108-410e-8a39-749ba37b9c2c" (UID: "3e7656fc-2108-410e-8a39-749ba37b9c2c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:35:33.354009 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:33.353960    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e7656fc-2108-410e-8a39-749ba37b9c2c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3e7656fc-2108-410e-8a39-749ba37b9c2c" (UID: "3e7656fc-2108-410e-8a39-749ba37b9c2c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:35:33.355680 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:33.355655    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e7656fc-2108-410e-8a39-749ba37b9c2c-kube-api-access-tmf8p" (OuterVolumeSpecName: "kube-api-access-tmf8p") pod "3e7656fc-2108-410e-8a39-749ba37b9c2c" (UID: "3e7656fc-2108-410e-8a39-749ba37b9c2c"). InnerVolumeSpecName "kube-api-access-tmf8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:35:33.355771 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:33.355679    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e7656fc-2108-410e-8a39-749ba37b9c2c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3e7656fc-2108-410e-8a39-749ba37b9c2c" (UID: "3e7656fc-2108-410e-8a39-749ba37b9c2c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:35:33.454562 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:33.454542    2523 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e7656fc-2108-410e-8a39-749ba37b9c2c-proxy-tls\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:35:33.454562 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:33.454560    2523 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e7656fc-2108-410e-8a39-749ba37b9c2c-cabundle-cert\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:35:33.454677 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:33.454573    2523 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e7656fc-2108-410e-8a39-749ba37b9c2c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:35:33.454677 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:33.454587    2523 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e7656fc-2108-410e-8a39-749ba37b9c2c-kserve-provision-location\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:35:33.454677 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:33.454595    2523 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmf8p\" (UniqueName: \"kubernetes.io/projected/3e7656fc-2108-410e-8a39-749ba37b9c2c-kube-api-access-tmf8p\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:35:33.806958 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:33.806931    2523 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b08bb0ac-24c9-4bf4-baf2-309acdd96f49" path="/var/lib/kubelet/pods/b08bb0ac-24c9-4bf4-baf2-309acdd96f49/volumes"
Sep 09 20:35:34.214049 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:34.214030    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-psxj5"
Sep 09 20:35:34.214486 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:34.214029    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-psxj5" event={"ID":"3e7656fc-2108-410e-8a39-749ba37b9c2c","Type":"ContainerDied","Data":"12a81d737342ea14f66646b9016b6a0ed982c183de8ac3490b57661bbe730c41"}
Sep 09 20:35:34.214486 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:34.214191    2523 scope.go:117] "RemoveContainer" containerID="b1070d14e30b2426249b5f5bc6ace58cdcaa9dadd2c16f932fd3a4c547d27449"
Sep 09 20:35:34.221678 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:34.221661    2523 scope.go:117] "RemoveContainer" containerID="d353f0d47d0be2c56bb7663c0f72ac049106acee9cad28cb6f1bf4159abeee2a"
Sep 09 20:35:34.228534 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:34.228518    2523 scope.go:117] "RemoveContainer" containerID="4eeef90533b9daef8938c1fa173dfc26f46d693bfcfef23a8335ca0872f2a662"
Sep 09 20:35:34.232961 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:34.232940    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-psxj5"]
Sep 09 20:35:34.238994 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:34.238974    2523 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-psxj5"]
Sep 09 20:35:35.812129 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:35:35.812092    2523 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e7656fc-2108-410e-8a39-749ba37b9c2c" path="/var/lib/kubelet/pods/3e7656fc-2108-410e-8a39-749ba37b9c2c/volumes"
Sep 09 20:37:24.557879 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:24.557847    2523 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr"]
Sep 09 20:37:24.558322 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:24.558105    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e7656fc-2108-410e-8a39-749ba37b9c2c" containerName="kserve-container"
Sep 09 20:37:24.558322 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:24.558115    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e7656fc-2108-410e-8a39-749ba37b9c2c" containerName="kserve-container"
Sep 09 20:37:24.558322 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:24.558128    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e7656fc-2108-410e-8a39-749ba37b9c2c" containerName="storage-initializer"
Sep 09 20:37:24.558322 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:24.558134    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e7656fc-2108-410e-8a39-749ba37b9c2c" containerName="storage-initializer"
Sep 09 20:37:24.558322 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:24.558141    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b08bb0ac-24c9-4bf4-baf2-309acdd96f49" containerName="kube-rbac-proxy"
Sep 09 20:37:24.558322 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:24.558161    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="b08bb0ac-24c9-4bf4-baf2-309acdd96f49" containerName="kube-rbac-proxy"
Sep 09 20:37:24.558322 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:24.558167    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b08bb0ac-24c9-4bf4-baf2-309acdd96f49" containerName="kserve-container"
Sep 09 20:37:24.558322 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:24.558172    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="b08bb0ac-24c9-4bf4-baf2-309acdd96f49" containerName="kserve-container"
Sep 09 20:37:24.558322 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:24.558178    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e7656fc-2108-410e-8a39-749ba37b9c2c" containerName="kube-rbac-proxy"
Sep 09 20:37:24.558322 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:24.558183    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e7656fc-2108-410e-8a39-749ba37b9c2c" containerName="kube-rbac-proxy"
Sep 09 20:37:24.558322 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:24.558192    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b08bb0ac-24c9-4bf4-baf2-309acdd96f49" containerName="storage-initializer"
Sep 09 20:37:24.558322 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:24.558197    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="b08bb0ac-24c9-4bf4-baf2-309acdd96f49" containerName="storage-initializer"
Sep 09 20:37:24.558322 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:24.558252    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e7656fc-2108-410e-8a39-749ba37b9c2c" containerName="kube-rbac-proxy"
Sep 09 20:37:24.558322 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:24.558260    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="b08bb0ac-24c9-4bf4-baf2-309acdd96f49" containerName="kube-rbac-proxy"
Sep 09 20:37:24.558322 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:24.558266    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="b08bb0ac-24c9-4bf4-baf2-309acdd96f49" containerName="kserve-container"
Sep 09 20:37:24.558322 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:24.558271    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e7656fc-2108-410e-8a39-749ba37b9c2c" containerName="kserve-container"
Sep 09 20:37:24.560249 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:24.560233    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr"
Sep 09 20:37:24.562693 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:24.562670    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:37:24.562693 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:24.562682    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 20:37:24.562863 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:24.562729    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 20:37:24.562863 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:24.562778    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-ks7mb\""
Sep 09 20:37:24.563598 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:24.563566    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
Sep 09 20:37:24.563598 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:24.563577    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
Sep 09 20:37:24.563763 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:24.563681    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 20:37:24.572530 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:24.572507    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr"]
Sep 09 20:37:24.677256 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:24.677235    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsqz2\" (UniqueName: \"kubernetes.io/projected/8a9d7bc0-003a-4823-bd27-86f78455e44b-kube-api-access-xsqz2\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr\" (UID: \"8a9d7bc0-003a-4823-bd27-86f78455e44b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr"
Sep 09 20:37:24.677363 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:24.677266    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a9d7bc0-003a-4823-bd27-86f78455e44b-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr\" (UID: \"8a9d7bc0-003a-4823-bd27-86f78455e44b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr"
Sep 09 20:37:24.677363 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:24.677291    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a9d7bc0-003a-4823-bd27-86f78455e44b-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr\" (UID: \"8a9d7bc0-003a-4823-bd27-86f78455e44b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr"
Sep 09 20:37:24.677363 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:24.677314    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a9d7bc0-003a-4823-bd27-86f78455e44b-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr\" (UID: \"8a9d7bc0-003a-4823-bd27-86f78455e44b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr"
Sep 09 20:37:24.677472 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:24.677369    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a9d7bc0-003a-4823-bd27-86f78455e44b-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr\" (UID: \"8a9d7bc0-003a-4823-bd27-86f78455e44b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr"
Sep 09 20:37:24.778634 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:24.778611    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a9d7bc0-003a-4823-bd27-86f78455e44b-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr\" (UID: \"8a9d7bc0-003a-4823-bd27-86f78455e44b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr"
Sep 09 20:37:24.778757 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:24.778641    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a9d7bc0-003a-4823-bd27-86f78455e44b-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr\" (UID: \"8a9d7bc0-003a-4823-bd27-86f78455e44b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr"
Sep 09 20:37:24.778757 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:24.778663    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a9d7bc0-003a-4823-bd27-86f78455e44b-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr\" (UID: \"8a9d7bc0-003a-4823-bd27-86f78455e44b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr"
Sep 09 20:37:24.778863 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:24.778776    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsqz2\" (UniqueName: \"kubernetes.io/projected/8a9d7bc0-003a-4823-bd27-86f78455e44b-kube-api-access-xsqz2\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr\" (UID: \"8a9d7bc0-003a-4823-bd27-86f78455e44b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr"
Sep 09 20:37:24.778863 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:24.778807    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a9d7bc0-003a-4823-bd27-86f78455e44b-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr\" (UID: \"8a9d7bc0-003a-4823-bd27-86f78455e44b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr"
Sep 09 20:37:24.779063 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:24.779043    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a9d7bc0-003a-4823-bd27-86f78455e44b-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr\" (UID: \"8a9d7bc0-003a-4823-bd27-86f78455e44b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr"
Sep 09 20:37:24.779329 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:24.779309    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a9d7bc0-003a-4823-bd27-86f78455e44b-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr\" (UID: \"8a9d7bc0-003a-4823-bd27-86f78455e44b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr"
Sep 09 20:37:24.779413 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:24.779369    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a9d7bc0-003a-4823-bd27-86f78455e44b-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr\" (UID: \"8a9d7bc0-003a-4823-bd27-86f78455e44b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr"
Sep 09 20:37:24.781268 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:24.781245    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a9d7bc0-003a-4823-bd27-86f78455e44b-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr\" (UID: \"8a9d7bc0-003a-4823-bd27-86f78455e44b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr"
Sep 09 20:37:24.789727 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:24.789710    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsqz2\" (UniqueName: \"kubernetes.io/projected/8a9d7bc0-003a-4823-bd27-86f78455e44b-kube-api-access-xsqz2\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr\" (UID: \"8a9d7bc0-003a-4823-bd27-86f78455e44b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr"
Sep 09 20:37:24.869946 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:24.869925    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr"
Sep 09 20:37:25.140115 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:25.140093    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr"]
Sep 09 20:37:25.142703 ip-10-0-132-226 kubenswrapper[2523]: W0909 20:37:25.142679    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a9d7bc0_003a_4823_bd27_86f78455e44b.slice/crio-86dbb98d2756e3eada3ea4ac2ab56803505594253ce039de9ccc2509424319d7 WatchSource:0}: Error finding container 86dbb98d2756e3eada3ea4ac2ab56803505594253ce039de9ccc2509424319d7: Status 404 returned error can't find the container with id 86dbb98d2756e3eada3ea4ac2ab56803505594253ce039de9ccc2509424319d7
Sep 09 20:37:25.609551 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:25.609518    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr" event={"ID":"8a9d7bc0-003a-4823-bd27-86f78455e44b","Type":"ContainerStarted","Data":"e08c0261e9cb79d676a198f20b94cbb56fe6edb332b5e385d7b53b842fe1d38a"}
Sep 09 20:37:25.609900 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:25.609575    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr" event={"ID":"8a9d7bc0-003a-4823-bd27-86f78455e44b","Type":"ContainerStarted","Data":"86dbb98d2756e3eada3ea4ac2ab56803505594253ce039de9ccc2509424319d7"}
Sep 09 20:37:29.625797 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:29.625763    2523 generic.go:356] "Generic (PLEG): container finished" podID="8a9d7bc0-003a-4823-bd27-86f78455e44b" containerID="e08c0261e9cb79d676a198f20b94cbb56fe6edb332b5e385d7b53b842fe1d38a" exitCode=0
Sep 09 20:37:29.626163 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:29.625838    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr" event={"ID":"8a9d7bc0-003a-4823-bd27-86f78455e44b","Type":"ContainerDied","Data":"e08c0261e9cb79d676a198f20b94cbb56fe6edb332b5e385d7b53b842fe1d38a"}
Sep 09 20:37:29.627015 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:29.626999    2523 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:37:32.638144 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:32.638105    2523 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-jw85f"]
Sep 09 20:37:32.643303 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:32.643277    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-jw85f"
Sep 09 20:37:32.645590 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:32.645564    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
Sep 09 20:37:32.645830 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:32.645813    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-predictor-serving-cert\""
Sep 09 20:37:32.651021 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:32.650998    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-jw85f"]
Sep 09 20:37:32.732738 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:32.732709    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggw7g\" (UniqueName: \"kubernetes.io/projected/7eeb09d5-686c-4957-801f-fe6f61e9b10a-kube-api-access-ggw7g\") pod \"isvc-xgboost-v2-predictor-fcc668bc-jw85f\" (UID: \"7eeb09d5-686c-4957-801f-fe6f61e9b10a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-jw85f"
Sep 09 20:37:32.732899 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:32.732755    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7eeb09d5-686c-4957-801f-fe6f61e9b10a-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-jw85f\" (UID: \"7eeb09d5-686c-4957-801f-fe6f61e9b10a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-jw85f"
Sep 09 20:37:32.732899 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:32.732785    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7eeb09d5-686c-4957-801f-fe6f61e9b10a-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-jw85f\" (UID: \"7eeb09d5-686c-4957-801f-fe6f61e9b10a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-jw85f"
Sep 09 20:37:32.732899 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:32.732863    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7eeb09d5-686c-4957-801f-fe6f61e9b10a-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-jw85f\" (UID: \"7eeb09d5-686c-4957-801f-fe6f61e9b10a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-jw85f"
Sep 09 20:37:32.733065 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:32.733020    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7eeb09d5-686c-4957-801f-fe6f61e9b10a-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-jw85f\" (UID: \"7eeb09d5-686c-4957-801f-fe6f61e9b10a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-jw85f"
Sep 09 20:37:32.834209 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:32.834173    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7eeb09d5-686c-4957-801f-fe6f61e9b10a-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-jw85f\" (UID: \"7eeb09d5-686c-4957-801f-fe6f61e9b10a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-jw85f"
Sep 09 20:37:32.834411 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:32.834289    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggw7g\" (UniqueName: \"kubernetes.io/projected/7eeb09d5-686c-4957-801f-fe6f61e9b10a-kube-api-access-ggw7g\") pod \"isvc-xgboost-v2-predictor-fcc668bc-jw85f\" (UID: \"7eeb09d5-686c-4957-801f-fe6f61e9b10a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-jw85f"
Sep 09 20:37:32.834411 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:32.834323    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7eeb09d5-686c-4957-801f-fe6f61e9b10a-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-jw85f\" (UID: \"7eeb09d5-686c-4957-801f-fe6f61e9b10a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-jw85f"
Sep 09 20:37:32.834411 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:32.834359    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7eeb09d5-686c-4957-801f-fe6f61e9b10a-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-jw85f\" (UID: \"7eeb09d5-686c-4957-801f-fe6f61e9b10a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-jw85f"
Sep 09 20:37:32.834411 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:32.834390    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7eeb09d5-686c-4957-801f-fe6f61e9b10a-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-jw85f\" (UID: \"7eeb09d5-686c-4957-801f-fe6f61e9b10a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-jw85f"
Sep 09 20:37:32.834665 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:32.834583    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7eeb09d5-686c-4957-801f-fe6f61e9b10a-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-jw85f\" (UID: \"7eeb09d5-686c-4957-801f-fe6f61e9b10a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-jw85f"
Sep 09 20:37:32.834727 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:37:32.834686    2523 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-predictor-serving-cert: secret "isvc-xgboost-v2-predictor-serving-cert" not found
Sep 09 20:37:32.834785 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:37:32.834761    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7eeb09d5-686c-4957-801f-fe6f61e9b10a-proxy-tls podName:7eeb09d5-686c-4957-801f-fe6f61e9b10a nodeName:}" failed. No retries permitted until 2026-09-09 20:37:33.334740197 +0000 UTC m=+2558.006759178 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7eeb09d5-686c-4957-801f-fe6f61e9b10a-proxy-tls") pod "isvc-xgboost-v2-predictor-fcc668bc-jw85f" (UID: "7eeb09d5-686c-4957-801f-fe6f61e9b10a") : secret "isvc-xgboost-v2-predictor-serving-cert" not found
Sep 09 20:37:32.835132 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:32.835104    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7eeb09d5-686c-4957-801f-fe6f61e9b10a-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-jw85f\" (UID: \"7eeb09d5-686c-4957-801f-fe6f61e9b10a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-jw85f"
Sep 09 20:37:32.835296 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:32.835190    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7eeb09d5-686c-4957-801f-fe6f61e9b10a-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-jw85f\" (UID: \"7eeb09d5-686c-4957-801f-fe6f61e9b10a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-jw85f"
Sep 09 20:37:32.845983 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:32.845961    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggw7g\" (UniqueName: \"kubernetes.io/projected/7eeb09d5-686c-4957-801f-fe6f61e9b10a-kube-api-access-ggw7g\") pod \"isvc-xgboost-v2-predictor-fcc668bc-jw85f\" (UID: \"7eeb09d5-686c-4957-801f-fe6f61e9b10a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-jw85f"
Sep 09 20:37:33.339699 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:33.339661    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7eeb09d5-686c-4957-801f-fe6f61e9b10a-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-jw85f\" (UID: \"7eeb09d5-686c-4957-801f-fe6f61e9b10a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-jw85f"
Sep 09 20:37:33.342415 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:33.342382    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7eeb09d5-686c-4957-801f-fe6f61e9b10a-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-jw85f\" (UID: \"7eeb09d5-686c-4957-801f-fe6f61e9b10a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-jw85f"
Sep 09 20:37:33.555978 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:33.555930    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-jw85f"
Sep 09 20:37:35.970678 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:35.970654    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-jw85f"]
Sep 09 20:37:35.972399 ip-10-0-132-226 kubenswrapper[2523]: W0909 20:37:35.972373    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eeb09d5_686c_4957_801f_fe6f61e9b10a.slice/crio-2659271c268b3a458eeaf0e941e26634b9ab532d46ef77063e781809ccdb31f6 WatchSource:0}: Error finding container 2659271c268b3a458eeaf0e941e26634b9ab532d46ef77063e781809ccdb31f6: Status 404 returned error can't find the container with id 2659271c268b3a458eeaf0e941e26634b9ab532d46ef77063e781809ccdb31f6
Sep 09 20:37:36.658225 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:36.658194    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr" event={"ID":"8a9d7bc0-003a-4823-bd27-86f78455e44b","Type":"ContainerStarted","Data":"b859711c246fc8ae554f409c3bb491c3534bb3a7424beca452744c32700aef55"}
Sep 09 20:37:36.658405 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:36.658264    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr" event={"ID":"8a9d7bc0-003a-4823-bd27-86f78455e44b","Type":"ContainerStarted","Data":"d3b3b3086342fdea1388cbad26334c12338f62c6501a7c723da7f311ef261989"}
Sep 09 20:37:36.658486 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:36.658471    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr"
Sep 09 20:37:36.659847 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:36.659810    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-jw85f" event={"ID":"7eeb09d5-686c-4957-801f-fe6f61e9b10a","Type":"ContainerStarted","Data":"b33511548031a12f0ead2c5d32de5a3de886e6d1b6840ce04fdaaf8082a49b8b"}
Sep 09 20:37:36.659970 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:36.659867    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-jw85f" event={"ID":"7eeb09d5-686c-4957-801f-fe6f61e9b10a","Type":"ContainerStarted","Data":"2659271c268b3a458eeaf0e941e26634b9ab532d46ef77063e781809ccdb31f6"}
Sep 09 20:37:36.678382 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:36.678339    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr" podStartSLOduration=2.796709795 podStartE2EDuration="12.678324513s" totalImagesPullingTime="6.134990264s" totalInitContainerRuntime="3.746624454s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:37:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:37:36.677781326 +0000 UTC m=+2561.349800323" watchObservedRunningTime="2026-09-09 20:37:36.678324513 +0000 UTC m=+2561.350343510"
Sep 09 20:37:37.664103 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:37.664066    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr"
Sep 09 20:37:37.665434 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:37.665406    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr" podUID="8a9d7bc0-003a-4823-bd27-86f78455e44b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 09 20:37:38.667883 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:38.667839    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr" podUID="8a9d7bc0-003a-4823-bd27-86f78455e44b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 09 20:37:40.676863 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:40.676830    2523 generic.go:356] "Generic (PLEG): container finished" podID="7eeb09d5-686c-4957-801f-fe6f61e9b10a" containerID="b33511548031a12f0ead2c5d32de5a3de886e6d1b6840ce04fdaaf8082a49b8b" exitCode=0
Sep 09 20:37:40.677289 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:40.676889    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-jw85f" event={"ID":"7eeb09d5-686c-4957-801f-fe6f61e9b10a","Type":"ContainerDied","Data":"b33511548031a12f0ead2c5d32de5a3de886e6d1b6840ce04fdaaf8082a49b8b"}
Sep 09 20:37:41.682561 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:41.682524    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-jw85f" event={"ID":"7eeb09d5-686c-4957-801f-fe6f61e9b10a","Type":"ContainerStarted","Data":"bf1eb93317f3f69d2b72a38feb9baa7efcf3eb6fe18a421e62a59469c8b394d6"}
Sep 09 20:37:41.682933 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:41.682588    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-jw85f" event={"ID":"7eeb09d5-686c-4957-801f-fe6f61e9b10a","Type":"ContainerStarted","Data":"9d0a55d42c1f6ab96e5b6ad29a57692f29d79490e527687a6c48c3e72f78b22d"}
Sep 09 20:37:41.682933 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:41.682820    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-jw85f"
Sep 09 20:37:41.702697 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:41.702658    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-jw85f" podStartSLOduration=5.888156609 podStartE2EDuration="9.702644881s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.814488272s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:37:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:37:41.701257582 +0000 UTC m=+2566.373276579" watchObservedRunningTime="2026-09-09 20:37:41.702644881 +0000 UTC m=+2566.374663878"
Sep 09 20:37:42.685983 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:42.685957    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-jw85f"
Sep 09 20:37:42.687437 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:42.687407    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-jw85f" podUID="7eeb09d5-686c-4957-801f-fe6f61e9b10a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 09 20:37:43.673663 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:43.673630    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr"
Sep 09 20:37:43.674176 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:43.674128    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr" podUID="8a9d7bc0-003a-4823-bd27-86f78455e44b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 09 20:37:43.689834 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:43.689803    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-jw85f" podUID="7eeb09d5-686c-4957-801f-fe6f61e9b10a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 09 20:37:48.694779 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:48.694750    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-jw85f"
Sep 09 20:37:48.695465 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:48.695434    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-jw85f" podUID="7eeb09d5-686c-4957-801f-fe6f61e9b10a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 09 20:37:53.674292 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:53.674255    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr" podUID="8a9d7bc0-003a-4823-bd27-86f78455e44b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 09 20:37:58.695852 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:37:58.695812    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-jw85f" podUID="7eeb09d5-686c-4957-801f-fe6f61e9b10a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 09 20:38:03.675112 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:03.675064    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr" podUID="8a9d7bc0-003a-4823-bd27-86f78455e44b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 09 20:38:08.695513 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:08.695479    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-jw85f" podUID="7eeb09d5-686c-4957-801f-fe6f61e9b10a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 09 20:38:13.674980 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:13.674937    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr" podUID="8a9d7bc0-003a-4823-bd27-86f78455e44b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 09 20:38:18.696122 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:18.696045    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-jw85f" podUID="7eeb09d5-686c-4957-801f-fe6f61e9b10a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 09 20:38:23.674881 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:23.674838    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr" podUID="8a9d7bc0-003a-4823-bd27-86f78455e44b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 09 20:38:28.695918 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:28.695872    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-jw85f" podUID="7eeb09d5-686c-4957-801f-fe6f61e9b10a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 09 20:38:33.675024 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:33.674983    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr" podUID="8a9d7bc0-003a-4823-bd27-86f78455e44b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 09 20:38:38.696721 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:38.696688    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-jw85f"
Sep 09 20:38:42.700799 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:42.700752    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-jw85f"]
Sep 09 20:38:42.701309 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:42.700975    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-jw85f" podUID="7eeb09d5-686c-4957-801f-fe6f61e9b10a" containerName="kserve-container" containerID="cri-o://9d0a55d42c1f6ab96e5b6ad29a57692f29d79490e527687a6c48c3e72f78b22d" gracePeriod=30
Sep 09 20:38:42.701309 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:42.701027    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-jw85f" podUID="7eeb09d5-686c-4957-801f-fe6f61e9b10a" containerName="kube-rbac-proxy" containerID="cri-o://bf1eb93317f3f69d2b72a38feb9baa7efcf3eb6fe18a421e62a59469c8b394d6" gracePeriod=30
Sep 09 20:38:42.915343 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:42.915313    2523 generic.go:356] "Generic (PLEG): container finished" podID="7eeb09d5-686c-4957-801f-fe6f61e9b10a" containerID="bf1eb93317f3f69d2b72a38feb9baa7efcf3eb6fe18a421e62a59469c8b394d6" exitCode=2
Sep 09 20:38:42.915432 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:42.915356    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-jw85f" event={"ID":"7eeb09d5-686c-4957-801f-fe6f61e9b10a","Type":"ContainerDied","Data":"bf1eb93317f3f69d2b72a38feb9baa7efcf3eb6fe18a421e62a59469c8b394d6"}
Sep 09 20:38:43.674891 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:43.674851    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr" podUID="8a9d7bc0-003a-4823-bd27-86f78455e44b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 09 20:38:43.690145 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:43.690112    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-jw85f" podUID="7eeb09d5-686c-4957-801f-fe6f61e9b10a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused"
Sep 09 20:38:45.643775 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:45.643744    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-jw85f"
Sep 09 20:38:45.751742 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:45.751671    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7eeb09d5-686c-4957-801f-fe6f61e9b10a-kube-rbac-proxy-sar-config\") pod \"7eeb09d5-686c-4957-801f-fe6f61e9b10a\" (UID: \"7eeb09d5-686c-4957-801f-fe6f61e9b10a\") "
Sep 09 20:38:45.751742 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:45.751719    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7eeb09d5-686c-4957-801f-fe6f61e9b10a-kserve-provision-location\") pod \"7eeb09d5-686c-4957-801f-fe6f61e9b10a\" (UID: \"7eeb09d5-686c-4957-801f-fe6f61e9b10a\") "
Sep 09 20:38:45.751903 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:45.751762    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7eeb09d5-686c-4957-801f-fe6f61e9b10a-cabundle-cert\") pod \"7eeb09d5-686c-4957-801f-fe6f61e9b10a\" (UID: \"7eeb09d5-686c-4957-801f-fe6f61e9b10a\") "
Sep 09 20:38:45.751903 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:45.751785    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7eeb09d5-686c-4957-801f-fe6f61e9b10a-proxy-tls\") pod \"7eeb09d5-686c-4957-801f-fe6f61e9b10a\" (UID: \"7eeb09d5-686c-4957-801f-fe6f61e9b10a\") "
Sep 09 20:38:45.751903 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:45.751803    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ggw7g\" (UniqueName: \"kubernetes.io/projected/7eeb09d5-686c-4957-801f-fe6f61e9b10a-kube-api-access-ggw7g\") pod \"7eeb09d5-686c-4957-801f-fe6f61e9b10a\" (UID: \"7eeb09d5-686c-4957-801f-fe6f61e9b10a\") "
Sep 09 20:38:45.752141 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:45.752112    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eeb09d5-686c-4957-801f-fe6f61e9b10a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7eeb09d5-686c-4957-801f-fe6f61e9b10a" (UID: "7eeb09d5-686c-4957-801f-fe6f61e9b10a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:38:45.752248 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:45.752120    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eeb09d5-686c-4957-801f-fe6f61e9b10a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7eeb09d5-686c-4957-801f-fe6f61e9b10a" (UID: "7eeb09d5-686c-4957-801f-fe6f61e9b10a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:38:45.752248 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:45.752123    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eeb09d5-686c-4957-801f-fe6f61e9b10a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7eeb09d5-686c-4957-801f-fe6f61e9b10a" (UID: "7eeb09d5-686c-4957-801f-fe6f61e9b10a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:38:45.753930 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:45.753908    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7eeb09d5-686c-4957-801f-fe6f61e9b10a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7eeb09d5-686c-4957-801f-fe6f61e9b10a" (UID: "7eeb09d5-686c-4957-801f-fe6f61e9b10a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:38:45.754086 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:45.754066    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eeb09d5-686c-4957-801f-fe6f61e9b10a-kube-api-access-ggw7g" (OuterVolumeSpecName: "kube-api-access-ggw7g") pod "7eeb09d5-686c-4957-801f-fe6f61e9b10a" (UID: "7eeb09d5-686c-4957-801f-fe6f61e9b10a"). InnerVolumeSpecName "kube-api-access-ggw7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:38:45.852442 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:45.852419    2523 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7eeb09d5-686c-4957-801f-fe6f61e9b10a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:38:45.852442 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:45.852439    2523 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7eeb09d5-686c-4957-801f-fe6f61e9b10a-kserve-provision-location\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:38:45.852557 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:45.852448    2523 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7eeb09d5-686c-4957-801f-fe6f61e9b10a-cabundle-cert\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:38:45.852557 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:45.852457    2523 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7eeb09d5-686c-4957-801f-fe6f61e9b10a-proxy-tls\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:38:45.852557 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:45.852464    2523 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggw7g\" (UniqueName: \"kubernetes.io/projected/7eeb09d5-686c-4957-801f-fe6f61e9b10a-kube-api-access-ggw7g\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:38:45.928171 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:45.928124    2523 generic.go:356] "Generic (PLEG): container finished" podID="7eeb09d5-686c-4957-801f-fe6f61e9b10a" containerID="9d0a55d42c1f6ab96e5b6ad29a57692f29d79490e527687a6c48c3e72f78b22d" exitCode=0
Sep 09 20:38:45.928270 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:45.928206    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-jw85f"
Sep 09 20:38:45.928270 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:45.928207    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-jw85f" event={"ID":"7eeb09d5-686c-4957-801f-fe6f61e9b10a","Type":"ContainerDied","Data":"9d0a55d42c1f6ab96e5b6ad29a57692f29d79490e527687a6c48c3e72f78b22d"}
Sep 09 20:38:45.928355 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:45.928276    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-jw85f" event={"ID":"7eeb09d5-686c-4957-801f-fe6f61e9b10a","Type":"ContainerDied","Data":"2659271c268b3a458eeaf0e941e26634b9ab532d46ef77063e781809ccdb31f6"}
Sep 09 20:38:45.928355 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:45.928313    2523 scope.go:117] "RemoveContainer" containerID="bf1eb93317f3f69d2b72a38feb9baa7efcf3eb6fe18a421e62a59469c8b394d6"
Sep 09 20:38:45.935917 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:45.935901    2523 scope.go:117] "RemoveContainer" containerID="9d0a55d42c1f6ab96e5b6ad29a57692f29d79490e527687a6c48c3e72f78b22d"
Sep 09 20:38:45.942645 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:45.942623    2523 scope.go:117] "RemoveContainer" containerID="b33511548031a12f0ead2c5d32de5a3de886e6d1b6840ce04fdaaf8082a49b8b"
Sep 09 20:38:45.944200 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:45.944179    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-jw85f"]
Sep 09 20:38:45.947953 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:45.947936    2523 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-jw85f"]
Sep 09 20:38:45.950289 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:45.950265    2523 scope.go:117] "RemoveContainer" containerID="bf1eb93317f3f69d2b72a38feb9baa7efcf3eb6fe18a421e62a59469c8b394d6"
Sep 09 20:38:45.950540 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:38:45.950521    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf1eb93317f3f69d2b72a38feb9baa7efcf3eb6fe18a421e62a59469c8b394d6\": container with ID starting with bf1eb93317f3f69d2b72a38feb9baa7efcf3eb6fe18a421e62a59469c8b394d6 not found: ID does not exist" containerID="bf1eb93317f3f69d2b72a38feb9baa7efcf3eb6fe18a421e62a59469c8b394d6"
Sep 09 20:38:45.950621 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:45.950551    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf1eb93317f3f69d2b72a38feb9baa7efcf3eb6fe18a421e62a59469c8b394d6"} err="failed to get container status \"bf1eb93317f3f69d2b72a38feb9baa7efcf3eb6fe18a421e62a59469c8b394d6\": rpc error: code = NotFound desc = could not find container \"bf1eb93317f3f69d2b72a38feb9baa7efcf3eb6fe18a421e62a59469c8b394d6\": container with ID starting with bf1eb93317f3f69d2b72a38feb9baa7efcf3eb6fe18a421e62a59469c8b394d6 not found: ID does not exist"
Sep 09 20:38:45.950621 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:45.950572    2523 scope.go:117] "RemoveContainer" containerID="9d0a55d42c1f6ab96e5b6ad29a57692f29d79490e527687a6c48c3e72f78b22d"
Sep 09 20:38:45.950822 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:38:45.950805    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d0a55d42c1f6ab96e5b6ad29a57692f29d79490e527687a6c48c3e72f78b22d\": container with ID starting with 9d0a55d42c1f6ab96e5b6ad29a57692f29d79490e527687a6c48c3e72f78b22d not found: ID does not exist" containerID="9d0a55d42c1f6ab96e5b6ad29a57692f29d79490e527687a6c48c3e72f78b22d"
Sep 09 20:38:45.950861 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:45.950828    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d0a55d42c1f6ab96e5b6ad29a57692f29d79490e527687a6c48c3e72f78b22d"} err="failed to get container status \"9d0a55d42c1f6ab96e5b6ad29a57692f29d79490e527687a6c48c3e72f78b22d\": rpc error: code = NotFound desc = could not find container \"9d0a55d42c1f6ab96e5b6ad29a57692f29d79490e527687a6c48c3e72f78b22d\": container with ID starting with 9d0a55d42c1f6ab96e5b6ad29a57692f29d79490e527687a6c48c3e72f78b22d not found: ID does not exist"
Sep 09 20:38:45.950861 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:45.950842    2523 scope.go:117] "RemoveContainer" containerID="b33511548031a12f0ead2c5d32de5a3de886e6d1b6840ce04fdaaf8082a49b8b"
Sep 09 20:38:45.951092 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:38:45.951076    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b33511548031a12f0ead2c5d32de5a3de886e6d1b6840ce04fdaaf8082a49b8b\": container with ID starting with b33511548031a12f0ead2c5d32de5a3de886e6d1b6840ce04fdaaf8082a49b8b not found: ID does not exist" containerID="b33511548031a12f0ead2c5d32de5a3de886e6d1b6840ce04fdaaf8082a49b8b"
Sep 09 20:38:45.951166 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:45.951100    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b33511548031a12f0ead2c5d32de5a3de886e6d1b6840ce04fdaaf8082a49b8b"} err="failed to get container status \"b33511548031a12f0ead2c5d32de5a3de886e6d1b6840ce04fdaaf8082a49b8b\": rpc error: code = NotFound desc = could not find container \"b33511548031a12f0ead2c5d32de5a3de886e6d1b6840ce04fdaaf8082a49b8b\": container with ID starting with b33511548031a12f0ead2c5d32de5a3de886e6d1b6840ce04fdaaf8082a49b8b not found: ID does not exist"
Sep 09 20:38:47.806677 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:47.806644    2523 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7eeb09d5-686c-4957-801f-fe6f61e9b10a" path="/var/lib/kubelet/pods/7eeb09d5-686c-4957-801f-fe6f61e9b10a/volumes"
Sep 09 20:38:49.812625 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:49.812598    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr"
Sep 09 20:38:54.957764 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:54.957729    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr"]
Sep 09 20:38:54.958237 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:54.958034    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr" podUID="8a9d7bc0-003a-4823-bd27-86f78455e44b" containerName="kserve-container" containerID="cri-o://d3b3b3086342fdea1388cbad26334c12338f62c6501a7c723da7f311ef261989" gracePeriod=30
Sep 09 20:38:54.958237 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:54.958089    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr" podUID="8a9d7bc0-003a-4823-bd27-86f78455e44b" containerName="kube-rbac-proxy" containerID="cri-o://b859711c246fc8ae554f409c3bb491c3534bb3a7424beca452744c32700aef55" gracePeriod=30
Sep 09 20:38:55.974379 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:55.974342    2523 generic.go:356] "Generic (PLEG): container finished" podID="8a9d7bc0-003a-4823-bd27-86f78455e44b" containerID="b859711c246fc8ae554f409c3bb491c3534bb3a7424beca452744c32700aef55" exitCode=2
Sep 09 20:38:55.974721 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:55.974419    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr" event={"ID":"8a9d7bc0-003a-4823-bd27-86f78455e44b","Type":"ContainerDied","Data":"b859711c246fc8ae554f409c3bb491c3534bb3a7424beca452744c32700aef55"}
Sep 09 20:38:58.668903 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:58.668868    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr" podUID="8a9d7bc0-003a-4823-bd27-86f78455e44b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Sep 09 20:38:59.499057 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:59.499033    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr"
Sep 09 20:38:59.654218 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:59.654139    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xsqz2\" (UniqueName: \"kubernetes.io/projected/8a9d7bc0-003a-4823-bd27-86f78455e44b-kube-api-access-xsqz2\") pod \"8a9d7bc0-003a-4823-bd27-86f78455e44b\" (UID: \"8a9d7bc0-003a-4823-bd27-86f78455e44b\") "
Sep 09 20:38:59.654218 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:59.654200    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a9d7bc0-003a-4823-bd27-86f78455e44b-kserve-provision-location\") pod \"8a9d7bc0-003a-4823-bd27-86f78455e44b\" (UID: \"8a9d7bc0-003a-4823-bd27-86f78455e44b\") "
Sep 09 20:38:59.654386 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:59.654230    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a9d7bc0-003a-4823-bd27-86f78455e44b-kube-rbac-proxy-sar-config\") pod \"8a9d7bc0-003a-4823-bd27-86f78455e44b\" (UID: \"8a9d7bc0-003a-4823-bd27-86f78455e44b\") "
Sep 09 20:38:59.654386 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:59.654269    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a9d7bc0-003a-4823-bd27-86f78455e44b-cabundle-cert\") pod \"8a9d7bc0-003a-4823-bd27-86f78455e44b\" (UID: \"8a9d7bc0-003a-4823-bd27-86f78455e44b\") "
Sep 09 20:38:59.654386 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:59.654327    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a9d7bc0-003a-4823-bd27-86f78455e44b-proxy-tls\") pod \"8a9d7bc0-003a-4823-bd27-86f78455e44b\" (UID: \"8a9d7bc0-003a-4823-bd27-86f78455e44b\") "
Sep 09 20:38:59.654562 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:59.654543    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a9d7bc0-003a-4823-bd27-86f78455e44b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8a9d7bc0-003a-4823-bd27-86f78455e44b" (UID: "8a9d7bc0-003a-4823-bd27-86f78455e44b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:38:59.654629 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:59.654610    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a9d7bc0-003a-4823-bd27-86f78455e44b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8a9d7bc0-003a-4823-bd27-86f78455e44b" (UID: "8a9d7bc0-003a-4823-bd27-86f78455e44b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:38:59.654706 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:59.654677    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a9d7bc0-003a-4823-bd27-86f78455e44b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8a9d7bc0-003a-4823-bd27-86f78455e44b" (UID: "8a9d7bc0-003a-4823-bd27-86f78455e44b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:38:59.656325 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:59.656297    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a9d7bc0-003a-4823-bd27-86f78455e44b-kube-api-access-xsqz2" (OuterVolumeSpecName: "kube-api-access-xsqz2") pod "8a9d7bc0-003a-4823-bd27-86f78455e44b" (UID: "8a9d7bc0-003a-4823-bd27-86f78455e44b"). InnerVolumeSpecName "kube-api-access-xsqz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:38:59.656544 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:59.656520    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a9d7bc0-003a-4823-bd27-86f78455e44b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8a9d7bc0-003a-4823-bd27-86f78455e44b" (UID: "8a9d7bc0-003a-4823-bd27-86f78455e44b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:38:59.754983 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:59.754958    2523 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a9d7bc0-003a-4823-bd27-86f78455e44b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:38:59.754983 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:59.754981    2523 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a9d7bc0-003a-4823-bd27-86f78455e44b-cabundle-cert\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:38:59.755327 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:59.754993    2523 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a9d7bc0-003a-4823-bd27-86f78455e44b-proxy-tls\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:38:59.755327 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:59.755000    2523 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsqz2\" (UniqueName: \"kubernetes.io/projected/8a9d7bc0-003a-4823-bd27-86f78455e44b-kube-api-access-xsqz2\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:38:59.755327 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:59.755009    2523 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a9d7bc0-003a-4823-bd27-86f78455e44b-kserve-provision-location\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:38:59.991659 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:59.991599    2523 generic.go:356] "Generic (PLEG): container finished" podID="8a9d7bc0-003a-4823-bd27-86f78455e44b" containerID="d3b3b3086342fdea1388cbad26334c12338f62c6501a7c723da7f311ef261989" exitCode=0
Sep 09 20:38:59.991659 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:59.991653    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr"
Sep 09 20:38:59.991783 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:59.991666    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr" event={"ID":"8a9d7bc0-003a-4823-bd27-86f78455e44b","Type":"ContainerDied","Data":"d3b3b3086342fdea1388cbad26334c12338f62c6501a7c723da7f311ef261989"}
Sep 09 20:38:59.991783 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:59.991724    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr" event={"ID":"8a9d7bc0-003a-4823-bd27-86f78455e44b","Type":"ContainerDied","Data":"86dbb98d2756e3eada3ea4ac2ab56803505594253ce039de9ccc2509424319d7"}
Sep 09 20:38:59.991783 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:59.991750    2523 scope.go:117] "RemoveContainer" containerID="b859711c246fc8ae554f409c3bb491c3534bb3a7424beca452744c32700aef55"
Sep 09 20:38:59.999644 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:38:59.999626    2523 scope.go:117] "RemoveContainer" containerID="d3b3b3086342fdea1388cbad26334c12338f62c6501a7c723da7f311ef261989"
Sep 09 20:39:00.006789 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:39:00.006775    2523 scope.go:117] "RemoveContainer" containerID="e08c0261e9cb79d676a198f20b94cbb56fe6edb332b5e385d7b53b842fe1d38a"
Sep 09 20:39:00.009829 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:39:00.009803    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr"]
Sep 09 20:39:00.010952 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:39:00.010932    2523 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-r2kqr"]
Sep 09 20:39:00.014115 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:39:00.014099    2523 scope.go:117] "RemoveContainer" containerID="b859711c246fc8ae554f409c3bb491c3534bb3a7424beca452744c32700aef55"
Sep 09 20:39:00.014435 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:39:00.014410    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b859711c246fc8ae554f409c3bb491c3534bb3a7424beca452744c32700aef55\": container with ID starting with b859711c246fc8ae554f409c3bb491c3534bb3a7424beca452744c32700aef55 not found: ID does not exist" containerID="b859711c246fc8ae554f409c3bb491c3534bb3a7424beca452744c32700aef55"
Sep 09 20:39:00.014473 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:39:00.014442    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b859711c246fc8ae554f409c3bb491c3534bb3a7424beca452744c32700aef55"} err="failed to get container status \"b859711c246fc8ae554f409c3bb491c3534bb3a7424beca452744c32700aef55\": rpc error: code = NotFound desc = could not find container \"b859711c246fc8ae554f409c3bb491c3534bb3a7424beca452744c32700aef55\": container with ID starting with b859711c246fc8ae554f409c3bb491c3534bb3a7424beca452744c32700aef55 not found: ID does not exist"
Sep 09 20:39:00.014473 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:39:00.014460    2523 scope.go:117] "RemoveContainer" containerID="d3b3b3086342fdea1388cbad26334c12338f62c6501a7c723da7f311ef261989"
Sep 09 20:39:00.014688 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:39:00.014671    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3b3b3086342fdea1388cbad26334c12338f62c6501a7c723da7f311ef261989\": container with ID starting with d3b3b3086342fdea1388cbad26334c12338f62c6501a7c723da7f311ef261989 not found: ID does not exist" containerID="d3b3b3086342fdea1388cbad26334c12338f62c6501a7c723da7f311ef261989"
Sep 09 20:39:00.014729 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:39:00.014695    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3b3b3086342fdea1388cbad26334c12338f62c6501a7c723da7f311ef261989"} err="failed to get container status \"d3b3b3086342fdea1388cbad26334c12338f62c6501a7c723da7f311ef261989\": rpc error: code = NotFound desc = could not find container \"d3b3b3086342fdea1388cbad26334c12338f62c6501a7c723da7f311ef261989\": container with ID starting with d3b3b3086342fdea1388cbad26334c12338f62c6501a7c723da7f311ef261989 not found: ID does not exist"
Sep 09 20:39:00.014729 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:39:00.014710    2523 scope.go:117] "RemoveContainer" containerID="e08c0261e9cb79d676a198f20b94cbb56fe6edb332b5e385d7b53b842fe1d38a"
Sep 09 20:39:00.014928 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:39:00.014913    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e08c0261e9cb79d676a198f20b94cbb56fe6edb332b5e385d7b53b842fe1d38a\": container with ID starting with e08c0261e9cb79d676a198f20b94cbb56fe6edb332b5e385d7b53b842fe1d38a not found: ID does not exist" containerID="e08c0261e9cb79d676a198f20b94cbb56fe6edb332b5e385d7b53b842fe1d38a"
Sep 09 20:39:00.014964 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:39:00.014932    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e08c0261e9cb79d676a198f20b94cbb56fe6edb332b5e385d7b53b842fe1d38a"} err="failed to get container status \"e08c0261e9cb79d676a198f20b94cbb56fe6edb332b5e385d7b53b842fe1d38a\": rpc error: code = NotFound desc = could not find container \"e08c0261e9cb79d676a198f20b94cbb56fe6edb332b5e385d7b53b842fe1d38a\": container with ID starting with e08c0261e9cb79d676a198f20b94cbb56fe6edb332b5e385d7b53b842fe1d38a not found: ID does not exist"
Sep 09 20:39:01.806630 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:39:01.806598    2523 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a9d7bc0-003a-4823-bd27-86f78455e44b" path="/var/lib/kubelet/pods/8a9d7bc0-003a-4823-bd27-86f78455e44b/volumes"
Sep 09 20:40:18.157665 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:18.157592    2523 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8"]
Sep 09 20:40:18.158084 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:18.157850    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a9d7bc0-003a-4823-bd27-86f78455e44b" containerName="storage-initializer"
Sep 09 20:40:18.158084 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:18.157861    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a9d7bc0-003a-4823-bd27-86f78455e44b" containerName="storage-initializer"
Sep 09 20:40:18.158084 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:18.157873    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a9d7bc0-003a-4823-bd27-86f78455e44b" containerName="kube-rbac-proxy"
Sep 09 20:40:18.158084 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:18.157878    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a9d7bc0-003a-4823-bd27-86f78455e44b" containerName="kube-rbac-proxy"
Sep 09 20:40:18.158084 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:18.157884    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7eeb09d5-686c-4957-801f-fe6f61e9b10a" containerName="kserve-container"
Sep 09 20:40:18.158084 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:18.157889    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="7eeb09d5-686c-4957-801f-fe6f61e9b10a" containerName="kserve-container"
Sep 09 20:40:18.158084 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:18.157897    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7eeb09d5-686c-4957-801f-fe6f61e9b10a" containerName="kube-rbac-proxy"
Sep 09 20:40:18.158084 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:18.157902    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="7eeb09d5-686c-4957-801f-fe6f61e9b10a" containerName="kube-rbac-proxy"
Sep 09 20:40:18.158084 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:18.157909    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a9d7bc0-003a-4823-bd27-86f78455e44b" containerName="kserve-container"
Sep 09 20:40:18.158084 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:18.157914    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a9d7bc0-003a-4823-bd27-86f78455e44b" containerName="kserve-container"
Sep 09 20:40:18.158084 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:18.157928    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7eeb09d5-686c-4957-801f-fe6f61e9b10a" containerName="storage-initializer"
Sep 09 20:40:18.158084 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:18.157933    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="7eeb09d5-686c-4957-801f-fe6f61e9b10a" containerName="storage-initializer"
Sep 09 20:40:18.158084 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:18.157973    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="7eeb09d5-686c-4957-801f-fe6f61e9b10a" containerName="kserve-container"
Sep 09 20:40:18.158084 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:18.157980    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a9d7bc0-003a-4823-bd27-86f78455e44b" containerName="kserve-container"
Sep 09 20:40:18.158084 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:18.157985    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a9d7bc0-003a-4823-bd27-86f78455e44b" containerName="kube-rbac-proxy"
Sep 09 20:40:18.158084 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:18.157991    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="7eeb09d5-686c-4957-801f-fe6f61e9b10a" containerName="kube-rbac-proxy"
Sep 09 20:40:18.160884 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:18.160868    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8"
Sep 09 20:40:18.164416 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:18.164392    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:40:18.164564 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:18.164431    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 09 20:40:18.164644 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:18.164593    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
Sep 09 20:40:18.164644 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:18.164607    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
Sep 09 20:40:18.164745 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:18.164612    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-ks7mb\""
Sep 09 20:40:18.165462 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:18.165446    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 20:40:18.165546 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:18.165450    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 20:40:18.181995 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:18.181974    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8"]
Sep 09 20:40:18.253850 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:18.253826    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f4103a1-5268-4298-a114-546110711bd8-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8\" (UID: \"9f4103a1-5268-4298-a114-546110711bd8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8"
Sep 09 20:40:18.253963 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:18.253858    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f4103a1-5268-4298-a114-546110711bd8-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8\" (UID: \"9f4103a1-5268-4298-a114-546110711bd8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8"
Sep 09 20:40:18.253963 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:18.253878    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdj6c\" (UniqueName: \"kubernetes.io/projected/9f4103a1-5268-4298-a114-546110711bd8-kube-api-access-pdj6c\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8\" (UID: \"9f4103a1-5268-4298-a114-546110711bd8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8"
Sep 09 20:40:18.254037 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:18.253955    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f4103a1-5268-4298-a114-546110711bd8-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8\" (UID: \"9f4103a1-5268-4298-a114-546110711bd8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8"
Sep 09 20:40:18.254037 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:18.253993    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f4103a1-5268-4298-a114-546110711bd8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8\" (UID: \"9f4103a1-5268-4298-a114-546110711bd8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8"
Sep 09 20:40:18.354544 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:18.354516    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f4103a1-5268-4298-a114-546110711bd8-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8\" (UID: \"9f4103a1-5268-4298-a114-546110711bd8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8"
Sep 09 20:40:18.354659 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:18.354550    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f4103a1-5268-4298-a114-546110711bd8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8\" (UID: \"9f4103a1-5268-4298-a114-546110711bd8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8"
Sep 09 20:40:18.354659 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:18.354581    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f4103a1-5268-4298-a114-546110711bd8-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8\" (UID: \"9f4103a1-5268-4298-a114-546110711bd8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8"
Sep 09 20:40:18.354659 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:18.354603    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f4103a1-5268-4298-a114-546110711bd8-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8\" (UID: \"9f4103a1-5268-4298-a114-546110711bd8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8"
Sep 09 20:40:18.354659 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:18.354630    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdj6c\" (UniqueName: \"kubernetes.io/projected/9f4103a1-5268-4298-a114-546110711bd8-kube-api-access-pdj6c\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8\" (UID: \"9f4103a1-5268-4298-a114-546110711bd8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8"
Sep 09 20:40:18.354847 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:40:18.354710    2523 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-global-pass-predictor-serving-cert" not found
Sep 09 20:40:18.354847 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:40:18.354787    2523 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f4103a1-5268-4298-a114-546110711bd8-proxy-tls podName:9f4103a1-5268-4298-a114-546110711bd8 nodeName:}" failed. No retries permitted until 2026-09-09 20:40:18.85476786 +0000 UTC m=+2723.526786848 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9f4103a1-5268-4298-a114-546110711bd8-proxy-tls") pod "isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8" (UID: "9f4103a1-5268-4298-a114-546110711bd8") : secret "isvc-sklearn-s3-tls-global-pass-predictor-serving-cert" not found
Sep 09 20:40:18.355034 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:18.355016    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f4103a1-5268-4298-a114-546110711bd8-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8\" (UID: \"9f4103a1-5268-4298-a114-546110711bd8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8"
Sep 09 20:40:18.355362 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:18.355341    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f4103a1-5268-4298-a114-546110711bd8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8\" (UID: \"9f4103a1-5268-4298-a114-546110711bd8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8"
Sep 09 20:40:18.355404 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:18.355341    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f4103a1-5268-4298-a114-546110711bd8-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8\" (UID: \"9f4103a1-5268-4298-a114-546110711bd8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8"
Sep 09 20:40:18.362858 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:18.362833    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdj6c\" (UniqueName: \"kubernetes.io/projected/9f4103a1-5268-4298-a114-546110711bd8-kube-api-access-pdj6c\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8\" (UID: \"9f4103a1-5268-4298-a114-546110711bd8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8"
Sep 09 20:40:18.858592 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:18.858569    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f4103a1-5268-4298-a114-546110711bd8-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8\" (UID: \"9f4103a1-5268-4298-a114-546110711bd8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8"
Sep 09 20:40:18.860843 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:18.860815    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f4103a1-5268-4298-a114-546110711bd8-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8\" (UID: \"9f4103a1-5268-4298-a114-546110711bd8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8"
Sep 09 20:40:19.070827 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:19.070801    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8"
Sep 09 20:40:19.341097 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:19.341075    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8"]
Sep 09 20:40:19.343942 ip-10-0-132-226 kubenswrapper[2523]: W0909 20:40:19.343919    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f4103a1_5268_4298_a114_546110711bd8.slice/crio-9afdc7f317725cfed5a5e7672135517fdfa847d067eed07e02434f95bfc303c7 WatchSource:0}: Error finding container 9afdc7f317725cfed5a5e7672135517fdfa847d067eed07e02434f95bfc303c7: Status 404 returned error can't find the container with id 9afdc7f317725cfed5a5e7672135517fdfa847d067eed07e02434f95bfc303c7
Sep 09 20:40:20.269978 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:20.269934    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8" event={"ID":"9f4103a1-5268-4298-a114-546110711bd8","Type":"ContainerStarted","Data":"d3d4192fbc8eb98917a4340b56cbf67581011acb6602f2417b654c55ca39530b"}
Sep 09 20:40:20.270178 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:20.270016    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8" event={"ID":"9f4103a1-5268-4298-a114-546110711bd8","Type":"ContainerStarted","Data":"9afdc7f317725cfed5a5e7672135517fdfa847d067eed07e02434f95bfc303c7"}
Sep 09 20:40:21.275838 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:21.275804    2523 generic.go:356] "Generic (PLEG): container finished" podID="9f4103a1-5268-4298-a114-546110711bd8" containerID="d3d4192fbc8eb98917a4340b56cbf67581011acb6602f2417b654c55ca39530b" exitCode=0
Sep 09 20:40:21.276260 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:21.275880    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8" event={"ID":"9f4103a1-5268-4298-a114-546110711bd8","Type":"ContainerDied","Data":"d3d4192fbc8eb98917a4340b56cbf67581011acb6602f2417b654c55ca39530b"}
Sep 09 20:40:22.281681 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:22.281640    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8" event={"ID":"9f4103a1-5268-4298-a114-546110711bd8","Type":"ContainerStarted","Data":"947af86f1ae1c25b1c299dc6bdaa346e0759ec94dfd4834e991c107671484765"}
Sep 09 20:40:22.282092 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:22.281725    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8"
Sep 09 20:40:22.282092 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:22.281746    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8" event={"ID":"9f4103a1-5268-4298-a114-546110711bd8","Type":"ContainerStarted","Data":"08813782da2476dabaf5767e4cac080a5ac72c7f3d9312a07f677b914a55b8e6"}
Sep 09 20:40:22.282092 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:22.281859    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8"
Sep 09 20:40:22.283196 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:22.283170    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8" podUID="9f4103a1-5268-4298-a114-546110711bd8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 09 20:40:22.301692 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:22.301646    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8" podStartSLOduration=3.163191855 podStartE2EDuration="4.301635959s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.138444104s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:40:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:40:22.300345592 +0000 UTC m=+2726.972364589" watchObservedRunningTime="2026-09-09 20:40:22.301635959 +0000 UTC m=+2726.973654955"
Sep 09 20:40:23.285702 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:23.285610    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8" podUID="9f4103a1-5268-4298-a114-546110711bd8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 09 20:40:28.290133 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:28.290104    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8"
Sep 09 20:40:28.290571 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:28.290545    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8" podUID="9f4103a1-5268-4298-a114-546110711bd8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 09 20:40:38.291341 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:38.291301    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8" podUID="9f4103a1-5268-4298-a114-546110711bd8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 09 20:40:48.291380 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:48.291339    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8" podUID="9f4103a1-5268-4298-a114-546110711bd8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 09 20:40:58.291100 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:40:58.291056    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8" podUID="9f4103a1-5268-4298-a114-546110711bd8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 09 20:41:08.291367 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:08.291323    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8" podUID="9f4103a1-5268-4298-a114-546110711bd8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 09 20:41:18.290873 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:18.290790    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8" podUID="9f4103a1-5268-4298-a114-546110711bd8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 09 20:41:28.292322 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:28.292292    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8"
Sep 09 20:41:38.149855 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:38.149814    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8"]
Sep 09 20:41:38.150430 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:38.150109    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8" podUID="9f4103a1-5268-4298-a114-546110711bd8" containerName="kserve-container" containerID="cri-o://08813782da2476dabaf5767e4cac080a5ac72c7f3d9312a07f677b914a55b8e6" gracePeriod=30
Sep 09 20:41:38.150430 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:38.150162    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8" podUID="9f4103a1-5268-4298-a114-546110711bd8" containerName="kube-rbac-proxy" containerID="cri-o://947af86f1ae1c25b1c299dc6bdaa346e0759ec94dfd4834e991c107671484765" gracePeriod=30
Sep 09 20:41:38.285866 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:38.285827    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8" podUID="9f4103a1-5268-4298-a114-546110711bd8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.46:8643/healthz\": dial tcp 10.132.0.46:8643: connect: connection refused"
Sep 09 20:41:38.291211 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:38.291186    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8" podUID="9f4103a1-5268-4298-a114-546110711bd8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 09 20:41:38.556354 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:38.556284    2523 generic.go:356] "Generic (PLEG): container finished" podID="9f4103a1-5268-4298-a114-546110711bd8" containerID="947af86f1ae1c25b1c299dc6bdaa346e0759ec94dfd4834e991c107671484765" exitCode=2
Sep 09 20:41:38.556354 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:38.556336    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8" event={"ID":"9f4103a1-5268-4298-a114-546110711bd8","Type":"ContainerDied","Data":"947af86f1ae1c25b1c299dc6bdaa346e0759ec94dfd4834e991c107671484765"}
Sep 09 20:41:39.239354 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:39.239321    2523 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-zpwwk"]
Sep 09 20:41:39.242374 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:39.242358    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-zpwwk"
Sep 09 20:41:39.245291 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:39.245266    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
Sep 09 20:41:39.245478 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:39.245466    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
Sep 09 20:41:39.254865 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:39.254844    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-zpwwk"]
Sep 09 20:41:39.304385 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:39.304352    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf49da33-bd77-4caa-81c3-f49539912e1c-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-zpwwk\" (UID: \"bf49da33-bd77-4caa-81c3-f49539912e1c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-zpwwk"
Sep 09 20:41:39.304502 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:39.304402    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf49da33-bd77-4caa-81c3-f49539912e1c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-zpwwk\" (UID: \"bf49da33-bd77-4caa-81c3-f49539912e1c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-zpwwk"
Sep 09 20:41:39.304502 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:39.304480    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmjx6\" (UniqueName: \"kubernetes.io/projected/bf49da33-bd77-4caa-81c3-f49539912e1c-kube-api-access-mmjx6\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-zpwwk\" (UID: \"bf49da33-bd77-4caa-81c3-f49539912e1c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-zpwwk"
Sep 09 20:41:39.304580 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:39.304531    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf49da33-bd77-4caa-81c3-f49539912e1c-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-zpwwk\" (UID: \"bf49da33-bd77-4caa-81c3-f49539912e1c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-zpwwk"
Sep 09 20:41:39.405403 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:39.405370    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf49da33-bd77-4caa-81c3-f49539912e1c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-zpwwk\" (UID: \"bf49da33-bd77-4caa-81c3-f49539912e1c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-zpwwk"
Sep 09 20:41:39.405528 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:39.405417    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmjx6\" (UniqueName: \"kubernetes.io/projected/bf49da33-bd77-4caa-81c3-f49539912e1c-kube-api-access-mmjx6\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-zpwwk\" (UID: \"bf49da33-bd77-4caa-81c3-f49539912e1c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-zpwwk"
Sep 09 20:41:39.405528 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:39.405449    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf49da33-bd77-4caa-81c3-f49539912e1c-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-zpwwk\" (UID: \"bf49da33-bd77-4caa-81c3-f49539912e1c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-zpwwk"
Sep 09 20:41:39.405528 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:39.405469    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf49da33-bd77-4caa-81c3-f49539912e1c-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-zpwwk\" (UID: \"bf49da33-bd77-4caa-81c3-f49539912e1c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-zpwwk"
Sep 09 20:41:39.405868 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:39.405852    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf49da33-bd77-4caa-81c3-f49539912e1c-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-zpwwk\" (UID: \"bf49da33-bd77-4caa-81c3-f49539912e1c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-zpwwk"
Sep 09 20:41:39.406049 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:39.406029    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf49da33-bd77-4caa-81c3-f49539912e1c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-zpwwk\" (UID: \"bf49da33-bd77-4caa-81c3-f49539912e1c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-zpwwk"
Sep 09 20:41:39.408024 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:39.408002    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf49da33-bd77-4caa-81c3-f49539912e1c-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-zpwwk\" (UID: \"bf49da33-bd77-4caa-81c3-f49539912e1c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-zpwwk"
Sep 09 20:41:39.413472 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:39.413453    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmjx6\" (UniqueName: \"kubernetes.io/projected/bf49da33-bd77-4caa-81c3-f49539912e1c-kube-api-access-mmjx6\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-zpwwk\" (UID: \"bf49da33-bd77-4caa-81c3-f49539912e1c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-zpwwk"
Sep 09 20:41:39.552743 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:39.552680    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-zpwwk"
Sep 09 20:41:39.821892 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:39.821868    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-zpwwk"]
Sep 09 20:41:39.824427 ip-10-0-132-226 kubenswrapper[2523]: W0909 20:41:39.824399    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf49da33_bd77_4caa_81c3_f49539912e1c.slice/crio-8227b8f2ce6ec86545d6c97de1e8b57d34fb15ec609860c8de0515c268ef6a1a WatchSource:0}: Error finding container 8227b8f2ce6ec86545d6c97de1e8b57d34fb15ec609860c8de0515c268ef6a1a: Status 404 returned error can't find the container with id 8227b8f2ce6ec86545d6c97de1e8b57d34fb15ec609860c8de0515c268ef6a1a
Sep 09 20:41:40.565906 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:40.565874    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-zpwwk" event={"ID":"bf49da33-bd77-4caa-81c3-f49539912e1c","Type":"ContainerStarted","Data":"8ab976648306e5296632fbfdfaf23713ef92545fce214ea037aeeafa10ef04e1"}
Sep 09 20:41:40.566303 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:40.565930    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-zpwwk" event={"ID":"bf49da33-bd77-4caa-81c3-f49539912e1c","Type":"ContainerStarted","Data":"8227b8f2ce6ec86545d6c97de1e8b57d34fb15ec609860c8de0515c268ef6a1a"}
Sep 09 20:41:42.002952 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:42.002929    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8"
Sep 09 20:41:42.023893 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:42.023870    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f4103a1-5268-4298-a114-546110711bd8-cabundle-cert\") pod \"9f4103a1-5268-4298-a114-546110711bd8\" (UID: \"9f4103a1-5268-4298-a114-546110711bd8\") "
Sep 09 20:41:42.024000 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:42.023907    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f4103a1-5268-4298-a114-546110711bd8-kserve-provision-location\") pod \"9f4103a1-5268-4298-a114-546110711bd8\" (UID: \"9f4103a1-5268-4298-a114-546110711bd8\") "
Sep 09 20:41:42.024000 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:42.023949    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f4103a1-5268-4298-a114-546110711bd8-kube-rbac-proxy-sar-config\") pod \"9f4103a1-5268-4298-a114-546110711bd8\" (UID: \"9f4103a1-5268-4298-a114-546110711bd8\") "
Sep 09 20:41:42.024000 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:42.023982    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f4103a1-5268-4298-a114-546110711bd8-proxy-tls\") pod \"9f4103a1-5268-4298-a114-546110711bd8\" (UID: \"9f4103a1-5268-4298-a114-546110711bd8\") "
Sep 09 20:41:42.024129 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:42.024039    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pdj6c\" (UniqueName: \"kubernetes.io/projected/9f4103a1-5268-4298-a114-546110711bd8-kube-api-access-pdj6c\") pod \"9f4103a1-5268-4298-a114-546110711bd8\" (UID: \"9f4103a1-5268-4298-a114-546110711bd8\") "
Sep 09 20:41:42.024339 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:42.024297    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f4103a1-5268-4298-a114-546110711bd8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9f4103a1-5268-4298-a114-546110711bd8" (UID: "9f4103a1-5268-4298-a114-546110711bd8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:41:42.024339 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:42.024308    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f4103a1-5268-4298-a114-546110711bd8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9f4103a1-5268-4298-a114-546110711bd8" (UID: "9f4103a1-5268-4298-a114-546110711bd8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:41:42.024471 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:42.024442    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f4103a1-5268-4298-a114-546110711bd8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9f4103a1-5268-4298-a114-546110711bd8" (UID: "9f4103a1-5268-4298-a114-546110711bd8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:41:42.026670 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:42.026637    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f4103a1-5268-4298-a114-546110711bd8-kube-api-access-pdj6c" (OuterVolumeSpecName: "kube-api-access-pdj6c") pod "9f4103a1-5268-4298-a114-546110711bd8" (UID: "9f4103a1-5268-4298-a114-546110711bd8"). InnerVolumeSpecName "kube-api-access-pdj6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:41:42.026775 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:42.026682    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f4103a1-5268-4298-a114-546110711bd8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9f4103a1-5268-4298-a114-546110711bd8" (UID: "9f4103a1-5268-4298-a114-546110711bd8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:41:42.125195 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:42.125118    2523 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f4103a1-5268-4298-a114-546110711bd8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:41:42.125195 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:42.125181    2523 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f4103a1-5268-4298-a114-546110711bd8-proxy-tls\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:41:42.125195 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:42.125196    2523 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdj6c\" (UniqueName: \"kubernetes.io/projected/9f4103a1-5268-4298-a114-546110711bd8-kube-api-access-pdj6c\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:41:42.125354 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:42.125209    2523 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f4103a1-5268-4298-a114-546110711bd8-cabundle-cert\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:41:42.125354 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:42.125222    2523 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f4103a1-5268-4298-a114-546110711bd8-kserve-provision-location\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:41:42.575918 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:42.575887    2523 generic.go:356] "Generic (PLEG): container finished" podID="9f4103a1-5268-4298-a114-546110711bd8" containerID="08813782da2476dabaf5767e4cac080a5ac72c7f3d9312a07f677b914a55b8e6" exitCode=0
Sep 09 20:41:42.576031 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:42.575940    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8" event={"ID":"9f4103a1-5268-4298-a114-546110711bd8","Type":"ContainerDied","Data":"08813782da2476dabaf5767e4cac080a5ac72c7f3d9312a07f677b914a55b8e6"}
Sep 09 20:41:42.576031 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:42.575943    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8"
Sep 09 20:41:42.576031 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:42.575986    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8" event={"ID":"9f4103a1-5268-4298-a114-546110711bd8","Type":"ContainerDied","Data":"9afdc7f317725cfed5a5e7672135517fdfa847d067eed07e02434f95bfc303c7"}
Sep 09 20:41:42.576031 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:42.576012    2523 scope.go:117] "RemoveContainer" containerID="947af86f1ae1c25b1c299dc6bdaa346e0759ec94dfd4834e991c107671484765"
Sep 09 20:41:42.584803 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:42.584782    2523 scope.go:117] "RemoveContainer" containerID="08813782da2476dabaf5767e4cac080a5ac72c7f3d9312a07f677b914a55b8e6"
Sep 09 20:41:42.591974 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:42.591956    2523 scope.go:117] "RemoveContainer" containerID="d3d4192fbc8eb98917a4340b56cbf67581011acb6602f2417b654c55ca39530b"
Sep 09 20:41:42.597990 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:42.597971    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8"]
Sep 09 20:41:42.601634 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:42.601337    2523 scope.go:117] "RemoveContainer" containerID="947af86f1ae1c25b1c299dc6bdaa346e0759ec94dfd4834e991c107671484765"
Sep 09 20:41:42.601731 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:41:42.601675    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"947af86f1ae1c25b1c299dc6bdaa346e0759ec94dfd4834e991c107671484765\": container with ID starting with 947af86f1ae1c25b1c299dc6bdaa346e0759ec94dfd4834e991c107671484765 not found: ID does not exist" containerID="947af86f1ae1c25b1c299dc6bdaa346e0759ec94dfd4834e991c107671484765"
Sep 09 20:41:42.601731 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:42.601706    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"947af86f1ae1c25b1c299dc6bdaa346e0759ec94dfd4834e991c107671484765"} err="failed to get container status \"947af86f1ae1c25b1c299dc6bdaa346e0759ec94dfd4834e991c107671484765\": rpc error: code = NotFound desc = could not find container \"947af86f1ae1c25b1c299dc6bdaa346e0759ec94dfd4834e991c107671484765\": container with ID starting with 947af86f1ae1c25b1c299dc6bdaa346e0759ec94dfd4834e991c107671484765 not found: ID does not exist"
Sep 09 20:41:42.601731 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:42.601721    2523 scope.go:117] "RemoveContainer" containerID="08813782da2476dabaf5767e4cac080a5ac72c7f3d9312a07f677b914a55b8e6"
Sep 09 20:41:42.602079 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:41:42.602062    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08813782da2476dabaf5767e4cac080a5ac72c7f3d9312a07f677b914a55b8e6\": container with ID starting with 08813782da2476dabaf5767e4cac080a5ac72c7f3d9312a07f677b914a55b8e6 not found: ID does not exist" containerID="08813782da2476dabaf5767e4cac080a5ac72c7f3d9312a07f677b914a55b8e6"
Sep 09 20:41:42.602187 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:42.602085    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08813782da2476dabaf5767e4cac080a5ac72c7f3d9312a07f677b914a55b8e6"} err="failed to get container status \"08813782da2476dabaf5767e4cac080a5ac72c7f3d9312a07f677b914a55b8e6\": rpc error: code = NotFound desc = could not find container \"08813782da2476dabaf5767e4cac080a5ac72c7f3d9312a07f677b914a55b8e6\": container with ID starting with 08813782da2476dabaf5767e4cac080a5ac72c7f3d9312a07f677b914a55b8e6 not found: ID does not exist"
Sep 09 20:41:42.602187 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:42.602098    2523 scope.go:117] "RemoveContainer" containerID="d3d4192fbc8eb98917a4340b56cbf67581011acb6602f2417b654c55ca39530b"
Sep 09 20:41:42.602359 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:41:42.602345    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3d4192fbc8eb98917a4340b56cbf67581011acb6602f2417b654c55ca39530b\": container with ID starting with d3d4192fbc8eb98917a4340b56cbf67581011acb6602f2417b654c55ca39530b not found: ID does not exist" containerID="d3d4192fbc8eb98917a4340b56cbf67581011acb6602f2417b654c55ca39530b"
Sep 09 20:41:42.602420 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:42.602362    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3d4192fbc8eb98917a4340b56cbf67581011acb6602f2417b654c55ca39530b"} err="failed to get container status \"d3d4192fbc8eb98917a4340b56cbf67581011acb6602f2417b654c55ca39530b\": rpc error: code = NotFound desc = could not find container \"d3d4192fbc8eb98917a4340b56cbf67581011acb6602f2417b654c55ca39530b\": container with ID starting with d3d4192fbc8eb98917a4340b56cbf67581011acb6602f2417b654c55ca39530b not found: ID does not exist"
Sep 09 20:41:42.603258 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:42.603239    2523 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-649878f5d-jjvz8"]
Sep 09 20:41:43.806991 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:43.806961    2523 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f4103a1-5268-4298-a114-546110711bd8" path="/var/lib/kubelet/pods/9f4103a1-5268-4298-a114-546110711bd8/volumes"
Sep 09 20:41:46.593197 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:46.593111    2523 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-zpwwk_bf49da33-bd77-4caa-81c3-f49539912e1c/storage-initializer/0.log"
Sep 09 20:41:46.593197 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:46.593174    2523 generic.go:356] "Generic (PLEG): container finished" podID="bf49da33-bd77-4caa-81c3-f49539912e1c" containerID="8ab976648306e5296632fbfdfaf23713ef92545fce214ea037aeeafa10ef04e1" exitCode=1
Sep 09 20:41:46.593532 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:46.593231    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-zpwwk" event={"ID":"bf49da33-bd77-4caa-81c3-f49539912e1c","Type":"ContainerDied","Data":"8ab976648306e5296632fbfdfaf23713ef92545fce214ea037aeeafa10ef04e1"}
Sep 09 20:41:47.598853 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:47.598829    2523 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-zpwwk_bf49da33-bd77-4caa-81c3-f49539912e1c/storage-initializer/0.log"
Sep 09 20:41:47.599247 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:47.598924    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-zpwwk" event={"ID":"bf49da33-bd77-4caa-81c3-f49539912e1c","Type":"ContainerStarted","Data":"66a6a69cbbf6b01d42beb80153aadf9e1ede866082a56d2c3e649418a5e6f959"}
Sep 09 20:41:49.207237 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:49.207198    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-zpwwk"]
Sep 09 20:41:49.207677 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:49.207441    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-zpwwk" podUID="bf49da33-bd77-4caa-81c3-f49539912e1c" containerName="storage-initializer" containerID="cri-o://66a6a69cbbf6b01d42beb80153aadf9e1ede866082a56d2c3e649418a5e6f959" gracePeriod=30
Sep 09 20:41:50.367198 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:50.367162    2523 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd"]
Sep 09 20:41:50.367647 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:50.367550    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f4103a1-5268-4298-a114-546110711bd8" containerName="kserve-container"
Sep 09 20:41:50.367647 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:50.367568    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f4103a1-5268-4298-a114-546110711bd8" containerName="kserve-container"
Sep 09 20:41:50.367647 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:50.367599    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f4103a1-5268-4298-a114-546110711bd8" containerName="kube-rbac-proxy"
Sep 09 20:41:50.367647 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:50.367607    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f4103a1-5268-4298-a114-546110711bd8" containerName="kube-rbac-proxy"
Sep 09 20:41:50.367647 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:50.367624    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f4103a1-5268-4298-a114-546110711bd8" containerName="storage-initializer"
Sep 09 20:41:50.367647 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:50.367631    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f4103a1-5268-4298-a114-546110711bd8" containerName="storage-initializer"
Sep 09 20:41:50.367970 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:50.367699    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f4103a1-5268-4298-a114-546110711bd8" containerName="kube-rbac-proxy"
Sep 09 20:41:50.367970 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:50.367710    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f4103a1-5268-4298-a114-546110711bd8" containerName="kserve-container"
Sep 09 20:41:50.370873 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:50.370851    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd"
Sep 09 20:41:50.374420 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:50.374385    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
Sep 09 20:41:50.374525 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:50.374512    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
Sep 09 20:41:50.375399 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:50.375383    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:41:50.390584 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:50.390561    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd"]
Sep 09 20:41:50.482561 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:50.482529    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03bf34c2-e750-4f7b-b200-fc4862eb4976-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd\" (UID: \"03bf34c2-e750-4f7b-b200-fc4862eb4976\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd"
Sep 09 20:41:50.482712 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:50.482570    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsrhq\" (UniqueName: \"kubernetes.io/projected/03bf34c2-e750-4f7b-b200-fc4862eb4976-kube-api-access-fsrhq\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd\" (UID: \"03bf34c2-e750-4f7b-b200-fc4862eb4976\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd"
Sep 09 20:41:50.482712 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:50.482629    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03bf34c2-e750-4f7b-b200-fc4862eb4976-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd\" (UID: \"03bf34c2-e750-4f7b-b200-fc4862eb4976\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd"
Sep 09 20:41:50.482712 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:50.482690    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03bf34c2-e750-4f7b-b200-fc4862eb4976-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd\" (UID: \"03bf34c2-e750-4f7b-b200-fc4862eb4976\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd"
Sep 09 20:41:50.482820 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:50.482710    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/03bf34c2-e750-4f7b-b200-fc4862eb4976-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd\" (UID: \"03bf34c2-e750-4f7b-b200-fc4862eb4976\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd"
Sep 09 20:41:50.583448 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:50.583423    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03bf34c2-e750-4f7b-b200-fc4862eb4976-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd\" (UID: \"03bf34c2-e750-4f7b-b200-fc4862eb4976\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd"
Sep 09 20:41:50.583601 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:50.583469    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03bf34c2-e750-4f7b-b200-fc4862eb4976-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd\" (UID: \"03bf34c2-e750-4f7b-b200-fc4862eb4976\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd"
Sep 09 20:41:50.583601 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:50.583494    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/03bf34c2-e750-4f7b-b200-fc4862eb4976-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd\" (UID: \"03bf34c2-e750-4f7b-b200-fc4862eb4976\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd"
Sep 09 20:41:50.583601 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:50.583520    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03bf34c2-e750-4f7b-b200-fc4862eb4976-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd\" (UID: \"03bf34c2-e750-4f7b-b200-fc4862eb4976\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd"
Sep 09 20:41:50.583601 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:50.583546    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsrhq\" (UniqueName: \"kubernetes.io/projected/03bf34c2-e750-4f7b-b200-fc4862eb4976-kube-api-access-fsrhq\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd\" (UID: \"03bf34c2-e750-4f7b-b200-fc4862eb4976\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd"
Sep 09 20:41:50.583864 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:50.583845    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03bf34c2-e750-4f7b-b200-fc4862eb4976-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd\" (UID: \"03bf34c2-e750-4f7b-b200-fc4862eb4976\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd"
Sep 09 20:41:50.584145 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:50.584124    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/03bf34c2-e750-4f7b-b200-fc4862eb4976-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd\" (UID: \"03bf34c2-e750-4f7b-b200-fc4862eb4976\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd"
Sep 09 20:41:50.584249 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:50.584192    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03bf34c2-e750-4f7b-b200-fc4862eb4976-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd\" (UID: \"03bf34c2-e750-4f7b-b200-fc4862eb4976\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd"
Sep 09 20:41:50.586165 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:50.586128    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03bf34c2-e750-4f7b-b200-fc4862eb4976-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd\" (UID: \"03bf34c2-e750-4f7b-b200-fc4862eb4976\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd"
Sep 09 20:41:50.591615 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:50.591595    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsrhq\" (UniqueName: \"kubernetes.io/projected/03bf34c2-e750-4f7b-b200-fc4862eb4976-kube-api-access-fsrhq\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd\" (UID: \"03bf34c2-e750-4f7b-b200-fc4862eb4976\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd"
Sep 09 20:41:50.680590 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:50.680537    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd"
Sep 09 20:41:50.953971 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:50.953904    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd"]
Sep 09 20:41:50.957594 ip-10-0-132-226 kubenswrapper[2523]: W0909 20:41:50.957564    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03bf34c2_e750_4f7b_b200_fc4862eb4976.slice/crio-53fbd92783b9903250f17d7a5971613c27c95ebe35a2481227b862f5cbe2fc27 WatchSource:0}: Error finding container 53fbd92783b9903250f17d7a5971613c27c95ebe35a2481227b862f5cbe2fc27: Status 404 returned error can't find the container with id 53fbd92783b9903250f17d7a5971613c27c95ebe35a2481227b862f5cbe2fc27
Sep 09 20:41:51.336631 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:51.336608    2523 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-zpwwk_bf49da33-bd77-4caa-81c3-f49539912e1c/storage-initializer/1.log"
Sep 09 20:41:51.337075 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:51.337055    2523 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-zpwwk_bf49da33-bd77-4caa-81c3-f49539912e1c/storage-initializer/0.log"
Sep 09 20:41:51.339421 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:51.339403    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-zpwwk"
Sep 09 20:41:51.389410 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:51.389390    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmjx6\" (UniqueName: \"kubernetes.io/projected/bf49da33-bd77-4caa-81c3-f49539912e1c-kube-api-access-mmjx6\") pod \"bf49da33-bd77-4caa-81c3-f49539912e1c\" (UID: \"bf49da33-bd77-4caa-81c3-f49539912e1c\") "
Sep 09 20:41:51.389782 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:51.389425    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf49da33-bd77-4caa-81c3-f49539912e1c-kube-rbac-proxy-sar-config\") pod \"bf49da33-bd77-4caa-81c3-f49539912e1c\" (UID: \"bf49da33-bd77-4caa-81c3-f49539912e1c\") "
Sep 09 20:41:51.389782 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:51.389466    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf49da33-bd77-4caa-81c3-f49539912e1c-kserve-provision-location\") pod \"bf49da33-bd77-4caa-81c3-f49539912e1c\" (UID: \"bf49da33-bd77-4caa-81c3-f49539912e1c\") "
Sep 09 20:41:51.389782 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:51.389506    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf49da33-bd77-4caa-81c3-f49539912e1c-proxy-tls\") pod \"bf49da33-bd77-4caa-81c3-f49539912e1c\" (UID: \"bf49da33-bd77-4caa-81c3-f49539912e1c\") "
Sep 09 20:41:51.389782 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:51.389684    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf49da33-bd77-4caa-81c3-f49539912e1c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bf49da33-bd77-4caa-81c3-f49539912e1c" (UID: "bf49da33-bd77-4caa-81c3-f49539912e1c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:41:51.389782 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:51.389736    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf49da33-bd77-4caa-81c3-f49539912e1c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bf49da33-bd77-4caa-81c3-f49539912e1c" (UID: "bf49da33-bd77-4caa-81c3-f49539912e1c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:41:51.391630 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:51.391609    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf49da33-bd77-4caa-81c3-f49539912e1c-kube-api-access-mmjx6" (OuterVolumeSpecName: "kube-api-access-mmjx6") pod "bf49da33-bd77-4caa-81c3-f49539912e1c" (UID: "bf49da33-bd77-4caa-81c3-f49539912e1c"). InnerVolumeSpecName "kube-api-access-mmjx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:41:51.391802 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:51.391784    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf49da33-bd77-4caa-81c3-f49539912e1c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bf49da33-bd77-4caa-81c3-f49539912e1c" (UID: "bf49da33-bd77-4caa-81c3-f49539912e1c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:41:51.490590 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:51.490537    2523 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmjx6\" (UniqueName: \"kubernetes.io/projected/bf49da33-bd77-4caa-81c3-f49539912e1c-kube-api-access-mmjx6\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:41:51.490590 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:51.490557    2523 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf49da33-bd77-4caa-81c3-f49539912e1c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:41:51.490590 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:51.490567    2523 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf49da33-bd77-4caa-81c3-f49539912e1c-kserve-provision-location\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:41:51.490590 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:51.490575    2523 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf49da33-bd77-4caa-81c3-f49539912e1c-proxy-tls\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:41:51.620165 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:51.620123    2523 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-zpwwk_bf49da33-bd77-4caa-81c3-f49539912e1c/storage-initializer/1.log"
Sep 09 20:41:51.620595 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:51.620580    2523 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-zpwwk_bf49da33-bd77-4caa-81c3-f49539912e1c/storage-initializer/0.log"
Sep 09 20:41:51.620660 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:51.620618    2523 generic.go:356] "Generic (PLEG): container finished" podID="bf49da33-bd77-4caa-81c3-f49539912e1c" containerID="66a6a69cbbf6b01d42beb80153aadf9e1ede866082a56d2c3e649418a5e6f959" exitCode=1
Sep 09 20:41:51.620712 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:51.620680    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-zpwwk"
Sep 09 20:41:51.620770 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:51.620711    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-zpwwk" event={"ID":"bf49da33-bd77-4caa-81c3-f49539912e1c","Type":"ContainerDied","Data":"66a6a69cbbf6b01d42beb80153aadf9e1ede866082a56d2c3e649418a5e6f959"}
Sep 09 20:41:51.620831 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:51.620822    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-zpwwk" event={"ID":"bf49da33-bd77-4caa-81c3-f49539912e1c","Type":"ContainerDied","Data":"8227b8f2ce6ec86545d6c97de1e8b57d34fb15ec609860c8de0515c268ef6a1a"}
Sep 09 20:41:51.620886 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:51.620863    2523 scope.go:117] "RemoveContainer" containerID="66a6a69cbbf6b01d42beb80153aadf9e1ede866082a56d2c3e649418a5e6f959"
Sep 09 20:41:51.622660 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:51.622633    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd" event={"ID":"03bf34c2-e750-4f7b-b200-fc4862eb4976","Type":"ContainerStarted","Data":"778153d644e971045ffb50cdfa2ef1ee2bfcad9dc1b3d4276f19f4565fc5e0d9"}
Sep 09 20:41:51.622777 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:51.622691    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd" event={"ID":"03bf34c2-e750-4f7b-b200-fc4862eb4976","Type":"ContainerStarted","Data":"53fbd92783b9903250f17d7a5971613c27c95ebe35a2481227b862f5cbe2fc27"}
Sep 09 20:41:51.629637 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:51.629617    2523 scope.go:117] "RemoveContainer" containerID="8ab976648306e5296632fbfdfaf23713ef92545fce214ea037aeeafa10ef04e1"
Sep 09 20:41:51.636854 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:51.636840    2523 scope.go:117] "RemoveContainer" containerID="66a6a69cbbf6b01d42beb80153aadf9e1ede866082a56d2c3e649418a5e6f959"
Sep 09 20:41:51.637089 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:41:51.637071    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66a6a69cbbf6b01d42beb80153aadf9e1ede866082a56d2c3e649418a5e6f959\": container with ID starting with 66a6a69cbbf6b01d42beb80153aadf9e1ede866082a56d2c3e649418a5e6f959 not found: ID does not exist" containerID="66a6a69cbbf6b01d42beb80153aadf9e1ede866082a56d2c3e649418a5e6f959"
Sep 09 20:41:51.637166 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:51.637094    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66a6a69cbbf6b01d42beb80153aadf9e1ede866082a56d2c3e649418a5e6f959"} err="failed to get container status \"66a6a69cbbf6b01d42beb80153aadf9e1ede866082a56d2c3e649418a5e6f959\": rpc error: code = NotFound desc = could not find container \"66a6a69cbbf6b01d42beb80153aadf9e1ede866082a56d2c3e649418a5e6f959\": container with ID starting with 66a6a69cbbf6b01d42beb80153aadf9e1ede866082a56d2c3e649418a5e6f959 not found: ID does not exist"
Sep 09 20:41:51.637166 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:51.637108    2523 scope.go:117] "RemoveContainer" containerID="8ab976648306e5296632fbfdfaf23713ef92545fce214ea037aeeafa10ef04e1"
Sep 09 20:41:51.637383 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:41:51.637366    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ab976648306e5296632fbfdfaf23713ef92545fce214ea037aeeafa10ef04e1\": container with ID starting with 8ab976648306e5296632fbfdfaf23713ef92545fce214ea037aeeafa10ef04e1 not found: ID does not exist" containerID="8ab976648306e5296632fbfdfaf23713ef92545fce214ea037aeeafa10ef04e1"
Sep 09 20:41:51.637433 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:51.637389    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ab976648306e5296632fbfdfaf23713ef92545fce214ea037aeeafa10ef04e1"} err="failed to get container status \"8ab976648306e5296632fbfdfaf23713ef92545fce214ea037aeeafa10ef04e1\": rpc error: code = NotFound desc = could not find container \"8ab976648306e5296632fbfdfaf23713ef92545fce214ea037aeeafa10ef04e1\": container with ID starting with 8ab976648306e5296632fbfdfaf23713ef92545fce214ea037aeeafa10ef04e1 not found: ID does not exist"
Sep 09 20:41:51.677777 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:51.677737    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-zpwwk"]
Sep 09 20:41:51.680757 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:51.680736    2523 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7447f98cd6-zpwwk"]
Sep 09 20:41:51.807346 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:51.807269    2523 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf49da33-bd77-4caa-81c3-f49539912e1c" path="/var/lib/kubelet/pods/bf49da33-bd77-4caa-81c3-f49539912e1c/volumes"
Sep 09 20:41:52.628234 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:52.628201    2523 generic.go:356] "Generic (PLEG): container finished" podID="03bf34c2-e750-4f7b-b200-fc4862eb4976" containerID="778153d644e971045ffb50cdfa2ef1ee2bfcad9dc1b3d4276f19f4565fc5e0d9" exitCode=0
Sep 09 20:41:52.628570 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:52.628253    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd" event={"ID":"03bf34c2-e750-4f7b-b200-fc4862eb4976","Type":"ContainerDied","Data":"778153d644e971045ffb50cdfa2ef1ee2bfcad9dc1b3d4276f19f4565fc5e0d9"}
Sep 09 20:41:53.636970 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:53.636941    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd" event={"ID":"03bf34c2-e750-4f7b-b200-fc4862eb4976","Type":"ContainerStarted","Data":"db6593dfe38b20151e98310049631736066a171dd355a77c4121288f0c0e8cd1"}
Sep 09 20:41:53.637413 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:53.636994    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd" event={"ID":"03bf34c2-e750-4f7b-b200-fc4862eb4976","Type":"ContainerStarted","Data":"e5e1cd281ac7d7a7323101f15d9ba0741245b404af500f80973142016048b6fd"}
Sep 09 20:41:53.637413 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:53.637073    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd"
Sep 09 20:41:53.658957 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:53.658914    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd" podStartSLOduration=2.62202325 podStartE2EDuration="3.658902743s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.036879493s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:41:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:41:53.65758043 +0000 UTC m=+2818.329599451" watchObservedRunningTime="2026-09-09 20:41:53.658902743 +0000 UTC m=+2818.330921739"
Sep 09 20:41:54.640971 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:54.640939    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd"
Sep 09 20:41:54.642259 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:54.642229    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd" podUID="03bf34c2-e750-4f7b-b200-fc4862eb4976" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 09 20:41:55.644945 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:41:55.644894    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd" podUID="03bf34c2-e750-4f7b-b200-fc4862eb4976" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 09 20:42:00.650214 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:42:00.650176    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd"
Sep 09 20:42:00.650765 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:42:00.650739    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd" podUID="03bf34c2-e750-4f7b-b200-fc4862eb4976" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 09 20:42:10.650934 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:42:10.650888    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd" podUID="03bf34c2-e750-4f7b-b200-fc4862eb4976" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 09 20:42:20.653646 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:42:20.653598    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd" podUID="03bf34c2-e750-4f7b-b200-fc4862eb4976" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 09 20:42:30.651209 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:42:30.651144    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd" podUID="03bf34c2-e750-4f7b-b200-fc4862eb4976" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 09 20:42:40.651326 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:42:40.651282    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd" podUID="03bf34c2-e750-4f7b-b200-fc4862eb4976" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 09 20:42:41.075872 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:42:41.075840    2523 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4ls7b"]
Sep 09 20:42:41.076121 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:42:41.076109    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf49da33-bd77-4caa-81c3-f49539912e1c" containerName="storage-initializer"
Sep 09 20:42:41.076203 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:42:41.076123    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf49da33-bd77-4caa-81c3-f49539912e1c" containerName="storage-initializer"
Sep 09 20:42:41.076203 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:42:41.076201    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf49da33-bd77-4caa-81c3-f49539912e1c" containerName="storage-initializer"
Sep 09 20:42:41.076281 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:42:41.076208    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf49da33-bd77-4caa-81c3-f49539912e1c" containerName="storage-initializer"
Sep 09 20:42:41.076281 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:42:41.076255    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf49da33-bd77-4caa-81c3-f49539912e1c" containerName="storage-initializer"
Sep 09 20:42:41.076281 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:42:41.076261    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf49da33-bd77-4caa-81c3-f49539912e1c" containerName="storage-initializer"
Sep 09 20:42:41.079128 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:42:41.079111    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4ls7b"
Sep 09 20:42:41.081569 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:42:41.081549    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\""
Sep 09 20:42:41.081707 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:42:41.081604    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
Sep 09 20:42:41.081828 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:42:41.081813    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 20:42:41.091103 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:42:41.091079    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4ls7b"]
Sep 09 20:42:41.231940 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:42:41.231911    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/228a0c04-4551-4a79-b3ee-7b8d4cfefa81-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-4ls7b\" (UID: \"228a0c04-4551-4a79-b3ee-7b8d4cfefa81\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4ls7b"
Sep 09 20:42:41.232057 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:42:41.231945    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/228a0c04-4551-4a79-b3ee-7b8d4cfefa81-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-4ls7b\" (UID: \"228a0c04-4551-4a79-b3ee-7b8d4cfefa81\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4ls7b"
Sep 09 20:42:41.232057 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:42:41.232017    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/228a0c04-4551-4a79-b3ee-7b8d4cfefa81-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-4ls7b\" (UID: \"228a0c04-4551-4a79-b3ee-7b8d4cfefa81\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4ls7b"
Sep 09 20:42:41.232057 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:42:41.232037    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8pxd\" (UniqueName: \"kubernetes.io/projected/228a0c04-4551-4a79-b3ee-7b8d4cfefa81-kube-api-access-f8pxd\") pod \"isvc-paddle-predictor-6d5c9fd4f6-4ls7b\" (UID: \"228a0c04-4551-4a79-b3ee-7b8d4cfefa81\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4ls7b"
Sep 09 20:42:41.232205 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:42:41.232059    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/228a0c04-4551-4a79-b3ee-7b8d4cfefa81-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-4ls7b\" (UID: \"228a0c04-4551-4a79-b3ee-7b8d4cfefa81\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4ls7b"
Sep 09 20:42:41.332598 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:42:41.332531    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/228a0c04-4551-4a79-b3ee-7b8d4cfefa81-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-4ls7b\" (UID: \"228a0c04-4551-4a79-b3ee-7b8d4cfefa81\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4ls7b"
Sep 09 20:42:41.332598 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:42:41.332576    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8pxd\" (UniqueName: \"kubernetes.io/projected/228a0c04-4551-4a79-b3ee-7b8d4cfefa81-kube-api-access-f8pxd\") pod \"isvc-paddle-predictor-6d5c9fd4f6-4ls7b\" (UID: \"228a0c04-4551-4a79-b3ee-7b8d4cfefa81\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4ls7b"
Sep 09 20:42:41.332773 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:42:41.332610    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/228a0c04-4551-4a79-b3ee-7b8d4cfefa81-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-4ls7b\" (UID: \"228a0c04-4551-4a79-b3ee-7b8d4cfefa81\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4ls7b"
Sep 09 20:42:41.332832 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:42:41.332781    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/228a0c04-4551-4a79-b3ee-7b8d4cfefa81-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-4ls7b\" (UID: \"228a0c04-4551-4a79-b3ee-7b8d4cfefa81\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4ls7b"
Sep 09 20:42:41.332832 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:42:41.332809    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/228a0c04-4551-4a79-b3ee-7b8d4cfefa81-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-4ls7b\" (UID: \"228a0c04-4551-4a79-b3ee-7b8d4cfefa81\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4ls7b"
Sep 09 20:42:41.333175 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:42:41.333123    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/228a0c04-4551-4a79-b3ee-7b8d4cfefa81-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-4ls7b\" (UID: \"228a0c04-4551-4a79-b3ee-7b8d4cfefa81\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4ls7b"
Sep 09 20:42:41.333508 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:42:41.333475    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/228a0c04-4551-4a79-b3ee-7b8d4cfefa81-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-4ls7b\" (UID: \"228a0c04-4551-4a79-b3ee-7b8d4cfefa81\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4ls7b"
Sep 09 20:42:41.333578 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:42:41.333526    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/228a0c04-4551-4a79-b3ee-7b8d4cfefa81-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-4ls7b\" (UID: \"228a0c04-4551-4a79-b3ee-7b8d4cfefa81\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4ls7b"
Sep 09 20:42:41.335144 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:42:41.335124    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/228a0c04-4551-4a79-b3ee-7b8d4cfefa81-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-4ls7b\" (UID: \"228a0c04-4551-4a79-b3ee-7b8d4cfefa81\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4ls7b"
Sep 09 20:42:41.341247 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:42:41.341225    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8pxd\" (UniqueName: \"kubernetes.io/projected/228a0c04-4551-4a79-b3ee-7b8d4cfefa81-kube-api-access-f8pxd\") pod \"isvc-paddle-predictor-6d5c9fd4f6-4ls7b\" (UID: \"228a0c04-4551-4a79-b3ee-7b8d4cfefa81\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4ls7b"
Sep 09 20:42:41.388737 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:42:41.388714    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4ls7b"
Sep 09 20:42:41.659716 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:42:41.659688    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4ls7b"]
Sep 09 20:42:41.662435 ip-10-0-132-226 kubenswrapper[2523]: W0909 20:42:41.662402    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod228a0c04_4551_4a79_b3ee_7b8d4cfefa81.slice/crio-660ae7e751274ce2d7dbf7f3f112a925c1ffcec114ab959364026ca285cdc5a6 WatchSource:0}: Error finding container 660ae7e751274ce2d7dbf7f3f112a925c1ffcec114ab959364026ca285cdc5a6: Status 404 returned error can't find the container with id 660ae7e751274ce2d7dbf7f3f112a925c1ffcec114ab959364026ca285cdc5a6
Sep 09 20:42:41.821089 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:42:41.821061    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4ls7b" event={"ID":"228a0c04-4551-4a79-b3ee-7b8d4cfefa81","Type":"ContainerStarted","Data":"e9b3f7f94564d501eb908129b0026277d9d7b0ef2b54d6de864f3725a61542e2"}
Sep 09 20:42:41.821260 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:42:41.821117    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4ls7b" event={"ID":"228a0c04-4551-4a79-b3ee-7b8d4cfefa81","Type":"ContainerStarted","Data":"660ae7e751274ce2d7dbf7f3f112a925c1ffcec114ab959364026ca285cdc5a6"}
Sep 09 20:42:47.844100 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:42:47.844068    2523 generic.go:356] "Generic (PLEG): container finished" podID="228a0c04-4551-4a79-b3ee-7b8d4cfefa81" containerID="e9b3f7f94564d501eb908129b0026277d9d7b0ef2b54d6de864f3725a61542e2" exitCode=0
Sep 09 20:42:47.844501 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:42:47.844135    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4ls7b" event={"ID":"228a0c04-4551-4a79-b3ee-7b8d4cfefa81","Type":"ContainerDied","Data":"e9b3f7f94564d501eb908129b0026277d9d7b0ef2b54d6de864f3725a61542e2"}
Sep 09 20:42:47.845348 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:42:47.845338    2523 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:42:50.651211 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:42:50.651173    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd" podUID="03bf34c2-e750-4f7b-b200-fc4862eb4976" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 09 20:42:59.897097 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:42:59.897063    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4ls7b" event={"ID":"228a0c04-4551-4a79-b3ee-7b8d4cfefa81","Type":"ContainerStarted","Data":"9f3dd715df2af0b388b8a08d5f725ae2c939985114eaf14797d889564f8a5528"}
Sep 09 20:42:59.897535 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:42:59.897137    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4ls7b" event={"ID":"228a0c04-4551-4a79-b3ee-7b8d4cfefa81","Type":"ContainerStarted","Data":"eec862445f68efa9c1847de9a42624e657e659b5d47c9280e6c30925e8f85911"}
Sep 09 20:42:59.897535 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:42:59.897361    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4ls7b"
Sep 09 20:42:59.916591 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:42:59.916553    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4ls7b" podStartSLOduration=1.962196562 podStartE2EDuration="18.91654019s" totalImagesPullingTime="11.140586026s" totalInitContainerRuntime="5.813757602s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:42:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:42:59.91427239 +0000 UTC m=+2884.586291378" watchObservedRunningTime="2026-09-09 20:42:59.91654019 +0000 UTC m=+2884.588559187"
Sep 09 20:43:00.650723 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:00.650685    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd" podUID="03bf34c2-e750-4f7b-b200-fc4862eb4976" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 09 20:43:00.900843 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:00.900768    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4ls7b"
Sep 09 20:43:00.902063 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:00.902034    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4ls7b" podUID="228a0c04-4551-4a79-b3ee-7b8d4cfefa81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 09 20:43:01.904794 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:01.904749    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4ls7b" podUID="228a0c04-4551-4a79-b3ee-7b8d4cfefa81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 09 20:43:06.909425 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:06.909393    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4ls7b"
Sep 09 20:43:06.909874 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:06.909785    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4ls7b" podUID="228a0c04-4551-4a79-b3ee-7b8d4cfefa81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 09 20:43:10.652086 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:10.652059    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd"
Sep 09 20:43:16.909966 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:16.909929    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4ls7b" podUID="228a0c04-4551-4a79-b3ee-7b8d4cfefa81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 09 20:43:20.361343 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:20.361312    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd"]
Sep 09 20:43:20.361728 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:20.361545    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd" podUID="03bf34c2-e750-4f7b-b200-fc4862eb4976" containerName="kserve-container" containerID="cri-o://e5e1cd281ac7d7a7323101f15d9ba0741245b404af500f80973142016048b6fd" gracePeriod=30
Sep 09 20:43:20.361728 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:20.361571    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd" podUID="03bf34c2-e750-4f7b-b200-fc4862eb4976" containerName="kube-rbac-proxy" containerID="cri-o://db6593dfe38b20151e98310049631736066a171dd355a77c4121288f0c0e8cd1" gracePeriod=30
Sep 09 20:43:20.645429 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:20.645324    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd" podUID="03bf34c2-e750-4f7b-b200-fc4862eb4976" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused"
Sep 09 20:43:20.650782 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:20.650731    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd" podUID="03bf34c2-e750-4f7b-b200-fc4862eb4976" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 09 20:43:20.975462 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:20.975385    2523 generic.go:356] "Generic (PLEG): container finished" podID="03bf34c2-e750-4f7b-b200-fc4862eb4976" containerID="db6593dfe38b20151e98310049631736066a171dd355a77c4121288f0c0e8cd1" exitCode=2
Sep 09 20:43:20.975603 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:20.975460    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd" event={"ID":"03bf34c2-e750-4f7b-b200-fc4862eb4976","Type":"ContainerDied","Data":"db6593dfe38b20151e98310049631736066a171dd355a77c4121288f0c0e8cd1"}
Sep 09 20:43:24.299526 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:24.299503    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd"
Sep 09 20:43:24.344767 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:24.344742    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fsrhq\" (UniqueName: \"kubernetes.io/projected/03bf34c2-e750-4f7b-b200-fc4862eb4976-kube-api-access-fsrhq\") pod \"03bf34c2-e750-4f7b-b200-fc4862eb4976\" (UID: \"03bf34c2-e750-4f7b-b200-fc4862eb4976\") "
Sep 09 20:43:24.344886 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:24.344777    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03bf34c2-e750-4f7b-b200-fc4862eb4976-kube-rbac-proxy-sar-config\") pod \"03bf34c2-e750-4f7b-b200-fc4862eb4976\" (UID: \"03bf34c2-e750-4f7b-b200-fc4862eb4976\") "
Sep 09 20:43:24.344886 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:24.344823    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03bf34c2-e750-4f7b-b200-fc4862eb4976-proxy-tls\") pod \"03bf34c2-e750-4f7b-b200-fc4862eb4976\" (UID: \"03bf34c2-e750-4f7b-b200-fc4862eb4976\") "
Sep 09 20:43:24.344965 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:24.344882    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/03bf34c2-e750-4f7b-b200-fc4862eb4976-cabundle-cert\") pod \"03bf34c2-e750-4f7b-b200-fc4862eb4976\" (UID: \"03bf34c2-e750-4f7b-b200-fc4862eb4976\") "
Sep 09 20:43:24.344965 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:24.344927    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03bf34c2-e750-4f7b-b200-fc4862eb4976-kserve-provision-location\") pod \"03bf34c2-e750-4f7b-b200-fc4862eb4976\" (UID: \"03bf34c2-e750-4f7b-b200-fc4862eb4976\") "
Sep 09 20:43:24.345206 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:24.345182    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03bf34c2-e750-4f7b-b200-fc4862eb4976-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "03bf34c2-e750-4f7b-b200-fc4862eb4976" (UID: "03bf34c2-e750-4f7b-b200-fc4862eb4976"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:43:24.345347 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:24.345326    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03bf34c2-e750-4f7b-b200-fc4862eb4976-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "03bf34c2-e750-4f7b-b200-fc4862eb4976" (UID: "03bf34c2-e750-4f7b-b200-fc4862eb4976"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:43:24.345408 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:24.345365    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03bf34c2-e750-4f7b-b200-fc4862eb4976-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "03bf34c2-e750-4f7b-b200-fc4862eb4976" (UID: "03bf34c2-e750-4f7b-b200-fc4862eb4976"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:43:24.346920 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:24.346899    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03bf34c2-e750-4f7b-b200-fc4862eb4976-kube-api-access-fsrhq" (OuterVolumeSpecName: "kube-api-access-fsrhq") pod "03bf34c2-e750-4f7b-b200-fc4862eb4976" (UID: "03bf34c2-e750-4f7b-b200-fc4862eb4976"). InnerVolumeSpecName "kube-api-access-fsrhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:43:24.347075 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:24.347060    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03bf34c2-e750-4f7b-b200-fc4862eb4976-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "03bf34c2-e750-4f7b-b200-fc4862eb4976" (UID: "03bf34c2-e750-4f7b-b200-fc4862eb4976"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:43:24.445779 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:24.445710    2523 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03bf34c2-e750-4f7b-b200-fc4862eb4976-kserve-provision-location\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:43:24.445779 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:24.445742    2523 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsrhq\" (UniqueName: \"kubernetes.io/projected/03bf34c2-e750-4f7b-b200-fc4862eb4976-kube-api-access-fsrhq\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:43:24.445779 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:24.445756    2523 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03bf34c2-e750-4f7b-b200-fc4862eb4976-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:43:24.445779 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:24.445768    2523 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03bf34c2-e750-4f7b-b200-fc4862eb4976-proxy-tls\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:43:24.445779 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:24.445781    2523 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/03bf34c2-e750-4f7b-b200-fc4862eb4976-cabundle-cert\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:43:24.991525 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:24.991488    2523 generic.go:356] "Generic (PLEG): container finished" podID="03bf34c2-e750-4f7b-b200-fc4862eb4976" containerID="e5e1cd281ac7d7a7323101f15d9ba0741245b404af500f80973142016048b6fd" exitCode=0
Sep 09 20:43:24.991705 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:24.991552    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd" event={"ID":"03bf34c2-e750-4f7b-b200-fc4862eb4976","Type":"ContainerDied","Data":"e5e1cd281ac7d7a7323101f15d9ba0741245b404af500f80973142016048b6fd"}
Sep 09 20:43:24.991705 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:24.991593    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd" event={"ID":"03bf34c2-e750-4f7b-b200-fc4862eb4976","Type":"ContainerDied","Data":"53fbd92783b9903250f17d7a5971613c27c95ebe35a2481227b862f5cbe2fc27"}
Sep 09 20:43:24.991705 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:24.991552    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd"
Sep 09 20:43:24.991705 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:24.991635    2523 scope.go:117] "RemoveContainer" containerID="db6593dfe38b20151e98310049631736066a171dd355a77c4121288f0c0e8cd1"
Sep 09 20:43:25.000067 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:25.000048    2523 scope.go:117] "RemoveContainer" containerID="e5e1cd281ac7d7a7323101f15d9ba0741245b404af500f80973142016048b6fd"
Sep 09 20:43:25.007301 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:25.007286    2523 scope.go:117] "RemoveContainer" containerID="778153d644e971045ffb50cdfa2ef1ee2bfcad9dc1b3d4276f19f4565fc5e0d9"
Sep 09 20:43:25.013195 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:25.013175    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd"]
Sep 09 20:43:25.014555 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:25.014535    2523 scope.go:117] "RemoveContainer" containerID="db6593dfe38b20151e98310049631736066a171dd355a77c4121288f0c0e8cd1"
Sep 09 20:43:25.014923 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:43:25.014900    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db6593dfe38b20151e98310049631736066a171dd355a77c4121288f0c0e8cd1\": container with ID starting with db6593dfe38b20151e98310049631736066a171dd355a77c4121288f0c0e8cd1 not found: ID does not exist" containerID="db6593dfe38b20151e98310049631736066a171dd355a77c4121288f0c0e8cd1"
Sep 09 20:43:25.015108 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:25.014933    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db6593dfe38b20151e98310049631736066a171dd355a77c4121288f0c0e8cd1"} err="failed to get container status \"db6593dfe38b20151e98310049631736066a171dd355a77c4121288f0c0e8cd1\": rpc error: code = NotFound desc = could not find container \"db6593dfe38b20151e98310049631736066a171dd355a77c4121288f0c0e8cd1\": container with ID starting with db6593dfe38b20151e98310049631736066a171dd355a77c4121288f0c0e8cd1 not found: ID does not exist"
Sep 09 20:43:25.015108 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:25.014955    2523 scope.go:117] "RemoveContainer" containerID="e5e1cd281ac7d7a7323101f15d9ba0741245b404af500f80973142016048b6fd"
Sep 09 20:43:25.015371 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:43:25.015306    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5e1cd281ac7d7a7323101f15d9ba0741245b404af500f80973142016048b6fd\": container with ID starting with e5e1cd281ac7d7a7323101f15d9ba0741245b404af500f80973142016048b6fd not found: ID does not exist" containerID="e5e1cd281ac7d7a7323101f15d9ba0741245b404af500f80973142016048b6fd"
Sep 09 20:43:25.015371 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:25.015338    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5e1cd281ac7d7a7323101f15d9ba0741245b404af500f80973142016048b6fd"} err="failed to get container status \"e5e1cd281ac7d7a7323101f15d9ba0741245b404af500f80973142016048b6fd\": rpc error: code = NotFound desc = could not find container \"e5e1cd281ac7d7a7323101f15d9ba0741245b404af500f80973142016048b6fd\": container with ID starting with e5e1cd281ac7d7a7323101f15d9ba0741245b404af500f80973142016048b6fd not found: ID does not exist"
Sep 09 20:43:25.015371 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:25.015358    2523 scope.go:117] "RemoveContainer" containerID="778153d644e971045ffb50cdfa2ef1ee2bfcad9dc1b3d4276f19f4565fc5e0d9"
Sep 09 20:43:25.015678 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:43:25.015658    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"778153d644e971045ffb50cdfa2ef1ee2bfcad9dc1b3d4276f19f4565fc5e0d9\": container with ID starting with 778153d644e971045ffb50cdfa2ef1ee2bfcad9dc1b3d4276f19f4565fc5e0d9 not found: ID does not exist" containerID="778153d644e971045ffb50cdfa2ef1ee2bfcad9dc1b3d4276f19f4565fc5e0d9"
Sep 09 20:43:25.015734 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:25.015687    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"778153d644e971045ffb50cdfa2ef1ee2bfcad9dc1b3d4276f19f4565fc5e0d9"} err="failed to get container status \"778153d644e971045ffb50cdfa2ef1ee2bfcad9dc1b3d4276f19f4565fc5e0d9\": rpc error: code = NotFound desc = could not find container \"778153d644e971045ffb50cdfa2ef1ee2bfcad9dc1b3d4276f19f4565fc5e0d9\": container with ID starting with 778153d644e971045ffb50cdfa2ef1ee2bfcad9dc1b3d4276f19f4565fc5e0d9 not found: ID does not exist"
Sep 09 20:43:25.016855 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:25.016838    2523 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7f4f7455bb-2pddd"]
Sep 09 20:43:25.812460 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:25.812413    2523 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03bf34c2-e750-4f7b-b200-fc4862eb4976" path="/var/lib/kubelet/pods/03bf34c2-e750-4f7b-b200-fc4862eb4976/volumes"
Sep 09 20:43:26.910063 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:26.910028    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4ls7b" podUID="228a0c04-4551-4a79-b3ee-7b8d4cfefa81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 09 20:43:36.910775 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:36.910738    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4ls7b" podUID="228a0c04-4551-4a79-b3ee-7b8d4cfefa81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 09 20:43:46.911094 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:46.911066    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4ls7b"
Sep 09 20:43:52.180802 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:52.180770    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4ls7b"]
Sep 09 20:43:52.181253 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:52.181016    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4ls7b" podUID="228a0c04-4551-4a79-b3ee-7b8d4cfefa81" containerName="kube-rbac-proxy" containerID="cri-o://9f3dd715df2af0b388b8a08d5f725ae2c939985114eaf14797d889564f8a5528" gracePeriod=30
Sep 09 20:43:52.181253 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:52.181008    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4ls7b" podUID="228a0c04-4551-4a79-b3ee-7b8d4cfefa81" containerName="kserve-container" containerID="cri-o://eec862445f68efa9c1847de9a42624e657e659b5d47c9280e6c30925e8f85911" gracePeriod=30
Sep 09 20:43:53.100305 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:53.100272    2523 generic.go:356] "Generic (PLEG): container finished" podID="228a0c04-4551-4a79-b3ee-7b8d4cfefa81" containerID="9f3dd715df2af0b388b8a08d5f725ae2c939985114eaf14797d889564f8a5528" exitCode=2
Sep 09 20:43:53.100469 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:53.100344    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4ls7b" event={"ID":"228a0c04-4551-4a79-b3ee-7b8d4cfefa81","Type":"ContainerDied","Data":"9f3dd715df2af0b388b8a08d5f725ae2c939985114eaf14797d889564f8a5528"}
Sep 09 20:43:54.619937 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:54.619917    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4ls7b"
Sep 09 20:43:54.745243 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:54.745184    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/228a0c04-4551-4a79-b3ee-7b8d4cfefa81-cabundle-cert\") pod \"228a0c04-4551-4a79-b3ee-7b8d4cfefa81\" (UID: \"228a0c04-4551-4a79-b3ee-7b8d4cfefa81\") "
Sep 09 20:43:54.745243 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:54.745219    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/228a0c04-4551-4a79-b3ee-7b8d4cfefa81-kube-rbac-proxy-sar-config\") pod \"228a0c04-4551-4a79-b3ee-7b8d4cfefa81\" (UID: \"228a0c04-4551-4a79-b3ee-7b8d4cfefa81\") "
Sep 09 20:43:54.745392 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:54.745246    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/228a0c04-4551-4a79-b3ee-7b8d4cfefa81-kserve-provision-location\") pod \"228a0c04-4551-4a79-b3ee-7b8d4cfefa81\" (UID: \"228a0c04-4551-4a79-b3ee-7b8d4cfefa81\") "
Sep 09 20:43:54.745392 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:54.745291    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/228a0c04-4551-4a79-b3ee-7b8d4cfefa81-proxy-tls\") pod \"228a0c04-4551-4a79-b3ee-7b8d4cfefa81\" (UID: \"228a0c04-4551-4a79-b3ee-7b8d4cfefa81\") "
Sep 09 20:43:54.745392 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:54.745310    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8pxd\" (UniqueName: \"kubernetes.io/projected/228a0c04-4551-4a79-b3ee-7b8d4cfefa81-kube-api-access-f8pxd\") pod \"228a0c04-4551-4a79-b3ee-7b8d4cfefa81\" (UID: \"228a0c04-4551-4a79-b3ee-7b8d4cfefa81\") "
Sep 09 20:43:54.745580 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:54.745554    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/228a0c04-4551-4a79-b3ee-7b8d4cfefa81-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "228a0c04-4551-4a79-b3ee-7b8d4cfefa81" (UID: "228a0c04-4551-4a79-b3ee-7b8d4cfefa81"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:43:54.745632 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:54.745574    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/228a0c04-4551-4a79-b3ee-7b8d4cfefa81-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "228a0c04-4551-4a79-b3ee-7b8d4cfefa81" (UID: "228a0c04-4551-4a79-b3ee-7b8d4cfefa81"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:43:54.747438 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:54.747407    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/228a0c04-4551-4a79-b3ee-7b8d4cfefa81-kube-api-access-f8pxd" (OuterVolumeSpecName: "kube-api-access-f8pxd") pod "228a0c04-4551-4a79-b3ee-7b8d4cfefa81" (UID: "228a0c04-4551-4a79-b3ee-7b8d4cfefa81"). InnerVolumeSpecName "kube-api-access-f8pxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:43:54.747535 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:54.747497    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/228a0c04-4551-4a79-b3ee-7b8d4cfefa81-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "228a0c04-4551-4a79-b3ee-7b8d4cfefa81" (UID: "228a0c04-4551-4a79-b3ee-7b8d4cfefa81"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:43:54.753648 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:54.753627    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/228a0c04-4551-4a79-b3ee-7b8d4cfefa81-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "228a0c04-4551-4a79-b3ee-7b8d4cfefa81" (UID: "228a0c04-4551-4a79-b3ee-7b8d4cfefa81"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:43:54.845841 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:54.845819    2523 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/228a0c04-4551-4a79-b3ee-7b8d4cfefa81-cabundle-cert\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:43:54.845841 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:54.845839    2523 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/228a0c04-4551-4a79-b3ee-7b8d4cfefa81-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:43:54.845936 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:54.845849    2523 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/228a0c04-4551-4a79-b3ee-7b8d4cfefa81-kserve-provision-location\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:43:54.845936 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:54.845858    2523 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/228a0c04-4551-4a79-b3ee-7b8d4cfefa81-proxy-tls\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:43:54.845936 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:54.845866    2523 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8pxd\" (UniqueName: \"kubernetes.io/projected/228a0c04-4551-4a79-b3ee-7b8d4cfefa81-kube-api-access-f8pxd\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:43:55.110308 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:55.110278    2523 generic.go:356] "Generic (PLEG): container finished" podID="228a0c04-4551-4a79-b3ee-7b8d4cfefa81" containerID="eec862445f68efa9c1847de9a42624e657e659b5d47c9280e6c30925e8f85911" exitCode=0
Sep 09 20:43:55.110410 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:55.110334    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4ls7b"
Sep 09 20:43:55.110410 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:55.110340    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4ls7b" event={"ID":"228a0c04-4551-4a79-b3ee-7b8d4cfefa81","Type":"ContainerDied","Data":"eec862445f68efa9c1847de9a42624e657e659b5d47c9280e6c30925e8f85911"}
Sep 09 20:43:55.110410 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:55.110388    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4ls7b" event={"ID":"228a0c04-4551-4a79-b3ee-7b8d4cfefa81","Type":"ContainerDied","Data":"660ae7e751274ce2d7dbf7f3f112a925c1ffcec114ab959364026ca285cdc5a6"}
Sep 09 20:43:55.110527 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:55.110413    2523 scope.go:117] "RemoveContainer" containerID="9f3dd715df2af0b388b8a08d5f725ae2c939985114eaf14797d889564f8a5528"
Sep 09 20:43:55.118711 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:55.118694    2523 scope.go:117] "RemoveContainer" containerID="eec862445f68efa9c1847de9a42624e657e659b5d47c9280e6c30925e8f85911"
Sep 09 20:43:55.126047 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:55.126030    2523 scope.go:117] "RemoveContainer" containerID="e9b3f7f94564d501eb908129b0026277d9d7b0ef2b54d6de864f3725a61542e2"
Sep 09 20:43:55.132040 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:55.132017    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4ls7b"]
Sep 09 20:43:55.133575 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:55.133558    2523 scope.go:117] "RemoveContainer" containerID="9f3dd715df2af0b388b8a08d5f725ae2c939985114eaf14797d889564f8a5528"
Sep 09 20:43:55.133826 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:43:55.133808    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f3dd715df2af0b388b8a08d5f725ae2c939985114eaf14797d889564f8a5528\": container with ID starting with 9f3dd715df2af0b388b8a08d5f725ae2c939985114eaf14797d889564f8a5528 not found: ID does not exist" containerID="9f3dd715df2af0b388b8a08d5f725ae2c939985114eaf14797d889564f8a5528"
Sep 09 20:43:55.133884 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:55.133832    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f3dd715df2af0b388b8a08d5f725ae2c939985114eaf14797d889564f8a5528"} err="failed to get container status \"9f3dd715df2af0b388b8a08d5f725ae2c939985114eaf14797d889564f8a5528\": rpc error: code = NotFound desc = could not find container \"9f3dd715df2af0b388b8a08d5f725ae2c939985114eaf14797d889564f8a5528\": container with ID starting with 9f3dd715df2af0b388b8a08d5f725ae2c939985114eaf14797d889564f8a5528 not found: ID does not exist"
Sep 09 20:43:55.133884 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:55.133847    2523 scope.go:117] "RemoveContainer" containerID="eec862445f68efa9c1847de9a42624e657e659b5d47c9280e6c30925e8f85911"
Sep 09 20:43:55.134065 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:43:55.134047    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eec862445f68efa9c1847de9a42624e657e659b5d47c9280e6c30925e8f85911\": container with ID starting with eec862445f68efa9c1847de9a42624e657e659b5d47c9280e6c30925e8f85911 not found: ID does not exist" containerID="eec862445f68efa9c1847de9a42624e657e659b5d47c9280e6c30925e8f85911"
Sep 09 20:43:55.134105 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:55.134070    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eec862445f68efa9c1847de9a42624e657e659b5d47c9280e6c30925e8f85911"} err="failed to get container status \"eec862445f68efa9c1847de9a42624e657e659b5d47c9280e6c30925e8f85911\": rpc error: code = NotFound desc = could not find container \"eec862445f68efa9c1847de9a42624e657e659b5d47c9280e6c30925e8f85911\": container with ID starting with eec862445f68efa9c1847de9a42624e657e659b5d47c9280e6c30925e8f85911 not found: ID does not exist"
Sep 09 20:43:55.134105 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:55.134098    2523 scope.go:117] "RemoveContainer" containerID="e9b3f7f94564d501eb908129b0026277d9d7b0ef2b54d6de864f3725a61542e2"
Sep 09 20:43:55.134426 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:43:55.134409    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9b3f7f94564d501eb908129b0026277d9d7b0ef2b54d6de864f3725a61542e2\": container with ID starting with e9b3f7f94564d501eb908129b0026277d9d7b0ef2b54d6de864f3725a61542e2 not found: ID does not exist" containerID="e9b3f7f94564d501eb908129b0026277d9d7b0ef2b54d6de864f3725a61542e2"
Sep 09 20:43:55.134539 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:55.134428    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9b3f7f94564d501eb908129b0026277d9d7b0ef2b54d6de864f3725a61542e2"} err="failed to get container status \"e9b3f7f94564d501eb908129b0026277d9d7b0ef2b54d6de864f3725a61542e2\": rpc error: code = NotFound desc = could not find container \"e9b3f7f94564d501eb908129b0026277d9d7b0ef2b54d6de864f3725a61542e2\": container with ID starting with e9b3f7f94564d501eb908129b0026277d9d7b0ef2b54d6de864f3725a61542e2 not found: ID does not exist"
Sep 09 20:43:55.136416 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:55.136393    2523 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-4ls7b"]
Sep 09 20:43:55.807543 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:55.807516    2523 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="228a0c04-4551-4a79-b3ee-7b8d4cfefa81" path="/var/lib/kubelet/pods/228a0c04-4551-4a79-b3ee-7b8d4cfefa81/volumes"
Sep 09 20:43:57.357546 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:57.357515    2523 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tb9ml"]
Sep 09 20:43:57.357930 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:57.357796    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03bf34c2-e750-4f7b-b200-fc4862eb4976" containerName="storage-initializer"
Sep 09 20:43:57.357930 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:57.357809    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="03bf34c2-e750-4f7b-b200-fc4862eb4976" containerName="storage-initializer"
Sep 09 20:43:57.357930 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:57.357818    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03bf34c2-e750-4f7b-b200-fc4862eb4976" containerName="kube-rbac-proxy"
Sep 09 20:43:57.357930 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:57.357824    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="03bf34c2-e750-4f7b-b200-fc4862eb4976" containerName="kube-rbac-proxy"
Sep 09 20:43:57.357930 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:57.357832    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="228a0c04-4551-4a79-b3ee-7b8d4cfefa81" containerName="kube-rbac-proxy"
Sep 09 20:43:57.357930 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:57.357837    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="228a0c04-4551-4a79-b3ee-7b8d4cfefa81" containerName="kube-rbac-proxy"
Sep 09 20:43:57.357930 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:57.357847    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="228a0c04-4551-4a79-b3ee-7b8d4cfefa81" containerName="kserve-container"
Sep 09 20:43:57.357930 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:57.357852    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="228a0c04-4551-4a79-b3ee-7b8d4cfefa81" containerName="kserve-container"
Sep 09 20:43:57.357930 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:57.357859    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03bf34c2-e750-4f7b-b200-fc4862eb4976" containerName="kserve-container"
Sep 09 20:43:57.357930 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:57.357864    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="03bf34c2-e750-4f7b-b200-fc4862eb4976" containerName="kserve-container"
Sep 09 20:43:57.357930 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:57.357870    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="228a0c04-4551-4a79-b3ee-7b8d4cfefa81" containerName="storage-initializer"
Sep 09 20:43:57.357930 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:57.357875    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="228a0c04-4551-4a79-b3ee-7b8d4cfefa81" containerName="storage-initializer"
Sep 09 20:43:57.357930 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:57.357922    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="03bf34c2-e750-4f7b-b200-fc4862eb4976" containerName="kserve-container"
Sep 09 20:43:57.357930 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:57.357928    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="03bf34c2-e750-4f7b-b200-fc4862eb4976" containerName="kube-rbac-proxy"
Sep 09 20:43:57.357930 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:57.357935    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="228a0c04-4551-4a79-b3ee-7b8d4cfefa81" containerName="kserve-container"
Sep 09 20:43:57.358482 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:57.357941    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="228a0c04-4551-4a79-b3ee-7b8d4cfefa81" containerName="kube-rbac-proxy"
Sep 09 20:43:57.361093 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:57.361077    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tb9ml"
Sep 09 20:43:57.363442 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:57.363415    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 20:43:57.363594 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:57.363449    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 20:43:57.363594 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:57.363477    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 20:43:57.363594 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:57.363500    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-ks7mb\""
Sep 09 20:43:57.364364 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:57.364346    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\""
Sep 09 20:43:57.364478 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:57.364420    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
Sep 09 20:43:57.364478 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:57.364428    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:43:57.370372 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:57.370351    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tb9ml"]
Sep 09 20:43:57.464540 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:57.464512    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6db5d4de-d763-4861-bebb-86b844764a3a-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-tb9ml\" (UID: \"6db5d4de-d763-4861-bebb-86b844764a3a\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tb9ml"
Sep 09 20:43:57.464680 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:57.464544    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6db5d4de-d763-4861-bebb-86b844764a3a-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-tb9ml\" (UID: \"6db5d4de-d763-4861-bebb-86b844764a3a\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tb9ml"
Sep 09 20:43:57.464680 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:57.464564    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2f5h\" (UniqueName: \"kubernetes.io/projected/6db5d4de-d763-4861-bebb-86b844764a3a-kube-api-access-f2f5h\") pod \"isvc-paddle-runtime-predictor-58b99fff76-tb9ml\" (UID: \"6db5d4de-d763-4861-bebb-86b844764a3a\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tb9ml"
Sep 09 20:43:57.464680 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:57.464623    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6db5d4de-d763-4861-bebb-86b844764a3a-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-tb9ml\" (UID: \"6db5d4de-d763-4861-bebb-86b844764a3a\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tb9ml"
Sep 09 20:43:57.464680 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:57.464652    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6db5d4de-d763-4861-bebb-86b844764a3a-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-tb9ml\" (UID: \"6db5d4de-d763-4861-bebb-86b844764a3a\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tb9ml"
Sep 09 20:43:57.565800 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:57.565765    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6db5d4de-d763-4861-bebb-86b844764a3a-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-tb9ml\" (UID: \"6db5d4de-d763-4861-bebb-86b844764a3a\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tb9ml"
Sep 09 20:43:57.565800 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:57.565804    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6db5d4de-d763-4861-bebb-86b844764a3a-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-tb9ml\" (UID: \"6db5d4de-d763-4861-bebb-86b844764a3a\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tb9ml"
Sep 09 20:43:57.565991 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:57.565844    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6db5d4de-d763-4861-bebb-86b844764a3a-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-tb9ml\" (UID: \"6db5d4de-d763-4861-bebb-86b844764a3a\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tb9ml"
Sep 09 20:43:57.565991 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:57.565863    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6db5d4de-d763-4861-bebb-86b844764a3a-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-tb9ml\" (UID: \"6db5d4de-d763-4861-bebb-86b844764a3a\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tb9ml"
Sep 09 20:43:57.565991 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:57.565883    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2f5h\" (UniqueName: \"kubernetes.io/projected/6db5d4de-d763-4861-bebb-86b844764a3a-kube-api-access-f2f5h\") pod \"isvc-paddle-runtime-predictor-58b99fff76-tb9ml\" (UID: \"6db5d4de-d763-4861-bebb-86b844764a3a\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tb9ml"
Sep 09 20:43:57.566334 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:57.566310    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6db5d4de-d763-4861-bebb-86b844764a3a-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-tb9ml\" (UID: \"6db5d4de-d763-4861-bebb-86b844764a3a\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tb9ml"
Sep 09 20:43:57.566594 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:57.566571    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6db5d4de-d763-4861-bebb-86b844764a3a-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-tb9ml\" (UID: \"6db5d4de-d763-4861-bebb-86b844764a3a\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tb9ml"
Sep 09 20:43:57.566917 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:57.566610    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6db5d4de-d763-4861-bebb-86b844764a3a-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-tb9ml\" (UID: \"6db5d4de-d763-4861-bebb-86b844764a3a\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tb9ml"
Sep 09 20:43:57.568455 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:57.568438    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6db5d4de-d763-4861-bebb-86b844764a3a-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-tb9ml\" (UID: \"6db5d4de-d763-4861-bebb-86b844764a3a\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tb9ml"
Sep 09 20:43:57.574803 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:57.574782    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2f5h\" (UniqueName: \"kubernetes.io/projected/6db5d4de-d763-4861-bebb-86b844764a3a-kube-api-access-f2f5h\") pod \"isvc-paddle-runtime-predictor-58b99fff76-tb9ml\" (UID: \"6db5d4de-d763-4861-bebb-86b844764a3a\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tb9ml"
Sep 09 20:43:57.671600 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:57.671543    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tb9ml"
Sep 09 20:43:57.942519 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:57.942451    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tb9ml"]
Sep 09 20:43:57.945317 ip-10-0-132-226 kubenswrapper[2523]: W0909 20:43:57.945295    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6db5d4de_d763_4861_bebb_86b844764a3a.slice/crio-12a0332cd2ce95cb838ab8705c2a983e35cfead2a13e75be00460c73cd745ca5 WatchSource:0}: Error finding container 12a0332cd2ce95cb838ab8705c2a983e35cfead2a13e75be00460c73cd745ca5: Status 404 returned error can't find the container with id 12a0332cd2ce95cb838ab8705c2a983e35cfead2a13e75be00460c73cd745ca5
Sep 09 20:43:58.124256 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:58.124225    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tb9ml" event={"ID":"6db5d4de-d763-4861-bebb-86b844764a3a","Type":"ContainerStarted","Data":"ce40f279e244c80ad7801ee908a35764b1052f9b18ad1d91397a8acc828c5195"}
Sep 09 20:43:58.124416 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:43:58.124286    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tb9ml" event={"ID":"6db5d4de-d763-4861-bebb-86b844764a3a","Type":"ContainerStarted","Data":"12a0332cd2ce95cb838ab8705c2a983e35cfead2a13e75be00460c73cd745ca5"}
Sep 09 20:44:03.143543 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:44:03.143512    2523 generic.go:356] "Generic (PLEG): container finished" podID="6db5d4de-d763-4861-bebb-86b844764a3a" containerID="ce40f279e244c80ad7801ee908a35764b1052f9b18ad1d91397a8acc828c5195" exitCode=0
Sep 09 20:44:03.143942 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:44:03.143585    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tb9ml" event={"ID":"6db5d4de-d763-4861-bebb-86b844764a3a","Type":"ContainerDied","Data":"ce40f279e244c80ad7801ee908a35764b1052f9b18ad1d91397a8acc828c5195"}
Sep 09 20:44:04.149948 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:44:04.149912    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tb9ml" event={"ID":"6db5d4de-d763-4861-bebb-86b844764a3a","Type":"ContainerStarted","Data":"54106d982f63bf13a8c2c8a671ac11ff5cb7f48fc4d93864cd653bee40e96845"}
Sep 09 20:44:04.150353 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:44:04.149974    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tb9ml" event={"ID":"6db5d4de-d763-4861-bebb-86b844764a3a","Type":"ContainerStarted","Data":"fc00212fa6ffb6c6a72f559724f1e01d361f3b4fe9abe2ecf909cbba049470c4"}
Sep 09 20:44:04.150353 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:44:04.150186    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tb9ml"
Sep 09 20:44:04.171644 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:44:04.171598    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tb9ml" podStartSLOduration=3.010890348 podStartE2EDuration="7.171583691s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.160693343s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:43:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:44:04.169354765 +0000 UTC m=+2948.841373764" watchObservedRunningTime="2026-09-09 20:44:04.171583691 +0000 UTC m=+2948.843602687"
Sep 09 20:44:05.153836 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:44:05.153802    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tb9ml"
Sep 09 20:44:05.155320 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:44:05.155294    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tb9ml" podUID="6db5d4de-d763-4861-bebb-86b844764a3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 09 20:44:06.157382 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:44:06.157342    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tb9ml" podUID="6db5d4de-d763-4861-bebb-86b844764a3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 09 20:44:11.161573 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:44:11.161547    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tb9ml"
Sep 09 20:44:11.162059 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:44:11.162028    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tb9ml" podUID="6db5d4de-d763-4861-bebb-86b844764a3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 09 20:44:21.161999 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:44:21.161960    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tb9ml" podUID="6db5d4de-d763-4861-bebb-86b844764a3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 09 20:44:31.162238 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:44:31.162196    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tb9ml" podUID="6db5d4de-d763-4861-bebb-86b844764a3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 09 20:44:41.162118 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:44:41.162072    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tb9ml" podUID="6db5d4de-d763-4861-bebb-86b844764a3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 09 20:44:51.162833 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:44:51.162806    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tb9ml"
Sep 09 20:44:58.484295 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:44:58.484257    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tb9ml"]
Sep 09 20:44:58.485037 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:44:58.484545    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tb9ml" podUID="6db5d4de-d763-4861-bebb-86b844764a3a" containerName="kserve-container" containerID="cri-o://fc00212fa6ffb6c6a72f559724f1e01d361f3b4fe9abe2ecf909cbba049470c4" gracePeriod=30
Sep 09 20:44:58.485037 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:44:58.484575    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tb9ml" podUID="6db5d4de-d763-4861-bebb-86b844764a3a" containerName="kube-rbac-proxy" containerID="cri-o://54106d982f63bf13a8c2c8a671ac11ff5cb7f48fc4d93864cd653bee40e96845" gracePeriod=30
Sep 09 20:44:59.354442 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:44:59.354408    2523 generic.go:356] "Generic (PLEG): container finished" podID="6db5d4de-d763-4861-bebb-86b844764a3a" containerID="54106d982f63bf13a8c2c8a671ac11ff5cb7f48fc4d93864cd653bee40e96845" exitCode=2
Sep 09 20:44:59.354645 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:44:59.354466    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tb9ml" event={"ID":"6db5d4de-d763-4861-bebb-86b844764a3a","Type":"ContainerDied","Data":"54106d982f63bf13a8c2c8a671ac11ff5cb7f48fc4d93864cd653bee40e96845"}
Sep 09 20:45:00.726493 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:00.726472    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tb9ml"
Sep 09 20:45:00.888370 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:00.888341    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6db5d4de-d763-4861-bebb-86b844764a3a-proxy-tls\") pod \"6db5d4de-d763-4861-bebb-86b844764a3a\" (UID: \"6db5d4de-d763-4861-bebb-86b844764a3a\") "
Sep 09 20:45:00.888520 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:00.888384    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6db5d4de-d763-4861-bebb-86b844764a3a-kube-rbac-proxy-sar-config\") pod \"6db5d4de-d763-4861-bebb-86b844764a3a\" (UID: \"6db5d4de-d763-4861-bebb-86b844764a3a\") "
Sep 09 20:45:00.888520 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:00.888420    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6db5d4de-d763-4861-bebb-86b844764a3a-kserve-provision-location\") pod \"6db5d4de-d763-4861-bebb-86b844764a3a\" (UID: \"6db5d4de-d763-4861-bebb-86b844764a3a\") "
Sep 09 20:45:00.888520 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:00.888449    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6db5d4de-d763-4861-bebb-86b844764a3a-cabundle-cert\") pod \"6db5d4de-d763-4861-bebb-86b844764a3a\" (UID: \"6db5d4de-d763-4861-bebb-86b844764a3a\") "
Sep 09 20:45:00.888690 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:00.888538    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2f5h\" (UniqueName: \"kubernetes.io/projected/6db5d4de-d763-4861-bebb-86b844764a3a-kube-api-access-f2f5h\") pod \"6db5d4de-d763-4861-bebb-86b844764a3a\" (UID: \"6db5d4de-d763-4861-bebb-86b844764a3a\") "
Sep 09 20:45:00.888769 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:00.888745    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6db5d4de-d763-4861-bebb-86b844764a3a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6db5d4de-d763-4861-bebb-86b844764a3a" (UID: "6db5d4de-d763-4861-bebb-86b844764a3a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:45:00.888877 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:00.888845    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6db5d4de-d763-4861-bebb-86b844764a3a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6db5d4de-d763-4861-bebb-86b844764a3a" (UID: "6db5d4de-d763-4861-bebb-86b844764a3a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:45:00.890564 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:00.890541    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6db5d4de-d763-4861-bebb-86b844764a3a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6db5d4de-d763-4861-bebb-86b844764a3a" (UID: "6db5d4de-d763-4861-bebb-86b844764a3a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:45:00.890683 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:00.890664    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6db5d4de-d763-4861-bebb-86b844764a3a-kube-api-access-f2f5h" (OuterVolumeSpecName: "kube-api-access-f2f5h") pod "6db5d4de-d763-4861-bebb-86b844764a3a" (UID: "6db5d4de-d763-4861-bebb-86b844764a3a"). InnerVolumeSpecName "kube-api-access-f2f5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:45:00.901587 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:00.901555    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db5d4de-d763-4861-bebb-86b844764a3a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6db5d4de-d763-4861-bebb-86b844764a3a" (UID: "6db5d4de-d763-4861-bebb-86b844764a3a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:45:00.989698 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:00.989673    2523 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2f5h\" (UniqueName: \"kubernetes.io/projected/6db5d4de-d763-4861-bebb-86b844764a3a-kube-api-access-f2f5h\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:45:00.989698 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:00.989697    2523 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6db5d4de-d763-4861-bebb-86b844764a3a-proxy-tls\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:45:00.989826 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:00.989711    2523 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6db5d4de-d763-4861-bebb-86b844764a3a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:45:00.989826 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:00.989723    2523 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6db5d4de-d763-4861-bebb-86b844764a3a-kserve-provision-location\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:45:00.989826 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:00.989734    2523 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6db5d4de-d763-4861-bebb-86b844764a3a-cabundle-cert\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:45:01.363024 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:01.362997    2523 generic.go:356] "Generic (PLEG): container finished" podID="6db5d4de-d763-4861-bebb-86b844764a3a" containerID="fc00212fa6ffb6c6a72f559724f1e01d361f3b4fe9abe2ecf909cbba049470c4" exitCode=0
Sep 09 20:45:01.363135 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:01.363029    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tb9ml" event={"ID":"6db5d4de-d763-4861-bebb-86b844764a3a","Type":"ContainerDied","Data":"fc00212fa6ffb6c6a72f559724f1e01d361f3b4fe9abe2ecf909cbba049470c4"}
Sep 09 20:45:01.363135 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:01.363067    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tb9ml"
Sep 09 20:45:01.363135 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:01.363105    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tb9ml" event={"ID":"6db5d4de-d763-4861-bebb-86b844764a3a","Type":"ContainerDied","Data":"12a0332cd2ce95cb838ab8705c2a983e35cfead2a13e75be00460c73cd745ca5"}
Sep 09 20:45:01.363262 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:01.363141    2523 scope.go:117] "RemoveContainer" containerID="54106d982f63bf13a8c2c8a671ac11ff5cb7f48fc4d93864cd653bee40e96845"
Sep 09 20:45:01.371344 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:01.371326    2523 scope.go:117] "RemoveContainer" containerID="fc00212fa6ffb6c6a72f559724f1e01d361f3b4fe9abe2ecf909cbba049470c4"
Sep 09 20:45:01.378476 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:01.378458    2523 scope.go:117] "RemoveContainer" containerID="ce40f279e244c80ad7801ee908a35764b1052f9b18ad1d91397a8acc828c5195"
Sep 09 20:45:01.385431 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:01.385410    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tb9ml"]
Sep 09 20:45:01.385596 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:01.385548    2523 scope.go:117] "RemoveContainer" containerID="54106d982f63bf13a8c2c8a671ac11ff5cb7f48fc4d93864cd653bee40e96845"
Sep 09 20:45:01.385804 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:45:01.385787    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54106d982f63bf13a8c2c8a671ac11ff5cb7f48fc4d93864cd653bee40e96845\": container with ID starting with 54106d982f63bf13a8c2c8a671ac11ff5cb7f48fc4d93864cd653bee40e96845 not found: ID does not exist" containerID="54106d982f63bf13a8c2c8a671ac11ff5cb7f48fc4d93864cd653bee40e96845"
Sep 09 20:45:01.385848 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:01.385812    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54106d982f63bf13a8c2c8a671ac11ff5cb7f48fc4d93864cd653bee40e96845"} err="failed to get container status \"54106d982f63bf13a8c2c8a671ac11ff5cb7f48fc4d93864cd653bee40e96845\": rpc error: code = NotFound desc = could not find container \"54106d982f63bf13a8c2c8a671ac11ff5cb7f48fc4d93864cd653bee40e96845\": container with ID starting with 54106d982f63bf13a8c2c8a671ac11ff5cb7f48fc4d93864cd653bee40e96845 not found: ID does not exist"
Sep 09 20:45:01.385848 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:01.385827    2523 scope.go:117] "RemoveContainer" containerID="fc00212fa6ffb6c6a72f559724f1e01d361f3b4fe9abe2ecf909cbba049470c4"
Sep 09 20:45:01.386044 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:45:01.386029    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc00212fa6ffb6c6a72f559724f1e01d361f3b4fe9abe2ecf909cbba049470c4\": container with ID starting with fc00212fa6ffb6c6a72f559724f1e01d361f3b4fe9abe2ecf909cbba049470c4 not found: ID does not exist" containerID="fc00212fa6ffb6c6a72f559724f1e01d361f3b4fe9abe2ecf909cbba049470c4"
Sep 09 20:45:01.386089 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:01.386048    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc00212fa6ffb6c6a72f559724f1e01d361f3b4fe9abe2ecf909cbba049470c4"} err="failed to get container status \"fc00212fa6ffb6c6a72f559724f1e01d361f3b4fe9abe2ecf909cbba049470c4\": rpc error: code = NotFound desc = could not find container \"fc00212fa6ffb6c6a72f559724f1e01d361f3b4fe9abe2ecf909cbba049470c4\": container with ID starting with fc00212fa6ffb6c6a72f559724f1e01d361f3b4fe9abe2ecf909cbba049470c4 not found: ID does not exist"
Sep 09 20:45:01.386089 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:01.386059    2523 scope.go:117] "RemoveContainer" containerID="ce40f279e244c80ad7801ee908a35764b1052f9b18ad1d91397a8acc828c5195"
Sep 09 20:45:01.386292 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:45:01.386277    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce40f279e244c80ad7801ee908a35764b1052f9b18ad1d91397a8acc828c5195\": container with ID starting with ce40f279e244c80ad7801ee908a35764b1052f9b18ad1d91397a8acc828c5195 not found: ID does not exist" containerID="ce40f279e244c80ad7801ee908a35764b1052f9b18ad1d91397a8acc828c5195"
Sep 09 20:45:01.386336 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:01.386295    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce40f279e244c80ad7801ee908a35764b1052f9b18ad1d91397a8acc828c5195"} err="failed to get container status \"ce40f279e244c80ad7801ee908a35764b1052f9b18ad1d91397a8acc828c5195\": rpc error: code = NotFound desc = could not find container \"ce40f279e244c80ad7801ee908a35764b1052f9b18ad1d91397a8acc828c5195\": container with ID starting with ce40f279e244c80ad7801ee908a35764b1052f9b18ad1d91397a8acc828c5195 not found: ID does not exist"
Sep 09 20:45:01.390573 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:01.390555    2523 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-tb9ml"]
Sep 09 20:45:01.811505 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:01.811428    2523 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6db5d4de-d763-4861-bebb-86b844764a3a" path="/var/lib/kubelet/pods/6db5d4de-d763-4861-bebb-86b844764a3a/volumes"
Sep 09 20:45:03.672828 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:03.672793    2523 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs"]
Sep 09 20:45:03.673316 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:03.673042    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6db5d4de-d763-4861-bebb-86b844764a3a" containerName="kube-rbac-proxy"
Sep 09 20:45:03.673316 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:03.673052    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="6db5d4de-d763-4861-bebb-86b844764a3a" containerName="kube-rbac-proxy"
Sep 09 20:45:03.673316 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:03.673069    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6db5d4de-d763-4861-bebb-86b844764a3a" containerName="kserve-container"
Sep 09 20:45:03.673316 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:03.673075    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="6db5d4de-d763-4861-bebb-86b844764a3a" containerName="kserve-container"
Sep 09 20:45:03.673316 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:03.673082    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6db5d4de-d763-4861-bebb-86b844764a3a" containerName="storage-initializer"
Sep 09 20:45:03.673316 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:03.673087    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="6db5d4de-d763-4861-bebb-86b844764a3a" containerName="storage-initializer"
Sep 09 20:45:03.673316 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:03.673133    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="6db5d4de-d763-4861-bebb-86b844764a3a" containerName="kserve-container"
Sep 09 20:45:03.673316 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:03.673164    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="6db5d4de-d763-4861-bebb-86b844764a3a" containerName="kube-rbac-proxy"
Sep 09 20:45:03.677686 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:03.677668    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs"
Sep 09 20:45:03.680532 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:03.680505    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-ks7mb\""
Sep 09 20:45:03.680532 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:03.680525    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 09 20:45:03.680750 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:03.680543    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:45:03.680750 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:03.680561    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 20:45:03.680867 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:03.680795    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 20:45:03.681424 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:03.681407    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
Sep 09 20:45:03.681498 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:03.681449    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 20:45:03.689534 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:03.689511    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs"]
Sep 09 20:45:03.810041 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:03.810019    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6aa09bc9-09cb-4a28-8af3-ee5033477a59-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs\" (UID: \"6aa09bc9-09cb-4a28-8af3-ee5033477a59\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs"
Sep 09 20:45:03.810202 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:03.810073    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ktzh\" (UniqueName: \"kubernetes.io/projected/6aa09bc9-09cb-4a28-8af3-ee5033477a59-kube-api-access-7ktzh\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs\" (UID: \"6aa09bc9-09cb-4a28-8af3-ee5033477a59\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs"
Sep 09 20:45:03.810202 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:03.810092    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6aa09bc9-09cb-4a28-8af3-ee5033477a59-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs\" (UID: \"6aa09bc9-09cb-4a28-8af3-ee5033477a59\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs"
Sep 09 20:45:03.810202 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:03.810132    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6aa09bc9-09cb-4a28-8af3-ee5033477a59-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs\" (UID: \"6aa09bc9-09cb-4a28-8af3-ee5033477a59\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs"
Sep 09 20:45:03.810314 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:03.810205    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6aa09bc9-09cb-4a28-8af3-ee5033477a59-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs\" (UID: \"6aa09bc9-09cb-4a28-8af3-ee5033477a59\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs"
Sep 09 20:45:03.911485 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:03.911459    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6aa09bc9-09cb-4a28-8af3-ee5033477a59-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs\" (UID: \"6aa09bc9-09cb-4a28-8af3-ee5033477a59\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs"
Sep 09 20:45:03.911594 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:03.911503    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ktzh\" (UniqueName: \"kubernetes.io/projected/6aa09bc9-09cb-4a28-8af3-ee5033477a59-kube-api-access-7ktzh\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs\" (UID: \"6aa09bc9-09cb-4a28-8af3-ee5033477a59\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs"
Sep 09 20:45:03.911594 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:03.911527    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6aa09bc9-09cb-4a28-8af3-ee5033477a59-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs\" (UID: \"6aa09bc9-09cb-4a28-8af3-ee5033477a59\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs"
Sep 09 20:45:03.911594 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:03.911552    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6aa09bc9-09cb-4a28-8af3-ee5033477a59-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs\" (UID: \"6aa09bc9-09cb-4a28-8af3-ee5033477a59\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs"
Sep 09 20:45:03.911717 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:03.911599    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6aa09bc9-09cb-4a28-8af3-ee5033477a59-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs\" (UID: \"6aa09bc9-09cb-4a28-8af3-ee5033477a59\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs"
Sep 09 20:45:03.911930 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:03.911915    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6aa09bc9-09cb-4a28-8af3-ee5033477a59-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs\" (UID: \"6aa09bc9-09cb-4a28-8af3-ee5033477a59\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs"
Sep 09 20:45:03.912268 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:03.912240    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6aa09bc9-09cb-4a28-8af3-ee5033477a59-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs\" (UID: \"6aa09bc9-09cb-4a28-8af3-ee5033477a59\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs"
Sep 09 20:45:03.912365 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:03.912269    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6aa09bc9-09cb-4a28-8af3-ee5033477a59-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs\" (UID: \"6aa09bc9-09cb-4a28-8af3-ee5033477a59\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs"
Sep 09 20:45:03.914219 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:03.914191    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6aa09bc9-09cb-4a28-8af3-ee5033477a59-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs\" (UID: \"6aa09bc9-09cb-4a28-8af3-ee5033477a59\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs"
Sep 09 20:45:03.920026 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:03.920002    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ktzh\" (UniqueName: \"kubernetes.io/projected/6aa09bc9-09cb-4a28-8af3-ee5033477a59-kube-api-access-7ktzh\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs\" (UID: \"6aa09bc9-09cb-4a28-8af3-ee5033477a59\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs"
Sep 09 20:45:03.987467 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:03.987411    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs"
Sep 09 20:45:04.259526 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:04.259503    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs"]
Sep 09 20:45:04.262207 ip-10-0-132-226 kubenswrapper[2523]: W0909 20:45:04.262142    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aa09bc9_09cb_4a28_8af3_ee5033477a59.slice/crio-05dba9b3ff21a9df32ad599099a60e3aa2381475a70586ec35228874a630518d WatchSource:0}: Error finding container 05dba9b3ff21a9df32ad599099a60e3aa2381475a70586ec35228874a630518d: Status 404 returned error can't find the container with id 05dba9b3ff21a9df32ad599099a60e3aa2381475a70586ec35228874a630518d
Sep 09 20:45:04.376459 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:04.376425    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs" event={"ID":"6aa09bc9-09cb-4a28-8af3-ee5033477a59","Type":"ContainerStarted","Data":"7b764cb1dfb7ccc18cd4cb83f97c0b7fee8047dfab1a5f0f67b22de51d5f941f"}
Sep 09 20:45:04.376604 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:04.376489    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs" event={"ID":"6aa09bc9-09cb-4a28-8af3-ee5033477a59","Type":"ContainerStarted","Data":"05dba9b3ff21a9df32ad599099a60e3aa2381475a70586ec35228874a630518d"}
Sep 09 20:45:09.400210 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:09.400183    2523 generic.go:356] "Generic (PLEG): container finished" podID="6aa09bc9-09cb-4a28-8af3-ee5033477a59" containerID="7b764cb1dfb7ccc18cd4cb83f97c0b7fee8047dfab1a5f0f67b22de51d5f941f" exitCode=0
Sep 09 20:45:09.400620 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:09.400291    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs" event={"ID":"6aa09bc9-09cb-4a28-8af3-ee5033477a59","Type":"ContainerDied","Data":"7b764cb1dfb7ccc18cd4cb83f97c0b7fee8047dfab1a5f0f67b22de51d5f941f"}
Sep 09 20:45:10.406602 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:10.406567    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs" event={"ID":"6aa09bc9-09cb-4a28-8af3-ee5033477a59","Type":"ContainerStarted","Data":"f36998239c2850c3626bddc56df173f867962bb1471f09722a22029d3da23d11"}
Sep 09 20:45:10.406949 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:10.406628    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs" event={"ID":"6aa09bc9-09cb-4a28-8af3-ee5033477a59","Type":"ContainerStarted","Data":"e95c607d985092bc140ffc67feec8639657d245b1f260c4e231f32cfd7983f9f"}
Sep 09 20:45:10.406949 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:10.406881    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs"
Sep 09 20:45:10.407051 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:10.407034    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs"
Sep 09 20:45:10.408333 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:10.408311    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs" podUID="6aa09bc9-09cb-4a28-8af3-ee5033477a59" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 09 20:45:10.430508 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:10.430462    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs" podStartSLOduration=2.359048494 podStartE2EDuration="7.430448754s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.07140026s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:45:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:45:10.429213128 +0000 UTC m=+3015.101232125" watchObservedRunningTime="2026-09-09 20:45:10.430448754 +0000 UTC m=+3015.102467752"
Sep 09 20:45:11.409989 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:11.409959    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs" podUID="6aa09bc9-09cb-4a28-8af3-ee5033477a59" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 09 20:45:12.413683 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:12.413647    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs" podUID="6aa09bc9-09cb-4a28-8af3-ee5033477a59" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 09 20:45:17.418680 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:17.418646    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs"
Sep 09 20:45:17.419327 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:17.419296    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs" podUID="6aa09bc9-09cb-4a28-8af3-ee5033477a59" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 09 20:45:27.419670 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:27.419626    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs" podUID="6aa09bc9-09cb-4a28-8af3-ee5033477a59" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 09 20:45:37.419214 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:37.419168    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs" podUID="6aa09bc9-09cb-4a28-8af3-ee5033477a59" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 09 20:45:47.419861 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:47.419821    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs" podUID="6aa09bc9-09cb-4a28-8af3-ee5033477a59" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 09 20:45:57.420356 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:45:57.420324    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs"
Sep 09 20:46:05.112815 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:05.112782    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs"]
Sep 09 20:46:05.113220 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:05.113052    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs" podUID="6aa09bc9-09cb-4a28-8af3-ee5033477a59" containerName="kserve-container" containerID="cri-o://e95c607d985092bc140ffc67feec8639657d245b1f260c4e231f32cfd7983f9f" gracePeriod=30
Sep 09 20:46:05.113220 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:05.113091    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs" podUID="6aa09bc9-09cb-4a28-8af3-ee5033477a59" containerName="kube-rbac-proxy" containerID="cri-o://f36998239c2850c3626bddc56df173f867962bb1471f09722a22029d3da23d11" gracePeriod=30
Sep 09 20:46:05.622906 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:05.622876    2523 generic.go:356] "Generic (PLEG): container finished" podID="6aa09bc9-09cb-4a28-8af3-ee5033477a59" containerID="f36998239c2850c3626bddc56df173f867962bb1471f09722a22029d3da23d11" exitCode=2
Sep 09 20:46:05.623072 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:05.622915    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs" event={"ID":"6aa09bc9-09cb-4a28-8af3-ee5033477a59","Type":"ContainerDied","Data":"f36998239c2850c3626bddc56df173f867962bb1471f09722a22029d3da23d11"}
Sep 09 20:46:07.457163 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:07.457124    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs"
Sep 09 20:46:07.534503 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:07.534439    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6aa09bc9-09cb-4a28-8af3-ee5033477a59-kube-rbac-proxy-sar-config\") pod \"6aa09bc9-09cb-4a28-8af3-ee5033477a59\" (UID: \"6aa09bc9-09cb-4a28-8af3-ee5033477a59\") "
Sep 09 20:46:07.534503 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:07.534475    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6aa09bc9-09cb-4a28-8af3-ee5033477a59-kserve-provision-location\") pod \"6aa09bc9-09cb-4a28-8af3-ee5033477a59\" (UID: \"6aa09bc9-09cb-4a28-8af3-ee5033477a59\") "
Sep 09 20:46:07.534656 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:07.534520    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6aa09bc9-09cb-4a28-8af3-ee5033477a59-proxy-tls\") pod \"6aa09bc9-09cb-4a28-8af3-ee5033477a59\" (UID: \"6aa09bc9-09cb-4a28-8af3-ee5033477a59\") "
Sep 09 20:46:07.534656 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:07.534543    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ktzh\" (UniqueName: \"kubernetes.io/projected/6aa09bc9-09cb-4a28-8af3-ee5033477a59-kube-api-access-7ktzh\") pod \"6aa09bc9-09cb-4a28-8af3-ee5033477a59\" (UID: \"6aa09bc9-09cb-4a28-8af3-ee5033477a59\") "
Sep 09 20:46:07.534656 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:07.534580    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6aa09bc9-09cb-4a28-8af3-ee5033477a59-cabundle-cert\") pod \"6aa09bc9-09cb-4a28-8af3-ee5033477a59\" (UID: \"6aa09bc9-09cb-4a28-8af3-ee5033477a59\") "
Sep 09 20:46:07.534916 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:07.534806    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aa09bc9-09cb-4a28-8af3-ee5033477a59-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6aa09bc9-09cb-4a28-8af3-ee5033477a59" (UID: "6aa09bc9-09cb-4a28-8af3-ee5033477a59"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:46:07.534990 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:07.534950    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aa09bc9-09cb-4a28-8af3-ee5033477a59-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6aa09bc9-09cb-4a28-8af3-ee5033477a59" (UID: "6aa09bc9-09cb-4a28-8af3-ee5033477a59"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:46:07.536778 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:07.536756    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6aa09bc9-09cb-4a28-8af3-ee5033477a59-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6aa09bc9-09cb-4a28-8af3-ee5033477a59" (UID: "6aa09bc9-09cb-4a28-8af3-ee5033477a59"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:46:07.536872 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:07.536812    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aa09bc9-09cb-4a28-8af3-ee5033477a59-kube-api-access-7ktzh" (OuterVolumeSpecName: "kube-api-access-7ktzh") pod "6aa09bc9-09cb-4a28-8af3-ee5033477a59" (UID: "6aa09bc9-09cb-4a28-8af3-ee5033477a59"). InnerVolumeSpecName "kube-api-access-7ktzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:46:07.544134 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:07.544110    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aa09bc9-09cb-4a28-8af3-ee5033477a59-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6aa09bc9-09cb-4a28-8af3-ee5033477a59" (UID: "6aa09bc9-09cb-4a28-8af3-ee5033477a59"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:46:07.631762 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:07.631739    2523 generic.go:356] "Generic (PLEG): container finished" podID="6aa09bc9-09cb-4a28-8af3-ee5033477a59" containerID="e95c607d985092bc140ffc67feec8639657d245b1f260c4e231f32cfd7983f9f" exitCode=0
Sep 09 20:46:07.631863 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:07.631793    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs"
Sep 09 20:46:07.631863 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:07.631832    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs" event={"ID":"6aa09bc9-09cb-4a28-8af3-ee5033477a59","Type":"ContainerDied","Data":"e95c607d985092bc140ffc67feec8639657d245b1f260c4e231f32cfd7983f9f"}
Sep 09 20:46:07.631986 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:07.631903    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs" event={"ID":"6aa09bc9-09cb-4a28-8af3-ee5033477a59","Type":"ContainerDied","Data":"05dba9b3ff21a9df32ad599099a60e3aa2381475a70586ec35228874a630518d"}
Sep 09 20:46:07.631986 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:07.631944    2523 scope.go:117] "RemoveContainer" containerID="f36998239c2850c3626bddc56df173f867962bb1471f09722a22029d3da23d11"
Sep 09 20:46:07.634925 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:07.634905    2523 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6aa09bc9-09cb-4a28-8af3-ee5033477a59-proxy-tls\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:46:07.634925 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:07.634926    2523 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ktzh\" (UniqueName: \"kubernetes.io/projected/6aa09bc9-09cb-4a28-8af3-ee5033477a59-kube-api-access-7ktzh\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:46:07.635034 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:07.634937    2523 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6aa09bc9-09cb-4a28-8af3-ee5033477a59-cabundle-cert\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:46:07.635034 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:07.634947    2523 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6aa09bc9-09cb-4a28-8af3-ee5033477a59-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:46:07.635034 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:07.634955    2523 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6aa09bc9-09cb-4a28-8af3-ee5033477a59-kserve-provision-location\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:46:07.639901 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:07.639882    2523 scope.go:117] "RemoveContainer" containerID="e95c607d985092bc140ffc67feec8639657d245b1f260c4e231f32cfd7983f9f"
Sep 09 20:46:07.647121 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:07.647106    2523 scope.go:117] "RemoveContainer" containerID="7b764cb1dfb7ccc18cd4cb83f97c0b7fee8047dfab1a5f0f67b22de51d5f941f"
Sep 09 20:46:07.653777 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:07.653763    2523 scope.go:117] "RemoveContainer" containerID="f36998239c2850c3626bddc56df173f867962bb1471f09722a22029d3da23d11"
Sep 09 20:46:07.654041 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:46:07.654024    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f36998239c2850c3626bddc56df173f867962bb1471f09722a22029d3da23d11\": container with ID starting with f36998239c2850c3626bddc56df173f867962bb1471f09722a22029d3da23d11 not found: ID does not exist" containerID="f36998239c2850c3626bddc56df173f867962bb1471f09722a22029d3da23d11"
Sep 09 20:46:07.654088 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:07.654047    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f36998239c2850c3626bddc56df173f867962bb1471f09722a22029d3da23d11"} err="failed to get container status \"f36998239c2850c3626bddc56df173f867962bb1471f09722a22029d3da23d11\": rpc error: code = NotFound desc = could not find container \"f36998239c2850c3626bddc56df173f867962bb1471f09722a22029d3da23d11\": container with ID starting with f36998239c2850c3626bddc56df173f867962bb1471f09722a22029d3da23d11 not found: ID does not exist"
Sep 09 20:46:07.654088 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:07.654062    2523 scope.go:117] "RemoveContainer" containerID="e95c607d985092bc140ffc67feec8639657d245b1f260c4e231f32cfd7983f9f"
Sep 09 20:46:07.654331 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:46:07.654316    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e95c607d985092bc140ffc67feec8639657d245b1f260c4e231f32cfd7983f9f\": container with ID starting with e95c607d985092bc140ffc67feec8639657d245b1f260c4e231f32cfd7983f9f not found: ID does not exist" containerID="e95c607d985092bc140ffc67feec8639657d245b1f260c4e231f32cfd7983f9f"
Sep 09 20:46:07.654388 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:07.654335    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e95c607d985092bc140ffc67feec8639657d245b1f260c4e231f32cfd7983f9f"} err="failed to get container status \"e95c607d985092bc140ffc67feec8639657d245b1f260c4e231f32cfd7983f9f\": rpc error: code = NotFound desc = could not find container \"e95c607d985092bc140ffc67feec8639657d245b1f260c4e231f32cfd7983f9f\": container with ID starting with e95c607d985092bc140ffc67feec8639657d245b1f260c4e231f32cfd7983f9f not found: ID does not exist"
Sep 09 20:46:07.654388 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:07.654347    2523 scope.go:117] "RemoveContainer" containerID="7b764cb1dfb7ccc18cd4cb83f97c0b7fee8047dfab1a5f0f67b22de51d5f941f"
Sep 09 20:46:07.654567 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:46:07.654547    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b764cb1dfb7ccc18cd4cb83f97c0b7fee8047dfab1a5f0f67b22de51d5f941f\": container with ID starting with 7b764cb1dfb7ccc18cd4cb83f97c0b7fee8047dfab1a5f0f67b22de51d5f941f not found: ID does not exist" containerID="7b764cb1dfb7ccc18cd4cb83f97c0b7fee8047dfab1a5f0f67b22de51d5f941f"
Sep 09 20:46:07.654628 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:07.654574    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b764cb1dfb7ccc18cd4cb83f97c0b7fee8047dfab1a5f0f67b22de51d5f941f"} err="failed to get container status \"7b764cb1dfb7ccc18cd4cb83f97c0b7fee8047dfab1a5f0f67b22de51d5f941f\": rpc error: code = NotFound desc = could not find container \"7b764cb1dfb7ccc18cd4cb83f97c0b7fee8047dfab1a5f0f67b22de51d5f941f\": container with ID starting with 7b764cb1dfb7ccc18cd4cb83f97c0b7fee8047dfab1a5f0f67b22de51d5f941f not found: ID does not exist"
Sep 09 20:46:07.660717 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:07.660694    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs"]
Sep 09 20:46:07.665331 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:07.665310    2523 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs"]
Sep 09 20:46:07.806465 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:07.806396    2523 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6aa09bc9-09cb-4a28-8af3-ee5033477a59" path="/var/lib/kubelet/pods/6aa09bc9-09cb-4a28-8af3-ee5033477a59/volumes"
Sep 09 20:46:08.414784 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:08.414752    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs" podUID="6aa09bc9-09cb-4a28-8af3-ee5033477a59" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.51:8643/healthz\": context deadline exceeded"
Sep 09 20:46:08.420049 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:08.420026    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qdmbs" podUID="6aa09bc9-09cb-4a28-8af3-ee5033477a59" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: i/o timeout"
Sep 09 20:46:10.402316 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:10.402284    2523 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9rlfx"]
Sep 09 20:46:10.402668 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:10.402529    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6aa09bc9-09cb-4a28-8af3-ee5033477a59" containerName="kserve-container"
Sep 09 20:46:10.402668 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:10.402539    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="6aa09bc9-09cb-4a28-8af3-ee5033477a59" containerName="kserve-container"
Sep 09 20:46:10.402668 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:10.402549    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6aa09bc9-09cb-4a28-8af3-ee5033477a59" containerName="storage-initializer"
Sep 09 20:46:10.402668 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:10.402555    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="6aa09bc9-09cb-4a28-8af3-ee5033477a59" containerName="storage-initializer"
Sep 09 20:46:10.402668 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:10.402566    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6aa09bc9-09cb-4a28-8af3-ee5033477a59" containerName="kube-rbac-proxy"
Sep 09 20:46:10.402668 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:10.402571    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="6aa09bc9-09cb-4a28-8af3-ee5033477a59" containerName="kube-rbac-proxy"
Sep 09 20:46:10.402668 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:10.402613    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="6aa09bc9-09cb-4a28-8af3-ee5033477a59" containerName="kserve-container"
Sep 09 20:46:10.402668 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:10.402622    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="6aa09bc9-09cb-4a28-8af3-ee5033477a59" containerName="kube-rbac-proxy"
Sep 09 20:46:10.411026 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:10.410990    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9rlfx"
Sep 09 20:46:10.414370 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:10.414347    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
Sep 09 20:46:10.414850 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:10.414811    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 20:46:10.414987 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:10.414899    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:46:10.414987 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:10.414948    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\""
Sep 09 20:46:10.415349 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:10.415331    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-ks7mb\""
Sep 09 20:46:10.415473 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:10.415382    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 20:46:10.415616 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:10.415601    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 20:46:10.428870 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:10.428850    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9rlfx"]
Sep 09 20:46:10.451497 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:10.451478    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9e0f08e2-6eed-4cfe-b988-3ae71544f5ff-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-9rlfx\" (UID: \"9e0f08e2-6eed-4cfe-b988-3ae71544f5ff\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9rlfx"
Sep 09 20:46:10.451620 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:10.451514    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e0f08e2-6eed-4cfe-b988-3ae71544f5ff-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-9rlfx\" (UID: \"9e0f08e2-6eed-4cfe-b988-3ae71544f5ff\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9rlfx"
Sep 09 20:46:10.451620 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:10.451535    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rr2g\" (UniqueName: \"kubernetes.io/projected/9e0f08e2-6eed-4cfe-b988-3ae71544f5ff-kube-api-access-6rr2g\") pod \"isvc-pmml-predictor-8674795f59-9rlfx\" (UID: \"9e0f08e2-6eed-4cfe-b988-3ae71544f5ff\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9rlfx"
Sep 09 20:46:10.451620 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:10.451564    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e0f08e2-6eed-4cfe-b988-3ae71544f5ff-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-9rlfx\" (UID: \"9e0f08e2-6eed-4cfe-b988-3ae71544f5ff\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9rlfx"
Sep 09 20:46:10.451620 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:10.451591    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e0f08e2-6eed-4cfe-b988-3ae71544f5ff-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-9rlfx\" (UID: \"9e0f08e2-6eed-4cfe-b988-3ae71544f5ff\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9rlfx"
Sep 09 20:46:10.552493 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:10.552472    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e0f08e2-6eed-4cfe-b988-3ae71544f5ff-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-9rlfx\" (UID: \"9e0f08e2-6eed-4cfe-b988-3ae71544f5ff\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9rlfx"
Sep 09 20:46:10.552583 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:10.552499    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rr2g\" (UniqueName: \"kubernetes.io/projected/9e0f08e2-6eed-4cfe-b988-3ae71544f5ff-kube-api-access-6rr2g\") pod \"isvc-pmml-predictor-8674795f59-9rlfx\" (UID: \"9e0f08e2-6eed-4cfe-b988-3ae71544f5ff\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9rlfx"
Sep 09 20:46:10.552583 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:10.552520    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e0f08e2-6eed-4cfe-b988-3ae71544f5ff-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-9rlfx\" (UID: \"9e0f08e2-6eed-4cfe-b988-3ae71544f5ff\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9rlfx"
Sep 09 20:46:10.552583 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:10.552537    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e0f08e2-6eed-4cfe-b988-3ae71544f5ff-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-9rlfx\" (UID: \"9e0f08e2-6eed-4cfe-b988-3ae71544f5ff\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9rlfx"
Sep 09 20:46:10.552583 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:10.552569    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9e0f08e2-6eed-4cfe-b988-3ae71544f5ff-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-9rlfx\" (UID: \"9e0f08e2-6eed-4cfe-b988-3ae71544f5ff\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9rlfx"
Sep 09 20:46:10.552960 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:10.552938    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e0f08e2-6eed-4cfe-b988-3ae71544f5ff-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-9rlfx\" (UID: \"9e0f08e2-6eed-4cfe-b988-3ae71544f5ff\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9rlfx"
Sep 09 20:46:10.553217 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:10.553199    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e0f08e2-6eed-4cfe-b988-3ae71544f5ff-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-9rlfx\" (UID: \"9e0f08e2-6eed-4cfe-b988-3ae71544f5ff\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9rlfx"
Sep 09 20:46:10.553290 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:10.553275    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9e0f08e2-6eed-4cfe-b988-3ae71544f5ff-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-9rlfx\" (UID: \"9e0f08e2-6eed-4cfe-b988-3ae71544f5ff\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9rlfx"
Sep 09 20:46:10.555144 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:10.555127    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e0f08e2-6eed-4cfe-b988-3ae71544f5ff-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-9rlfx\" (UID: \"9e0f08e2-6eed-4cfe-b988-3ae71544f5ff\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9rlfx"
Sep 09 20:46:10.568689 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:10.568671    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rr2g\" (UniqueName: \"kubernetes.io/projected/9e0f08e2-6eed-4cfe-b988-3ae71544f5ff-kube-api-access-6rr2g\") pod \"isvc-pmml-predictor-8674795f59-9rlfx\" (UID: \"9e0f08e2-6eed-4cfe-b988-3ae71544f5ff\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9rlfx"
Sep 09 20:46:10.720232 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:10.720181    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9rlfx"
Sep 09 20:46:11.004388 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:11.004324    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9rlfx"]
Sep 09 20:46:11.007785 ip-10-0-132-226 kubenswrapper[2523]: W0909 20:46:11.007758    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e0f08e2_6eed_4cfe_b988_3ae71544f5ff.slice/crio-3801f1b0f9b36b2c5869bd139c991c84e6dd11674ba3724c79901f1aa8b7d16c WatchSource:0}: Error finding container 3801f1b0f9b36b2c5869bd139c991c84e6dd11674ba3724c79901f1aa8b7d16c: Status 404 returned error can't find the container with id 3801f1b0f9b36b2c5869bd139c991c84e6dd11674ba3724c79901f1aa8b7d16c
Sep 09 20:46:11.650341 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:11.650308    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9rlfx" event={"ID":"9e0f08e2-6eed-4cfe-b988-3ae71544f5ff","Type":"ContainerStarted","Data":"7452f02c9630d7fb1250d7cdc43b361f2e761fdaa84eaf863924b5efd1573915"}
Sep 09 20:46:11.650712 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:11.650373    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9rlfx" event={"ID":"9e0f08e2-6eed-4cfe-b988-3ae71544f5ff","Type":"ContainerStarted","Data":"3801f1b0f9b36b2c5869bd139c991c84e6dd11674ba3724c79901f1aa8b7d16c"}
Sep 09 20:46:15.668137 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:15.668105    2523 generic.go:356] "Generic (PLEG): container finished" podID="9e0f08e2-6eed-4cfe-b988-3ae71544f5ff" containerID="7452f02c9630d7fb1250d7cdc43b361f2e761fdaa84eaf863924b5efd1573915" exitCode=0
Sep 09 20:46:15.668621 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:15.668183    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9rlfx" event={"ID":"9e0f08e2-6eed-4cfe-b988-3ae71544f5ff","Type":"ContainerDied","Data":"7452f02c9630d7fb1250d7cdc43b361f2e761fdaa84eaf863924b5efd1573915"}
Sep 09 20:46:23.702022 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:23.701989    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9rlfx" event={"ID":"9e0f08e2-6eed-4cfe-b988-3ae71544f5ff","Type":"ContainerStarted","Data":"56b88a76b5d089c864847c57aa8600d3b08f907e8e1e064fe912fe5cb106b59c"}
Sep 09 20:46:23.702416 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:23.702050    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9rlfx" event={"ID":"9e0f08e2-6eed-4cfe-b988-3ae71544f5ff","Type":"ContainerStarted","Data":"b352096b9752c222657143d95cdc38b0d0d6a4ce2a93c2c2745c07bda86abada"}
Sep 09 20:46:23.702416 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:23.702255    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9rlfx"
Sep 09 20:46:23.751608 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:23.751565    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9rlfx" podStartSLOduration=3.036081039 podStartE2EDuration="13.751553685s" totalImagesPullingTime="6.984808621s" totalInitContainerRuntime="3.730664025s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:46:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:46:23.741953693 +0000 UTC m=+3088.413972701" watchObservedRunningTime="2026-09-09 20:46:23.751553685 +0000 UTC m=+3088.423572682"
Sep 09 20:46:24.706005 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:24.705973    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9rlfx"
Sep 09 20:46:24.707205 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:24.707173    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9rlfx" podUID="9e0f08e2-6eed-4cfe-b988-3ae71544f5ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 09 20:46:25.710023 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:25.709978    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9rlfx" podUID="9e0f08e2-6eed-4cfe-b988-3ae71544f5ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 09 20:46:30.714231 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:30.714202    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9rlfx"
Sep 09 20:46:30.714719 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:30.714693    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9rlfx" podUID="9e0f08e2-6eed-4cfe-b988-3ae71544f5ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 09 20:46:40.715127 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:40.715081    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9rlfx" podUID="9e0f08e2-6eed-4cfe-b988-3ae71544f5ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 09 20:46:50.714893 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:46:50.714857    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9rlfx" podUID="9e0f08e2-6eed-4cfe-b988-3ae71544f5ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 09 20:47:00.715578 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:47:00.715539    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9rlfx" podUID="9e0f08e2-6eed-4cfe-b988-3ae71544f5ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 09 20:47:10.714931 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:47:10.714893    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9rlfx" podUID="9e0f08e2-6eed-4cfe-b988-3ae71544f5ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 09 20:47:20.715122 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:47:20.715082    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9rlfx" podUID="9e0f08e2-6eed-4cfe-b988-3ae71544f5ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 09 20:47:30.714694 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:47:30.714586    2523 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9rlfx" podUID="9e0f08e2-6eed-4cfe-b988-3ae71544f5ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 09 20:47:40.715302 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:47:40.715275    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9rlfx"
Sep 09 20:47:51.571628 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:47:51.571594    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9rlfx"]
Sep 09 20:47:51.572070 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:47:51.571836    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9rlfx" podUID="9e0f08e2-6eed-4cfe-b988-3ae71544f5ff" containerName="kserve-container" containerID="cri-o://b352096b9752c222657143d95cdc38b0d0d6a4ce2a93c2c2745c07bda86abada" gracePeriod=30
Sep 09 20:47:51.572070 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:47:51.571870    2523 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9rlfx" podUID="9e0f08e2-6eed-4cfe-b988-3ae71544f5ff" containerName="kube-rbac-proxy" containerID="cri-o://56b88a76b5d089c864847c57aa8600d3b08f907e8e1e064fe912fe5cb106b59c" gracePeriod=30
Sep 09 20:47:52.024428 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:47:52.024353    2523 generic.go:356] "Generic (PLEG): container finished" podID="9e0f08e2-6eed-4cfe-b988-3ae71544f5ff" containerID="56b88a76b5d089c864847c57aa8600d3b08f907e8e1e064fe912fe5cb106b59c" exitCode=2
Sep 09 20:47:52.024554 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:47:52.024426    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9rlfx" event={"ID":"9e0f08e2-6eed-4cfe-b988-3ae71544f5ff","Type":"ContainerDied","Data":"56b88a76b5d089c864847c57aa8600d3b08f907e8e1e064fe912fe5cb106b59c"}
Sep 09 20:47:54.308741 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:47:54.308721    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9rlfx"
Sep 09 20:47:54.353396 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:47:54.353370    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e0f08e2-6eed-4cfe-b988-3ae71544f5ff-kserve-provision-location\") pod \"9e0f08e2-6eed-4cfe-b988-3ae71544f5ff\" (UID: \"9e0f08e2-6eed-4cfe-b988-3ae71544f5ff\") "
Sep 09 20:47:54.353523 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:47:54.353409    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e0f08e2-6eed-4cfe-b988-3ae71544f5ff-proxy-tls\") pod \"9e0f08e2-6eed-4cfe-b988-3ae71544f5ff\" (UID: \"9e0f08e2-6eed-4cfe-b988-3ae71544f5ff\") "
Sep 09 20:47:54.353523 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:47:54.353441    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rr2g\" (UniqueName: \"kubernetes.io/projected/9e0f08e2-6eed-4cfe-b988-3ae71544f5ff-kube-api-access-6rr2g\") pod \"9e0f08e2-6eed-4cfe-b988-3ae71544f5ff\" (UID: \"9e0f08e2-6eed-4cfe-b988-3ae71544f5ff\") "
Sep 09 20:47:54.353647 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:47:54.353558    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9e0f08e2-6eed-4cfe-b988-3ae71544f5ff-cabundle-cert\") pod \"9e0f08e2-6eed-4cfe-b988-3ae71544f5ff\" (UID: \"9e0f08e2-6eed-4cfe-b988-3ae71544f5ff\") "
Sep 09 20:47:54.353706 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:47:54.353648    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e0f08e2-6eed-4cfe-b988-3ae71544f5ff-kube-rbac-proxy-sar-config\") pod \"9e0f08e2-6eed-4cfe-b988-3ae71544f5ff\" (UID: \"9e0f08e2-6eed-4cfe-b988-3ae71544f5ff\") "
Sep 09 20:47:54.353768 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:47:54.353720    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e0f08e2-6eed-4cfe-b988-3ae71544f5ff-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9e0f08e2-6eed-4cfe-b988-3ae71544f5ff" (UID: "9e0f08e2-6eed-4cfe-b988-3ae71544f5ff"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:47:54.353948 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:47:54.353929    2523 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e0f08e2-6eed-4cfe-b988-3ae71544f5ff-kserve-provision-location\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:47:54.354226 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:47:54.353985    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e0f08e2-6eed-4cfe-b988-3ae71544f5ff-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9e0f08e2-6eed-4cfe-b988-3ae71544f5ff" (UID: "9e0f08e2-6eed-4cfe-b988-3ae71544f5ff"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:47:54.354226 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:47:54.354034    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e0f08e2-6eed-4cfe-b988-3ae71544f5ff-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9e0f08e2-6eed-4cfe-b988-3ae71544f5ff" (UID: "9e0f08e2-6eed-4cfe-b988-3ae71544f5ff"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:47:54.355640 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:47:54.355621    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e0f08e2-6eed-4cfe-b988-3ae71544f5ff-kube-api-access-6rr2g" (OuterVolumeSpecName: "kube-api-access-6rr2g") pod "9e0f08e2-6eed-4cfe-b988-3ae71544f5ff" (UID: "9e0f08e2-6eed-4cfe-b988-3ae71544f5ff"). InnerVolumeSpecName "kube-api-access-6rr2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:47:54.355730 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:47:54.355662    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e0f08e2-6eed-4cfe-b988-3ae71544f5ff-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9e0f08e2-6eed-4cfe-b988-3ae71544f5ff" (UID: "9e0f08e2-6eed-4cfe-b988-3ae71544f5ff"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:47:54.454934 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:47:54.454877    2523 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9e0f08e2-6eed-4cfe-b988-3ae71544f5ff-cabundle-cert\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:47:54.454934 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:47:54.454902    2523 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e0f08e2-6eed-4cfe-b988-3ae71544f5ff-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:47:54.454934 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:47:54.454916    2523 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e0f08e2-6eed-4cfe-b988-3ae71544f5ff-proxy-tls\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:47:54.454934 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:47:54.454929    2523 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rr2g\" (UniqueName: \"kubernetes.io/projected/9e0f08e2-6eed-4cfe-b988-3ae71544f5ff-kube-api-access-6rr2g\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:47:55.036310 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:47:55.036279    2523 generic.go:356] "Generic (PLEG): container finished" podID="9e0f08e2-6eed-4cfe-b988-3ae71544f5ff" containerID="b352096b9752c222657143d95cdc38b0d0d6a4ce2a93c2c2745c07bda86abada" exitCode=0
Sep 09 20:47:55.036459 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:47:55.036339    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9rlfx"
Sep 09 20:47:55.036459 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:47:55.036364    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9rlfx" event={"ID":"9e0f08e2-6eed-4cfe-b988-3ae71544f5ff","Type":"ContainerDied","Data":"b352096b9752c222657143d95cdc38b0d0d6a4ce2a93c2c2745c07bda86abada"}
Sep 09 20:47:55.036459 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:47:55.036435    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9rlfx" event={"ID":"9e0f08e2-6eed-4cfe-b988-3ae71544f5ff","Type":"ContainerDied","Data":"3801f1b0f9b36b2c5869bd139c991c84e6dd11674ba3724c79901f1aa8b7d16c"}
Sep 09 20:47:55.036602 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:47:55.036462    2523 scope.go:117] "RemoveContainer" containerID="56b88a76b5d089c864847c57aa8600d3b08f907e8e1e064fe912fe5cb106b59c"
Sep 09 20:47:55.044851 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:47:55.044835    2523 scope.go:117] "RemoveContainer" containerID="b352096b9752c222657143d95cdc38b0d0d6a4ce2a93c2c2745c07bda86abada"
Sep 09 20:47:55.053905 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:47:55.053886    2523 scope.go:117] "RemoveContainer" containerID="7452f02c9630d7fb1250d7cdc43b361f2e761fdaa84eaf863924b5efd1573915"
Sep 09 20:47:55.061310 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:47:55.061294    2523 scope.go:117] "RemoveContainer" containerID="56b88a76b5d089c864847c57aa8600d3b08f907e8e1e064fe912fe5cb106b59c"
Sep 09 20:47:55.061565 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:47:55.061548    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56b88a76b5d089c864847c57aa8600d3b08f907e8e1e064fe912fe5cb106b59c\": container with ID starting with 56b88a76b5d089c864847c57aa8600d3b08f907e8e1e064fe912fe5cb106b59c not found: ID does not exist" containerID="56b88a76b5d089c864847c57aa8600d3b08f907e8e1e064fe912fe5cb106b59c"
Sep 09 20:47:55.061608 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:47:55.061572    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56b88a76b5d089c864847c57aa8600d3b08f907e8e1e064fe912fe5cb106b59c"} err="failed to get container status \"56b88a76b5d089c864847c57aa8600d3b08f907e8e1e064fe912fe5cb106b59c\": rpc error: code = NotFound desc = could not find container \"56b88a76b5d089c864847c57aa8600d3b08f907e8e1e064fe912fe5cb106b59c\": container with ID starting with 56b88a76b5d089c864847c57aa8600d3b08f907e8e1e064fe912fe5cb106b59c not found: ID does not exist"
Sep 09 20:47:55.061608 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:47:55.061588    2523 scope.go:117] "RemoveContainer" containerID="b352096b9752c222657143d95cdc38b0d0d6a4ce2a93c2c2745c07bda86abada"
Sep 09 20:47:55.061842 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:47:55.061823    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b352096b9752c222657143d95cdc38b0d0d6a4ce2a93c2c2745c07bda86abada\": container with ID starting with b352096b9752c222657143d95cdc38b0d0d6a4ce2a93c2c2745c07bda86abada not found: ID does not exist" containerID="b352096b9752c222657143d95cdc38b0d0d6a4ce2a93c2c2745c07bda86abada"
Sep 09 20:47:55.061887 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:47:55.061848    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b352096b9752c222657143d95cdc38b0d0d6a4ce2a93c2c2745c07bda86abada"} err="failed to get container status \"b352096b9752c222657143d95cdc38b0d0d6a4ce2a93c2c2745c07bda86abada\": rpc error: code = NotFound desc = could not find container \"b352096b9752c222657143d95cdc38b0d0d6a4ce2a93c2c2745c07bda86abada\": container with ID starting with b352096b9752c222657143d95cdc38b0d0d6a4ce2a93c2c2745c07bda86abada not found: ID does not exist"
Sep 09 20:47:55.061887 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:47:55.061864    2523 scope.go:117] "RemoveContainer" containerID="7452f02c9630d7fb1250d7cdc43b361f2e761fdaa84eaf863924b5efd1573915"
Sep 09 20:47:55.062101 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:47:55.062084    2523 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7452f02c9630d7fb1250d7cdc43b361f2e761fdaa84eaf863924b5efd1573915\": container with ID starting with 7452f02c9630d7fb1250d7cdc43b361f2e761fdaa84eaf863924b5efd1573915 not found: ID does not exist" containerID="7452f02c9630d7fb1250d7cdc43b361f2e761fdaa84eaf863924b5efd1573915"
Sep 09 20:47:55.062174 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:47:55.062104    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7452f02c9630d7fb1250d7cdc43b361f2e761fdaa84eaf863924b5efd1573915"} err="failed to get container status \"7452f02c9630d7fb1250d7cdc43b361f2e761fdaa84eaf863924b5efd1573915\": rpc error: code = NotFound desc = could not find container \"7452f02c9630d7fb1250d7cdc43b361f2e761fdaa84eaf863924b5efd1573915\": container with ID starting with 7452f02c9630d7fb1250d7cdc43b361f2e761fdaa84eaf863924b5efd1573915 not found: ID does not exist"
Sep 09 20:47:55.071538 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:47:55.071518    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9rlfx"]
Sep 09 20:47:55.081402 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:47:55.081384    2523 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9rlfx"]
Sep 09 20:47:55.807593 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:47:55.807568    2523 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e0f08e2-6eed-4cfe-b988-3ae71544f5ff" path="/var/lib/kubelet/pods/9e0f08e2-6eed-4cfe-b988-3ae71544f5ff/volumes"
Sep 09 20:50:22.822077 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:22.822045    2523 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-887399-predictor-5c6454c5f7-b2q6z"]
Sep 09 20:50:22.822558 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:22.822330    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e0f08e2-6eed-4cfe-b988-3ae71544f5ff" containerName="storage-initializer"
Sep 09 20:50:22.822558 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:22.822341    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e0f08e2-6eed-4cfe-b988-3ae71544f5ff" containerName="storage-initializer"
Sep 09 20:50:22.822558 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:22.822351    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e0f08e2-6eed-4cfe-b988-3ae71544f5ff" containerName="kserve-container"
Sep 09 20:50:22.822558 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:22.822357    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e0f08e2-6eed-4cfe-b988-3ae71544f5ff" containerName="kserve-container"
Sep 09 20:50:22.822558 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:22.822366    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e0f08e2-6eed-4cfe-b988-3ae71544f5ff" containerName="kube-rbac-proxy"
Sep 09 20:50:22.822558 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:22.822371    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e0f08e2-6eed-4cfe-b988-3ae71544f5ff" containerName="kube-rbac-proxy"
Sep 09 20:50:22.822558 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:22.822421    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e0f08e2-6eed-4cfe-b988-3ae71544f5ff" containerName="kube-rbac-proxy"
Sep 09 20:50:22.822558 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:22.822430    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e0f08e2-6eed-4cfe-b988-3ae71544f5ff" containerName="kserve-container"
Sep 09 20:50:22.825289 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:22.825272    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-887399-predictor-5c6454c5f7-b2q6z"
Sep 09 20:50:22.828184 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:22.828133    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-887399-predictor-serving-cert\""
Sep 09 20:50:22.828324 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:22.828181    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-887399-dockercfg-8kblm\""
Sep 09 20:50:22.828683 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:22.828666    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 20:50:22.829320 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:22.829298    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:50:22.829402 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:22.829356    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 20:50:22.829402 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:22.829298    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-887399\""
Sep 09 20:50:22.829627 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:22.829604    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-887399-kube-rbac-proxy-sar-config\""
Sep 09 20:50:22.847495 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:22.847472    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-887399-predictor-5c6454c5f7-b2q6z"]
Sep 09 20:50:22.926777 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:22.926752    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7945721-0b16-4b1f-84f9-7a7953021c34-kserve-provision-location\") pod \"isvc-secondary-887399-predictor-5c6454c5f7-b2q6z\" (UID: \"c7945721-0b16-4b1f-84f9-7a7953021c34\") " pod="kserve-ci-e2e-test/isvc-secondary-887399-predictor-5c6454c5f7-b2q6z"
Sep 09 20:50:22.926869 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:22.926796    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7945721-0b16-4b1f-84f9-7a7953021c34-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-887399-predictor-5c6454c5f7-b2q6z\" (UID: \"c7945721-0b16-4b1f-84f9-7a7953021c34\") " pod="kserve-ci-e2e-test/isvc-secondary-887399-predictor-5c6454c5f7-b2q6z"
Sep 09 20:50:22.926869 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:22.926813    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdmt4\" (UniqueName: \"kubernetes.io/projected/c7945721-0b16-4b1f-84f9-7a7953021c34-kube-api-access-wdmt4\") pod \"isvc-secondary-887399-predictor-5c6454c5f7-b2q6z\" (UID: \"c7945721-0b16-4b1f-84f9-7a7953021c34\") " pod="kserve-ci-e2e-test/isvc-secondary-887399-predictor-5c6454c5f7-b2q6z"
Sep 09 20:50:22.926869 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:22.926829    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7945721-0b16-4b1f-84f9-7a7953021c34-cabundle-cert\") pod \"isvc-secondary-887399-predictor-5c6454c5f7-b2q6z\" (UID: \"c7945721-0b16-4b1f-84f9-7a7953021c34\") " pod="kserve-ci-e2e-test/isvc-secondary-887399-predictor-5c6454c5f7-b2q6z"
Sep 09 20:50:22.926995 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:22.926872    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7945721-0b16-4b1f-84f9-7a7953021c34-proxy-tls\") pod \"isvc-secondary-887399-predictor-5c6454c5f7-b2q6z\" (UID: \"c7945721-0b16-4b1f-84f9-7a7953021c34\") " pod="kserve-ci-e2e-test/isvc-secondary-887399-predictor-5c6454c5f7-b2q6z"
Sep 09 20:50:23.027885 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:23.027855    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7945721-0b16-4b1f-84f9-7a7953021c34-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-887399-predictor-5c6454c5f7-b2q6z\" (UID: \"c7945721-0b16-4b1f-84f9-7a7953021c34\") " pod="kserve-ci-e2e-test/isvc-secondary-887399-predictor-5c6454c5f7-b2q6z"
Sep 09 20:50:23.027985 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:23.027897    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdmt4\" (UniqueName: \"kubernetes.io/projected/c7945721-0b16-4b1f-84f9-7a7953021c34-kube-api-access-wdmt4\") pod \"isvc-secondary-887399-predictor-5c6454c5f7-b2q6z\" (UID: \"c7945721-0b16-4b1f-84f9-7a7953021c34\") " pod="kserve-ci-e2e-test/isvc-secondary-887399-predictor-5c6454c5f7-b2q6z"
Sep 09 20:50:23.027985 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:23.027919    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7945721-0b16-4b1f-84f9-7a7953021c34-cabundle-cert\") pod \"isvc-secondary-887399-predictor-5c6454c5f7-b2q6z\" (UID: \"c7945721-0b16-4b1f-84f9-7a7953021c34\") " pod="kserve-ci-e2e-test/isvc-secondary-887399-predictor-5c6454c5f7-b2q6z"
Sep 09 20:50:23.027985 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:23.027948    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7945721-0b16-4b1f-84f9-7a7953021c34-proxy-tls\") pod \"isvc-secondary-887399-predictor-5c6454c5f7-b2q6z\" (UID: \"c7945721-0b16-4b1f-84f9-7a7953021c34\") " pod="kserve-ci-e2e-test/isvc-secondary-887399-predictor-5c6454c5f7-b2q6z"
Sep 09 20:50:23.028124 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:23.027998    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7945721-0b16-4b1f-84f9-7a7953021c34-kserve-provision-location\") pod \"isvc-secondary-887399-predictor-5c6454c5f7-b2q6z\" (UID: \"c7945721-0b16-4b1f-84f9-7a7953021c34\") " pod="kserve-ci-e2e-test/isvc-secondary-887399-predictor-5c6454c5f7-b2q6z"
Sep 09 20:50:23.028386 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:23.028371    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7945721-0b16-4b1f-84f9-7a7953021c34-kserve-provision-location\") pod \"isvc-secondary-887399-predictor-5c6454c5f7-b2q6z\" (UID: \"c7945721-0b16-4b1f-84f9-7a7953021c34\") " pod="kserve-ci-e2e-test/isvc-secondary-887399-predictor-5c6454c5f7-b2q6z"
Sep 09 20:50:23.028559 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:23.028543    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7945721-0b16-4b1f-84f9-7a7953021c34-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-887399-predictor-5c6454c5f7-b2q6z\" (UID: \"c7945721-0b16-4b1f-84f9-7a7953021c34\") " pod="kserve-ci-e2e-test/isvc-secondary-887399-predictor-5c6454c5f7-b2q6z"
Sep 09 20:50:23.028630 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:23.028614    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7945721-0b16-4b1f-84f9-7a7953021c34-cabundle-cert\") pod \"isvc-secondary-887399-predictor-5c6454c5f7-b2q6z\" (UID: \"c7945721-0b16-4b1f-84f9-7a7953021c34\") " pod="kserve-ci-e2e-test/isvc-secondary-887399-predictor-5c6454c5f7-b2q6z"
Sep 09 20:50:23.030467 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:23.030450    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7945721-0b16-4b1f-84f9-7a7953021c34-proxy-tls\") pod \"isvc-secondary-887399-predictor-5c6454c5f7-b2q6z\" (UID: \"c7945721-0b16-4b1f-84f9-7a7953021c34\") " pod="kserve-ci-e2e-test/isvc-secondary-887399-predictor-5c6454c5f7-b2q6z"
Sep 09 20:50:23.035905 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:23.035880    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdmt4\" (UniqueName: \"kubernetes.io/projected/c7945721-0b16-4b1f-84f9-7a7953021c34-kube-api-access-wdmt4\") pod \"isvc-secondary-887399-predictor-5c6454c5f7-b2q6z\" (UID: \"c7945721-0b16-4b1f-84f9-7a7953021c34\") " pod="kserve-ci-e2e-test/isvc-secondary-887399-predictor-5c6454c5f7-b2q6z"
Sep 09 20:50:23.135158 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:23.135099    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-887399-predictor-5c6454c5f7-b2q6z"
Sep 09 20:50:23.406645 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:23.406573    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-887399-predictor-5c6454c5f7-b2q6z"]
Sep 09 20:50:23.408740 ip-10-0-132-226 kubenswrapper[2523]: W0909 20:50:23.408713    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7945721_0b16_4b1f_84f9_7a7953021c34.slice/crio-172087019aa19cf724e92c57b3e3f7420ef122a77eb94890932346e128827e11 WatchSource:0}: Error finding container 172087019aa19cf724e92c57b3e3f7420ef122a77eb94890932346e128827e11: Status 404 returned error can't find the container with id 172087019aa19cf724e92c57b3e3f7420ef122a77eb94890932346e128827e11
Sep 09 20:50:23.557809 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:23.557774    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-887399-predictor-5c6454c5f7-b2q6z" event={"ID":"c7945721-0b16-4b1f-84f9-7a7953021c34","Type":"ContainerStarted","Data":"03afc6ea3f296226e8fe1e9a910f0d73d5dea08e0c98033a6de4293404f042ec"}
Sep 09 20:50:23.557949 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:23.557881    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-887399-predictor-5c6454c5f7-b2q6z" event={"ID":"c7945721-0b16-4b1f-84f9-7a7953021c34","Type":"ContainerStarted","Data":"172087019aa19cf724e92c57b3e3f7420ef122a77eb94890932346e128827e11"}
Sep 09 20:50:30.581840 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:30.581815    2523 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-887399-predictor-5c6454c5f7-b2q6z_c7945721-0b16-4b1f-84f9-7a7953021c34/storage-initializer/0.log"
Sep 09 20:50:30.582205 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:30.581849    2523 generic.go:356] "Generic (PLEG): container finished" podID="c7945721-0b16-4b1f-84f9-7a7953021c34" containerID="03afc6ea3f296226e8fe1e9a910f0d73d5dea08e0c98033a6de4293404f042ec" exitCode=1
Sep 09 20:50:30.582205 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:30.581893    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-887399-predictor-5c6454c5f7-b2q6z" event={"ID":"c7945721-0b16-4b1f-84f9-7a7953021c34","Type":"ContainerDied","Data":"03afc6ea3f296226e8fe1e9a910f0d73d5dea08e0c98033a6de4293404f042ec"}
Sep 09 20:50:31.586897 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:31.586869    2523 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-887399-predictor-5c6454c5f7-b2q6z_c7945721-0b16-4b1f-84f9-7a7953021c34/storage-initializer/0.log"
Sep 09 20:50:31.587313 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:31.586933    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-887399-predictor-5c6454c5f7-b2q6z" event={"ID":"c7945721-0b16-4b1f-84f9-7a7953021c34","Type":"ContainerStarted","Data":"1b82eb9919b0a64ac56d9df2cb96768e5c6a3681e02073de15be86215b712f75"}
Sep 09 20:50:34.598931 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:34.598902    2523 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-887399-predictor-5c6454c5f7-b2q6z_c7945721-0b16-4b1f-84f9-7a7953021c34/storage-initializer/1.log"
Sep 09 20:50:34.599347 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:34.599330    2523 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-887399-predictor-5c6454c5f7-b2q6z_c7945721-0b16-4b1f-84f9-7a7953021c34/storage-initializer/0.log"
Sep 09 20:50:34.599398 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:34.599366    2523 generic.go:356] "Generic (PLEG): container finished" podID="c7945721-0b16-4b1f-84f9-7a7953021c34" containerID="1b82eb9919b0a64ac56d9df2cb96768e5c6a3681e02073de15be86215b712f75" exitCode=1
Sep 09 20:50:34.599441 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:34.599414    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-887399-predictor-5c6454c5f7-b2q6z" event={"ID":"c7945721-0b16-4b1f-84f9-7a7953021c34","Type":"ContainerDied","Data":"1b82eb9919b0a64ac56d9df2cb96768e5c6a3681e02073de15be86215b712f75"}
Sep 09 20:50:34.599496 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:34.599468    2523 scope.go:117] "RemoveContainer" containerID="03afc6ea3f296226e8fe1e9a910f0d73d5dea08e0c98033a6de4293404f042ec"
Sep 09 20:50:34.599809 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:34.599791    2523 scope.go:117] "RemoveContainer" containerID="03afc6ea3f296226e8fe1e9a910f0d73d5dea08e0c98033a6de4293404f042ec"
Sep 09 20:50:34.609536 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:50:34.609514    2523 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-887399-predictor-5c6454c5f7-b2q6z_kserve-ci-e2e-test(c7945721-0b16-4b1f-84f9-7a7953021c34)\"" pod="kserve-ci-e2e-test/isvc-secondary-887399-predictor-5c6454c5f7-b2q6z" podUID="c7945721-0b16-4b1f-84f9-7a7953021c34"
Sep 09 20:50:34.609788 ip-10-0-132-226 kubenswrapper[2523]: E0909 20:50:34.609762    2523 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-887399-predictor-5c6454c5f7-b2q6z_kserve-ci-e2e-test_c7945721-0b16-4b1f-84f9-7a7953021c34_0 in pod sandbox 172087019aa19cf724e92c57b3e3f7420ef122a77eb94890932346e128827e11 from index: no such id: '03afc6ea3f296226e8fe1e9a910f0d73d5dea08e0c98033a6de4293404f042ec'" containerID="03afc6ea3f296226e8fe1e9a910f0d73d5dea08e0c98033a6de4293404f042ec"
Sep 09 20:50:34.609857 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:34.609801    2523 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03afc6ea3f296226e8fe1e9a910f0d73d5dea08e0c98033a6de4293404f042ec"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-887399-predictor-5c6454c5f7-b2q6z_kserve-ci-e2e-test_c7945721-0b16-4b1f-84f9-7a7953021c34_0 in pod sandbox 172087019aa19cf724e92c57b3e3f7420ef122a77eb94890932346e128827e11 from index: no such id: '03afc6ea3f296226e8fe1e9a910f0d73d5dea08e0c98033a6de4293404f042ec'"
Sep 09 20:50:35.604162 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:35.604122    2523 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-887399-predictor-5c6454c5f7-b2q6z_c7945721-0b16-4b1f-84f9-7a7953021c34/storage-initializer/1.log"
Sep 09 20:50:42.738960 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:42.738930    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-887399-predictor-5c6454c5f7-b2q6z"]
Sep 09 20:50:42.863654 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:42.863635    2523 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-887399-predictor-5c6454c5f7-b2q6z_c7945721-0b16-4b1f-84f9-7a7953021c34/storage-initializer/1.log"
Sep 09 20:50:42.866050 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:42.866035    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-887399-predictor-5c6454c5f7-b2q6z"
Sep 09 20:50:42.962389 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:42.962364    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wdmt4\" (UniqueName: \"kubernetes.io/projected/c7945721-0b16-4b1f-84f9-7a7953021c34-kube-api-access-wdmt4\") pod \"c7945721-0b16-4b1f-84f9-7a7953021c34\" (UID: \"c7945721-0b16-4b1f-84f9-7a7953021c34\") "
Sep 09 20:50:42.962496 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:42.962395    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7945721-0b16-4b1f-84f9-7a7953021c34-kserve-provision-location\") pod \"c7945721-0b16-4b1f-84f9-7a7953021c34\" (UID: \"c7945721-0b16-4b1f-84f9-7a7953021c34\") "
Sep 09 20:50:42.962496 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:42.962414    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7945721-0b16-4b1f-84f9-7a7953021c34-cabundle-cert\") pod \"c7945721-0b16-4b1f-84f9-7a7953021c34\" (UID: \"c7945721-0b16-4b1f-84f9-7a7953021c34\") "
Sep 09 20:50:42.962496 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:42.962476    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7945721-0b16-4b1f-84f9-7a7953021c34-kube-rbac-proxy-sar-config\") pod \"c7945721-0b16-4b1f-84f9-7a7953021c34\" (UID: \"c7945721-0b16-4b1f-84f9-7a7953021c34\") "
Sep 09 20:50:42.962665 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:42.962518    2523 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7945721-0b16-4b1f-84f9-7a7953021c34-proxy-tls\") pod \"c7945721-0b16-4b1f-84f9-7a7953021c34\" (UID: \"c7945721-0b16-4b1f-84f9-7a7953021c34\") "
Sep 09 20:50:42.962665 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:42.962612    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7945721-0b16-4b1f-84f9-7a7953021c34-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c7945721-0b16-4b1f-84f9-7a7953021c34" (UID: "c7945721-0b16-4b1f-84f9-7a7953021c34"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:50:42.962787 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:42.962699    2523 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7945721-0b16-4b1f-84f9-7a7953021c34-kserve-provision-location\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:50:42.962866 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:42.962844    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7945721-0b16-4b1f-84f9-7a7953021c34-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c7945721-0b16-4b1f-84f9-7a7953021c34" (UID: "c7945721-0b16-4b1f-84f9-7a7953021c34"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:50:42.962914 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:42.962851    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7945721-0b16-4b1f-84f9-7a7953021c34-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c7945721-0b16-4b1f-84f9-7a7953021c34" (UID: "c7945721-0b16-4b1f-84f9-7a7953021c34"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:50:42.964469 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:42.964451    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7945721-0b16-4b1f-84f9-7a7953021c34-kube-api-access-wdmt4" (OuterVolumeSpecName: "kube-api-access-wdmt4") pod "c7945721-0b16-4b1f-84f9-7a7953021c34" (UID: "c7945721-0b16-4b1f-84f9-7a7953021c34"). InnerVolumeSpecName "kube-api-access-wdmt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:50:42.964619 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:42.964604    2523 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7945721-0b16-4b1f-84f9-7a7953021c34-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c7945721-0b16-4b1f-84f9-7a7953021c34" (UID: "c7945721-0b16-4b1f-84f9-7a7953021c34"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:50:43.063080 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:43.063028    2523 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdmt4\" (UniqueName: \"kubernetes.io/projected/c7945721-0b16-4b1f-84f9-7a7953021c34-kube-api-access-wdmt4\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:50:43.063080 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:43.063051    2523 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7945721-0b16-4b1f-84f9-7a7953021c34-cabundle-cert\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:50:43.063080 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:43.063060    2523 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7945721-0b16-4b1f-84f9-7a7953021c34-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:50:43.063080 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:43.063069    2523 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7945721-0b16-4b1f-84f9-7a7953021c34-proxy-tls\") on node \"ip-10-0-132-226.ec2.internal\" DevicePath \"\""
Sep 09 20:50:43.633730 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:43.633708    2523 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-887399-predictor-5c6454c5f7-b2q6z_c7945721-0b16-4b1f-84f9-7a7953021c34/storage-initializer/1.log"
Sep 09 20:50:43.633904 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:43.633803    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-887399-predictor-5c6454c5f7-b2q6z" event={"ID":"c7945721-0b16-4b1f-84f9-7a7953021c34","Type":"ContainerDied","Data":"172087019aa19cf724e92c57b3e3f7420ef122a77eb94890932346e128827e11"}
Sep 09 20:50:43.633904 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:43.633809    2523 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-887399-predictor-5c6454c5f7-b2q6z"
Sep 09 20:50:43.633904 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:43.633879    2523 scope.go:117] "RemoveContainer" containerID="1b82eb9919b0a64ac56d9df2cb96768e5c6a3681e02073de15be86215b712f75"
Sep 09 20:50:43.670536 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:43.670512    2523 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-887399-predictor-5c6454c5f7-b2q6z"]
Sep 09 20:50:43.676983 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:43.676964    2523 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-887399-predictor-5c6454c5f7-b2q6z"]
Sep 09 20:50:43.806736 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:50:43.806710    2523 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7945721-0b16-4b1f-84f9-7a7953021c34" path="/var/lib/kubelet/pods/c7945721-0b16-4b1f-84f9-7a7953021c34/volumes"
Sep 09 20:51:25.145686 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:25.145655    2523 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5xxks/must-gather-k7428"]
Sep 09 20:51:25.146091 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:25.145906    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7945721-0b16-4b1f-84f9-7a7953021c34" containerName="storage-initializer"
Sep 09 20:51:25.146091 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:25.145916    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7945721-0b16-4b1f-84f9-7a7953021c34" containerName="storage-initializer"
Sep 09 20:51:25.146091 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:25.145930    2523 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7945721-0b16-4b1f-84f9-7a7953021c34" containerName="storage-initializer"
Sep 09 20:51:25.146091 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:25.145935    2523 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7945721-0b16-4b1f-84f9-7a7953021c34" containerName="storage-initializer"
Sep 09 20:51:25.146091 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:25.145981    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7945721-0b16-4b1f-84f9-7a7953021c34" containerName="storage-initializer"
Sep 09 20:51:25.146091 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:25.145989    2523 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7945721-0b16-4b1f-84f9-7a7953021c34" containerName="storage-initializer"
Sep 09 20:51:25.150024 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:25.150005    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5xxks/must-gather-k7428"
Sep 09 20:51:25.152601 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:25.152582    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5xxks\"/\"kube-root-ca.crt\""
Sep 09 20:51:25.152768 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:25.152755    2523 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5xxks\"/\"openshift-service-ca.crt\""
Sep 09 20:51:25.153634 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:25.153614    2523 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5xxks\"/\"default-dockercfg-rdwtx\""
Sep 09 20:51:25.161968 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:25.161948    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5xxks/must-gather-k7428"]
Sep 09 20:51:25.228105 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:25.228082    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/77590cab-f436-4b7b-a9bf-23c29fc00fd3-must-gather-output\") pod \"must-gather-k7428\" (UID: \"77590cab-f436-4b7b-a9bf-23c29fc00fd3\") " pod="openshift-must-gather-5xxks/must-gather-k7428"
Sep 09 20:51:25.228213 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:25.228124    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9m8k\" (UniqueName: \"kubernetes.io/projected/77590cab-f436-4b7b-a9bf-23c29fc00fd3-kube-api-access-s9m8k\") pod \"must-gather-k7428\" (UID: \"77590cab-f436-4b7b-a9bf-23c29fc00fd3\") " pod="openshift-must-gather-5xxks/must-gather-k7428"
Sep 09 20:51:25.329132 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:25.329106    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9m8k\" (UniqueName: \"kubernetes.io/projected/77590cab-f436-4b7b-a9bf-23c29fc00fd3-kube-api-access-s9m8k\") pod \"must-gather-k7428\" (UID: \"77590cab-f436-4b7b-a9bf-23c29fc00fd3\") " pod="openshift-must-gather-5xxks/must-gather-k7428"
Sep 09 20:51:25.329247 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:25.329180    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/77590cab-f436-4b7b-a9bf-23c29fc00fd3-must-gather-output\") pod \"must-gather-k7428\" (UID: \"77590cab-f436-4b7b-a9bf-23c29fc00fd3\") " pod="openshift-must-gather-5xxks/must-gather-k7428"
Sep 09 20:51:25.329453 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:25.329437    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/77590cab-f436-4b7b-a9bf-23c29fc00fd3-must-gather-output\") pod \"must-gather-k7428\" (UID: \"77590cab-f436-4b7b-a9bf-23c29fc00fd3\") " pod="openshift-must-gather-5xxks/must-gather-k7428"
Sep 09 20:51:25.337660 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:25.337635    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9m8k\" (UniqueName: \"kubernetes.io/projected/77590cab-f436-4b7b-a9bf-23c29fc00fd3-kube-api-access-s9m8k\") pod \"must-gather-k7428\" (UID: \"77590cab-f436-4b7b-a9bf-23c29fc00fd3\") " pod="openshift-must-gather-5xxks/must-gather-k7428"
Sep 09 20:51:25.459366 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:25.459316    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5xxks/must-gather-k7428"
Sep 09 20:51:25.729300 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:25.729228    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5xxks/must-gather-k7428"]
Sep 09 20:51:25.732851 ip-10-0-132-226 kubenswrapper[2523]: W0909 20:51:25.732821    2523 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod77590cab_f436_4b7b_a9bf_23c29fc00fd3.slice/crio-c57ef6f3762ceba119afbf7fb0735fa76574d4f2776e403315a0f2fdf694f382 WatchSource:0}: Error finding container c57ef6f3762ceba119afbf7fb0735fa76574d4f2776e403315a0f2fdf694f382: Status 404 returned error can't find the container with id c57ef6f3762ceba119afbf7fb0735fa76574d4f2776e403315a0f2fdf694f382
Sep 09 20:51:25.735037 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:25.735019    2523 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:51:25.780075 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:25.780050    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5xxks/must-gather-k7428" event={"ID":"77590cab-f436-4b7b-a9bf-23c29fc00fd3","Type":"ContainerStarted","Data":"c57ef6f3762ceba119afbf7fb0735fa76574d4f2776e403315a0f2fdf694f382"}
Sep 09 20:51:27.791415 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:27.791375    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5xxks/must-gather-k7428" event={"ID":"77590cab-f436-4b7b-a9bf-23c29fc00fd3","Type":"ContainerStarted","Data":"a68bde3cfe8780dfa6de69d0cf48dbe67360db3481a819040e221e2b4c9f298d"}
Sep 09 20:51:27.791415 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:27.791422    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5xxks/must-gather-k7428" event={"ID":"77590cab-f436-4b7b-a9bf-23c29fc00fd3","Type":"ContainerStarted","Data":"f5b8d775cccbd5ba012c85a896f624c2fe69a6ebad2ad95716142c1f607d20ef"}
Sep 09 20:51:27.810296 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:27.809936    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5xxks/must-gather-k7428" podStartSLOduration=1.4586439740000001 podStartE2EDuration="2.809913133s" totalImagesPullingTime="1.351269159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:51:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:51:27.807675386 +0000 UTC m=+3392.479694381" watchObservedRunningTime="2026-09-09 20:51:27.809913133 +0000 UTC m=+3392.481932133"
Sep 09 20:51:28.832641 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:28.832612    2523 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-gn8cc_b2cc1cab-a99d-4733-8623-2ccf3fb89961/global-pull-secret-syncer/0.log"
Sep 09 20:51:29.043548 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:29.043512    2523 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vkhvk_8978aa77-ea38-48bd-b86f-fcaf6a8cfcbd/konnectivity-agent/0.log"
Sep 09 20:51:29.108789 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:29.108714    2523 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-226.ec2.internal_f7c24ed75a93d3886eeb14da20c53cdf/haproxy/0.log"
Sep 09 20:51:33.210057 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:33.210015    2523 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-8zpvv_72f24ca2-96db-441c-a051-9fbf6f361baa/cluster-monitoring-operator/0.log"
Sep 09 20:51:33.537707 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:33.537635    2523 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lhjfm_3cce0c6c-933f-4559-89ae-6af9e6373cc9/node-exporter/0.log"
Sep 09 20:51:33.563899 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:33.563867    2523 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lhjfm_3cce0c6c-933f-4559-89ae-6af9e6373cc9/kube-rbac-proxy/0.log"
Sep 09 20:51:33.593695 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:33.593663    2523 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lhjfm_3cce0c6c-933f-4559-89ae-6af9e6373cc9/init-textfile/0.log"
Sep 09 20:51:35.600003 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:35.599971    2523 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-qx2rd_b6011a45-1105-4276-85a6-2145931807ba/networking-console-plugin/0.log"
Sep 09 20:51:36.656247 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:36.656210    2523 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5xxks/perf-node-gather-daemonset-pzzfl"]
Sep 09 20:51:36.662550 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:36.662522    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5xxks/perf-node-gather-daemonset-pzzfl"
Sep 09 20:51:36.672935 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:36.672902    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5xxks/perf-node-gather-daemonset-pzzfl"]
Sep 09 20:51:36.732617 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:36.732572    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/96aeae1e-b4f5-4136-afbe-3df255b6bb59-proc\") pod \"perf-node-gather-daemonset-pzzfl\" (UID: \"96aeae1e-b4f5-4136-afbe-3df255b6bb59\") " pod="openshift-must-gather-5xxks/perf-node-gather-daemonset-pzzfl"
Sep 09 20:51:36.732617 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:36.732618    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/96aeae1e-b4f5-4136-afbe-3df255b6bb59-podres\") pod \"perf-node-gather-daemonset-pzzfl\" (UID: \"96aeae1e-b4f5-4136-afbe-3df255b6bb59\") " pod="openshift-must-gather-5xxks/perf-node-gather-daemonset-pzzfl"
Sep 09 20:51:36.732858 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:36.732694    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jljzf\" (UniqueName: \"kubernetes.io/projected/96aeae1e-b4f5-4136-afbe-3df255b6bb59-kube-api-access-jljzf\") pod \"perf-node-gather-daemonset-pzzfl\" (UID: \"96aeae1e-b4f5-4136-afbe-3df255b6bb59\") " pod="openshift-must-gather-5xxks/perf-node-gather-daemonset-pzzfl"
Sep 09 20:51:36.732858 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:36.732748    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/96aeae1e-b4f5-4136-afbe-3df255b6bb59-sys\") pod \"perf-node-gather-daemonset-pzzfl\" (UID: \"96aeae1e-b4f5-4136-afbe-3df255b6bb59\") " pod="openshift-must-gather-5xxks/perf-node-gather-daemonset-pzzfl"
Sep 09 20:51:36.732858 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:36.732838    2523 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/96aeae1e-b4f5-4136-afbe-3df255b6bb59-lib-modules\") pod \"perf-node-gather-daemonset-pzzfl\" (UID: \"96aeae1e-b4f5-4136-afbe-3df255b6bb59\") " pod="openshift-must-gather-5xxks/perf-node-gather-daemonset-pzzfl"
Sep 09 20:51:36.834230 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:36.834190    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/96aeae1e-b4f5-4136-afbe-3df255b6bb59-proc\") pod \"perf-node-gather-daemonset-pzzfl\" (UID: \"96aeae1e-b4f5-4136-afbe-3df255b6bb59\") " pod="openshift-must-gather-5xxks/perf-node-gather-daemonset-pzzfl"
Sep 09 20:51:36.834230 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:36.834230    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/96aeae1e-b4f5-4136-afbe-3df255b6bb59-podres\") pod \"perf-node-gather-daemonset-pzzfl\" (UID: \"96aeae1e-b4f5-4136-afbe-3df255b6bb59\") " pod="openshift-must-gather-5xxks/perf-node-gather-daemonset-pzzfl"
Sep 09 20:51:36.834490 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:36.834259    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jljzf\" (UniqueName: \"kubernetes.io/projected/96aeae1e-b4f5-4136-afbe-3df255b6bb59-kube-api-access-jljzf\") pod \"perf-node-gather-daemonset-pzzfl\" (UID: \"96aeae1e-b4f5-4136-afbe-3df255b6bb59\") " pod="openshift-must-gather-5xxks/perf-node-gather-daemonset-pzzfl"
Sep 09 20:51:36.834490 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:36.834285    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/96aeae1e-b4f5-4136-afbe-3df255b6bb59-sys\") pod \"perf-node-gather-daemonset-pzzfl\" (UID: \"96aeae1e-b4f5-4136-afbe-3df255b6bb59\") " pod="openshift-must-gather-5xxks/perf-node-gather-daemonset-pzzfl"
Sep 09 20:51:36.834490 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:36.834334    2523 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/96aeae1e-b4f5-4136-afbe-3df255b6bb59-lib-modules\") pod \"perf-node-gather-daemonset-pzzfl\" (UID: \"96aeae1e-b4f5-4136-afbe-3df255b6bb59\") " pod="openshift-must-gather-5xxks/perf-node-gather-daemonset-pzzfl"
Sep 09 20:51:36.834490 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:36.834376    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/96aeae1e-b4f5-4136-afbe-3df255b6bb59-sys\") pod \"perf-node-gather-daemonset-pzzfl\" (UID: \"96aeae1e-b4f5-4136-afbe-3df255b6bb59\") " pod="openshift-must-gather-5xxks/perf-node-gather-daemonset-pzzfl"
Sep 09 20:51:36.834490 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:36.834334    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/96aeae1e-b4f5-4136-afbe-3df255b6bb59-proc\") pod \"perf-node-gather-daemonset-pzzfl\" (UID: \"96aeae1e-b4f5-4136-afbe-3df255b6bb59\") " pod="openshift-must-gather-5xxks/perf-node-gather-daemonset-pzzfl"
Sep 09 20:51:36.834490 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:36.834407    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/96aeae1e-b4f5-4136-afbe-3df255b6bb59-podres\") pod \"perf-node-gather-daemonset-pzzfl\" (UID: \"96aeae1e-b4f5-4136-afbe-3df255b6bb59\") " pod="openshift-must-gather-5xxks/perf-node-gather-daemonset-pzzfl"
Sep 09 20:51:36.834818 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:36.834485    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/96aeae1e-b4f5-4136-afbe-3df255b6bb59-lib-modules\") pod \"perf-node-gather-daemonset-pzzfl\" (UID: \"96aeae1e-b4f5-4136-afbe-3df255b6bb59\") " pod="openshift-must-gather-5xxks/perf-node-gather-daemonset-pzzfl"
Sep 09 20:51:36.844131 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:36.844107    2523 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jljzf\" (UniqueName: \"kubernetes.io/projected/96aeae1e-b4f5-4136-afbe-3df255b6bb59-kube-api-access-jljzf\") pod \"perf-node-gather-daemonset-pzzfl\" (UID: \"96aeae1e-b4f5-4136-afbe-3df255b6bb59\") " pod="openshift-must-gather-5xxks/perf-node-gather-daemonset-pzzfl"
Sep 09 20:51:36.976818 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:36.976543    2523 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5xxks/perf-node-gather-daemonset-pzzfl"
Sep 09 20:51:37.262187 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:37.258517    2523 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5xxks/perf-node-gather-daemonset-pzzfl"]
Sep 09 20:51:37.867115 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:37.867077    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5xxks/perf-node-gather-daemonset-pzzfl" event={"ID":"96aeae1e-b4f5-4136-afbe-3df255b6bb59","Type":"ContainerStarted","Data":"303adc1ef45b9413afde3603f2efb6c745d9a6d7c54866a288187520c5a4105b"}
Sep 09 20:51:37.867557 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:37.867142    2523 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5xxks/perf-node-gather-daemonset-pzzfl" event={"ID":"96aeae1e-b4f5-4136-afbe-3df255b6bb59","Type":"ContainerStarted","Data":"263cc9cd1ca36753f34aa44cfce52f3b4d066bb7e2f653979cc407ec75cb1542"}
Sep 09 20:51:37.867557 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:37.867192    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5xxks/perf-node-gather-daemonset-pzzfl"
Sep 09 20:51:37.885755 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:37.885705    2523 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5xxks/perf-node-gather-daemonset-pzzfl" podStartSLOduration=1.885689876 podStartE2EDuration="1.885689876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:51:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:51:37.884707586 +0000 UTC m=+3402.556726594" watchObservedRunningTime="2026-09-09 20:51:37.885689876 +0000 UTC m=+3402.557708873"
Sep 09 20:51:38.170098 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:38.170028    2523 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zr55l_1d3bbfe8-34df-41e1-b959-766ff5100634/dns/0.log"
Sep 09 20:51:38.194353 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:38.194326    2523 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zr55l_1d3bbfe8-34df-41e1-b959-766ff5100634/kube-rbac-proxy/0.log"
Sep 09 20:51:38.328528 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:38.328503    2523 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-w2tb2_8fe231fb-4f9c-478e-b2c5-95a65a627385/dns-node-resolver/0.log"
Sep 09 20:51:38.848526 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:38.848501    2523 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ctlrg_8d8ec74b-cbcb-4c36-a904-21e6a6460f6b/node-ca/0.log"
Sep 09 20:51:40.157254 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:40.157226    2523 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rq89t_8029d5a2-cb5c-4ab4-93cf-d9da4f3bcaea/serve-healthcheck-canary/0.log"
Sep 09 20:51:40.514871 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:40.514805    2523 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-fn5ll_9e6f3e8a-9da3-48f8-a081-dfcfa2917b40/insights-operator/0.log"
Sep 09 20:51:40.515185 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:40.515169    2523 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-fn5ll_9e6f3e8a-9da3-48f8-a081-dfcfa2917b40/insights-operator/1.log"
Sep 09 20:51:40.689471 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:40.689430    2523 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-z7hmm_27f5043a-c597-4d58-9a73-c0b1cd9071b3/kube-rbac-proxy/0.log"
Sep 09 20:51:40.712407 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:40.712386    2523 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-z7hmm_27f5043a-c597-4d58-9a73-c0b1cd9071b3/exporter/0.log"
Sep 09 20:51:40.740163 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:40.740124    2523 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-z7hmm_27f5043a-c597-4d58-9a73-c0b1cd9071b3/extractor/0.log"
Sep 09 20:51:43.375308 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:43.375282    2523 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-645f9c8fd6-ddscs_fa3e7d87-045f-480a-9f42-28ffb444623e/manager/0.log"
Sep 09 20:51:43.881275 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:43.881246    2523 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5xxks/perf-node-gather-daemonset-pzzfl"
Sep 09 20:51:44.139324 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:44.139258    2523 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-serving-67b7477c6d-r95ll_e9013ad1-a536-407f-916a-41300babbaf6/seaweedfs-tls-serving/0.log"
Sep 09 20:51:50.152522 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:50.152471    2523 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4jptv_556a154a-15d2-4313-93b6-42580b64443c/kube-multus-additional-cni-plugins/0.log"
Sep 09 20:51:50.176057 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:50.176026    2523 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4jptv_556a154a-15d2-4313-93b6-42580b64443c/egress-router-binary-copy/0.log"
Sep 09 20:51:50.199780 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:50.199761    2523 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4jptv_556a154a-15d2-4313-93b6-42580b64443c/cni-plugins/0.log"
Sep 09 20:51:50.221967 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:50.221948    2523 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4jptv_556a154a-15d2-4313-93b6-42580b64443c/bond-cni-plugin/0.log"
Sep 09 20:51:50.247237 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:50.247208    2523 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4jptv_556a154a-15d2-4313-93b6-42580b64443c/routeoverride-cni/0.log"
Sep 09 20:51:50.271144 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:50.271123    2523 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4jptv_556a154a-15d2-4313-93b6-42580b64443c/whereabouts-cni-bincopy/0.log"
Sep 09 20:51:50.302469 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:50.302452    2523 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4jptv_556a154a-15d2-4313-93b6-42580b64443c/whereabouts-cni/0.log"
Sep 09 20:51:50.887419 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:50.887395    2523 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-xnmz6_5ac07c25-2d29-4115-851c-1c3602361e6a/kube-multus/0.log"
Sep 09 20:51:50.970204 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:50.970179    2523 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-tmvpv_21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8/network-metrics-daemon/0.log"
Sep 09 20:51:50.993766 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:50.993745    2523 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-tmvpv_21f9ae32-c8d6-42ea-94cf-4cfe645ee5c8/kube-rbac-proxy/0.log"
Sep 09 20:51:51.825086 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:51.825043    2523 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c72g2_ad886add-2c65-470b-8935-efe692cafcec/ovn-controller/0.log"
Sep 09 20:51:51.845610 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:51.845590    2523 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c72g2_ad886add-2c65-470b-8935-efe692cafcec/ovn-acl-logging/0.log"
Sep 09 20:51:51.872701 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:51.872677    2523 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c72g2_ad886add-2c65-470b-8935-efe692cafcec/kube-rbac-proxy-node/0.log"
Sep 09 20:51:51.903724 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:51.903698    2523 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c72g2_ad886add-2c65-470b-8935-efe692cafcec/kube-rbac-proxy-ovn-metrics/0.log"
Sep 09 20:51:51.926744 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:51.926718    2523 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c72g2_ad886add-2c65-470b-8935-efe692cafcec/northd/0.log"
Sep 09 20:51:51.952321 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:51.952304    2523 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c72g2_ad886add-2c65-470b-8935-efe692cafcec/nbdb/0.log"
Sep 09 20:51:51.979513 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:51.979492    2523 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c72g2_ad886add-2c65-470b-8935-efe692cafcec/sbdb/0.log"
Sep 09 20:51:52.028429 ip-10-0-132-226 kubenswrapper[2523]: I0909 20:51:52.028397    2523 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c72g2_ad886add-2c65-470b-8935-efe692cafcec/ovnkube-controller/0.log"