Sep 16 18:40:23.197154 ip-10-0-129-190 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 18:40:23.627386 ip-10-0-129-190 kubenswrapper[2586]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:40:23.627386 ip-10-0-129-190 kubenswrapper[2586]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 18:40:23.627386 ip-10-0-129-190 kubenswrapper[2586]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:40:23.627386 ip-10-0-129-190 kubenswrapper[2586]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 18:40:23.627386 ip-10-0-129-190 kubenswrapper[2586]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:40:23.630295 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.630206    2586 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 18:40:23.634045 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634025    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:23.634045 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634042    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:23.634045 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634047    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:23.634045 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634052    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:23.634286 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634056    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:23.634286 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634060    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:23.634286 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634064    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:23.634286 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634067    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:23.634286 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634071    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:23.634286 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634076    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:23.634286 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634080    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:23.634286 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634084    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:23.634286 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634088    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:23.634286 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634092    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:23.634286 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634096    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:23.634286 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634099    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:23.634286 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634103    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:23.634286 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634120    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:23.634286 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634126    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:23.634286 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634130    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:23.634286 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634134    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:23.634286 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634139    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:23.634286 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634143    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:23.634286 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634147    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:23.635123 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634150    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:23.635123 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634154    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:23.635123 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634159    2586 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:23.635123 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634163    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:23.635123 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634167    2586 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:23.635123 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634172    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:23.635123 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634176    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:23.635123 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634181    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:23.635123 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634185    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:23.635123 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634189    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:23.635123 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634193    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:23.635123 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634197    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:23.635123 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634202    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:23.635123 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634207    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:23.635123 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634212    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:23.635123 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634216    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:23.635123 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634221    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:23.635123 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634225    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:23.635123 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634231    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:23.635123 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634236    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:23.635985 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634241    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:23.635985 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634245    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:23.635985 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634249    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:23.635985 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634253    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:23.635985 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634257    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:23.635985 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634261    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:23.635985 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634266    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:23.635985 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634271    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:23.635985 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634275    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:23.635985 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634279    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:23.635985 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634283    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:23.635985 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634287    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:23.635985 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634291    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:23.635985 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634295    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:23.635985 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634300    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:23.635985 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634304    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:23.635985 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634308    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:23.635985 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634313    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:23.635985 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634317    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:23.635985 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634321    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:23.636844 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634325    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:23.636844 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634330    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:23.636844 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634334    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:23.636844 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634339    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:23.636844 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634343    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:23.636844 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634347    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:23.636844 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634352    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:23.636844 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634356    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:23.636844 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634360    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:23.636844 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634365    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:23.636844 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634369    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:23.636844 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634374    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:23.636844 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634378    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:23.636844 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634383    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:23.636844 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634387    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:23.636844 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634391    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:23.636844 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634397    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:23.636844 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634402    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:23.636844 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634406    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:23.636844 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634411    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:23.637689 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634418    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:23.637689 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634423    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:23.637689 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634429    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:23.637689 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634436    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:23.637689 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634441    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:23.637689 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634445    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:23.637689 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634449    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:23.637689 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634453    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:23.637689 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634458    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:23.637689 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634462    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:23.637689 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634466    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:23.637689 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634470    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:23.637689 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634474    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:23.637689 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634478    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:23.637689 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.634482    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:23.637689 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635659    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:23.637689 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635671    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:23.637689 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635678    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:23.638469 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635684    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:23.638469 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635688    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:23.638469 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635692    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:23.638469 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635697    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:23.638469 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635701    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:23.638469 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635705    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:23.638469 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635709    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:23.638469 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635715    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:23.638469 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635719    2586 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:23.638469 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635723    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:23.638469 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635728    2586 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:23.638469 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635733    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:23.638469 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635737    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:23.638469 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635741    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:23.638469 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635745    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:23.638469 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635749    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:23.638469 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635753    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:23.638469 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635757    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:23.638469 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635761    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:23.638469 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635766    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:23.639164 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635770    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:23.639164 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635774    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:23.639164 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635778    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:23.639164 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635782    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:23.639164 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635787    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:23.639164 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635791    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:23.639164 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635795    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:23.639164 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635799    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:23.639164 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635802    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:23.639164 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635807    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:23.639164 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635811    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:23.639164 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635835    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:23.639164 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635840    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:23.639164 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635844    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:23.639164 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635848    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:23.639164 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635853    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:23.639164 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635857    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:23.639164 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635861    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:23.639164 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635866    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:23.639164 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635870    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:23.639729 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635875    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:23.639729 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635879    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:23.639729 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635883    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:23.639729 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635889    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:23.639729 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635895    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:23.639729 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635902    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:23.639729 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635906    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:23.639729 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635911    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:23.639729 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635915    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:23.639729 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635920    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:23.639729 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635925    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:23.639729 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635930    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:23.639729 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635934    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:23.639729 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635938    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:23.639729 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635942    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:23.639729 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635946    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:23.639729 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635950    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:23.639729 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635955    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:23.639729 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635959    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:23.640285 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635965    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:23.640285 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635969    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:23.640285 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635973    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:23.640285 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635977    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:23.640285 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635984    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:23.640285 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635988    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:23.640285 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635992    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:23.640285 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.635996    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:23.640285 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636000    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:23.640285 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636004    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:23.640285 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636008    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:23.640285 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636012    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:23.640285 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636016    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:23.640285 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636020    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:23.640285 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636025    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:23.640285 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636030    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:23.640285 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636036    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:23.640285 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636043    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:23.640285 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636047    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:23.640285 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636051    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:23.640907 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636055    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:23.640907 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636059    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:23.640907 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636064    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:23.640907 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636068    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:23.640907 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636072    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:23.640907 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636076    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:23.640907 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636081    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:23.640907 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636085    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:23.640907 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636089    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:23.640907 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636093    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:23.640907 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636097    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:23.640907 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636101    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:23.640907 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636105    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:23.640907 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636109    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:23.640907 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636114    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:23.640907 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636118    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:23.640907 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636124    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:23.640907 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636730    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:23.640907 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636739    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:23.641393 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636743    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:23.641393 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636747    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:23.641393 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636752    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:23.641393 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636756    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:23.641393 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636761    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:23.641393 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636766    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:23.641393 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636771    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:23.641393 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636776    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:23.641393 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636781    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:23.641393 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636785    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:23.641393 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636789    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:23.641393 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636793    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:23.641393 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636798    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:23.641393 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636802    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:23.641393 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636806    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:23.641393 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636810    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:23.641393 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636832    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:23.641393 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636837    2586 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:23.641393 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636841    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:23.641393 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636845    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:23.641953 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636849    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:23.641953 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636853    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:23.641953 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636857    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:23.641953 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636861    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:23.641953 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636865    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:23.641953 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636869    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:23.641953 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636873    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:23.641953 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636915    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:23.641953 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636922    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:23.641953 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636932    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:23.641953 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636936    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:23.641953 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636941    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:23.641953 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636945    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:23.641953 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636951    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:23.641953 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636955    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:23.641953 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636959    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:23.641953 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636963    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:23.641953 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636968    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:23.641953 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636972    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:23.641953 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636976    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:23.642497 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636982    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:23.642497 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636986    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:23.642497 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636990    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:23.642497 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636994    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:23.642497 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.636999    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:23.642497 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.637002    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:23.642497 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.637006    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:23.642497 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.637011    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:23.642497 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.637015    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:23.642497 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.637019    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:23.642497 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.637023    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:23.642497 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.637027    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:23.642497 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.637031    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:23.642497 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.637035    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:23.642497 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.637039    2586 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:23.642497 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.637043    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:23.642497 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.637047    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:23.642497 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.637052    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:23.642497 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.637058    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:23.643129 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.637064    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:23.643129 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.637069    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:23.643129 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.637075    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:23.643129 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.637080    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:23.643129 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.637084    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:23.643129 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.637088    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:23.643129 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.637092    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:23.643129 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.637096    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:23.643129 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.637100    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:23.643129 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.637105    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:23.643129 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.637109    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:23.643129 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.637114    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:23.643129 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.637118    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:23.643129 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.637122    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:23.643129 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.637127    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:23.643129 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.637132    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:23.643129 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.637136    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:23.643129 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.637140    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:23.643129 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.637144    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:23.643129 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.637148    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:23.643606 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.637152    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:23.643606 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.637156    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:23.643606 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.637160    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:23.643606 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.637166    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:23.643606 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.637172    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:23.643606 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.637177    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:23.643606 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.637182    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:23.643606 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.637186    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:23.643606 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.637191    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:23.643606 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.637195    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:23.643606 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.637199    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:23.643606 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.637203    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:23.643606 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.637207    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:23.643606 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.637211    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:23.643606 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.637219    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:23.643606 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.637223    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:23.643606 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.637228    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:23.643606 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.637232    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:23.643606 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637329    2586 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 18:40:23.643606 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637375    2586 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 18:40:23.644200 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637384    2586 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 18:40:23.644200 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637391    2586 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 18:40:23.644200 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637397    2586 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 18:40:23.644200 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637402    2586 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 18:40:23.644200 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637409    2586 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 18:40:23.644200 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637415    2586 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 18:40:23.644200 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637420    2586 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 18:40:23.644200 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637428    2586 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 18:40:23.644200 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637434    2586 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 18:40:23.644200 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637439    2586 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 18:40:23.644200 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637444    2586 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 18:40:23.644200 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637449    2586 flags.go:64] FLAG: --cgroup-root=""
Sep 16 18:40:23.644200 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637454    2586 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 18:40:23.644200 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637459    2586 flags.go:64] FLAG: --client-ca-file=""
Sep 16 18:40:23.644200 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637464    2586 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 18:40:23.644200 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637468    2586 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 18:40:23.644200 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637475    2586 flags.go:64] FLAG: --cluster-domain=""
Sep 16 18:40:23.644200 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637480    2586 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 18:40:23.644200 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637484    2586 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 18:40:23.644200 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637490    2586 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 18:40:23.644200 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637495    2586 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 18:40:23.644200 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637501    2586 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 18:40:23.644200 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637506    2586 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 18:40:23.644776 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637512    2586 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 18:40:23.644776 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637517    2586 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 18:40:23.644776 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637521    2586 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 18:40:23.644776 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637530    2586 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 18:40:23.644776 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637536    2586 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 18:40:23.644776 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637541    2586 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 18:40:23.644776 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637546    2586 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 18:40:23.644776 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637552    2586 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 18:40:23.644776 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637557    2586 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 18:40:23.644776 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637561    2586 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 18:40:23.644776 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637566    2586 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 18:40:23.644776 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637571    2586 flags.go:64] FLAG: --enable-server="true"
Sep 16 18:40:23.644776 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637576    2586 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 18:40:23.644776 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637582    2586 flags.go:64] FLAG: --event-burst="100"
Sep 16 18:40:23.644776 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637587    2586 flags.go:64] FLAG: --event-qps="50"
Sep 16 18:40:23.644776 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637592    2586 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 18:40:23.644776 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637597    2586 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 18:40:23.644776 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637602    2586 flags.go:64] FLAG: --eviction-hard=""
Sep 16 18:40:23.644776 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637609    2586 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 18:40:23.644776 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637613    2586 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 18:40:23.644776 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637619    2586 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 18:40:23.644776 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637623    2586 flags.go:64] FLAG: --eviction-soft=""
Sep 16 18:40:23.644776 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637628    2586 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 18:40:23.644776 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637633    2586 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 18:40:23.644776 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637637    2586 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 18:40:23.645411 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637642    2586 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 18:40:23.645411 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637646    2586 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 18:40:23.645411 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637651    2586 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 18:40:23.645411 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637655    2586 flags.go:64] FLAG: --feature-gates=""
Sep 16 18:40:23.645411 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637662    2586 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 18:40:23.645411 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637667    2586 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 18:40:23.645411 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637671    2586 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 18:40:23.645411 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637676    2586 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 18:40:23.645411 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637681    2586 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 18:40:23.645411 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637685    2586 flags.go:64] FLAG: --help="false"
Sep 16 18:40:23.645411 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637694    2586 flags.go:64] FLAG: --hostname-override="ip-10-0-129-190.ec2.internal"
Sep 16 18:40:23.645411 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637699    2586 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 18:40:23.645411 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637704    2586 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 18:40:23.645411 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637708    2586 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 18:40:23.645411 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637714    2586 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 18:40:23.645411 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637720    2586 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 18:40:23.645411 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637724    2586 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 18:40:23.645411 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637729    2586 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 18:40:23.645411 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637733    2586 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 18:40:23.645411 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637738    2586 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 18:40:23.645411 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637743    2586 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 18:40:23.645411 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637748    2586 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 18:40:23.645411 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637752    2586 flags.go:64] FLAG: --kube-reserved=""
Sep 16 18:40:23.645411 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637757    2586 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 18:40:23.646014 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637762    2586 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 18:40:23.646014 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637768    2586 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 18:40:23.646014 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637773    2586 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 18:40:23.646014 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637778    2586 flags.go:64] FLAG: --lock-file=""
Sep 16 18:40:23.646014 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637782    2586 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 18:40:23.646014 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637787    2586 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 18:40:23.646014 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637792    2586 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 18:40:23.646014 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637801    2586 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 18:40:23.646014 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637805    2586 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 18:40:23.646014 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637810    2586 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 18:40:23.646014 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637832    2586 flags.go:64] FLAG: --logging-format="text"
Sep 16 18:40:23.646014 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637837    2586 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 18:40:23.646014 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637843    2586 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 18:40:23.646014 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637848    2586 flags.go:64] FLAG: --manifest-url=""
Sep 16 18:40:23.646014 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637852    2586 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 18:40:23.646014 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637859    2586 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 18:40:23.646014 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637863    2586 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 18:40:23.646014 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637870    2586 flags.go:64] FLAG: --max-pods="110"
Sep 16 18:40:23.646014 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637877    2586 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 18:40:23.646014 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637882    2586 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 18:40:23.646014 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637887    2586 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 18:40:23.646014 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637892    2586 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 18:40:23.646014 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637896    2586 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 18:40:23.646014 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637901    2586 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 18:40:23.646014 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637906    2586 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 18:40:23.646628 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637916    2586 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 18:40:23.646628 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637921    2586 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 18:40:23.646628 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637926    2586 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 18:40:23.646628 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637932    2586 flags.go:64] FLAG: --pod-cidr=""
Sep 16 18:40:23.646628 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637936    2586 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 18:40:23.646628 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637944    2586 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 18:40:23.646628 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637949    2586 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 18:40:23.646628 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637954    2586 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 18:40:23.646628 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637959    2586 flags.go:64] FLAG: --port="10250"
Sep 16 18:40:23.646628 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637965    2586 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 18:40:23.646628 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637969    2586 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0bc0787047daab26e"
Sep 16 18:40:23.646628 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637975    2586 flags.go:64] FLAG: --qos-reserved=""
Sep 16 18:40:23.646628 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637979    2586 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 18:40:23.646628 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637984    2586 flags.go:64] FLAG: --register-node="true"
Sep 16 18:40:23.646628 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637989    2586 flags.go:64] FLAG: --register-with-taints=""
Sep 16 18:40:23.646628 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.637995    2586 flags.go:64] FLAG: --registry-burst="10"
Sep 16 18:40:23.646628 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.638000    2586 flags.go:64] FLAG: --registry-qps="5"
Sep 16 18:40:23.646628 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.638005    2586 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 18:40:23.646628 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.638009    2586 flags.go:64] FLAG: --reserved-memory=""
Sep 16 18:40:23.646628 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.638015    2586 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 18:40:23.646628 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.638020    2586 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 18:40:23.646628 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.638025    2586 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 18:40:23.646628 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.638030    2586 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 18:40:23.646628 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.638034    2586 flags.go:64] FLAG: --runonce="false"
Sep 16 18:40:23.646628 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.638039    2586 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 18:40:23.647324 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.638044    2586 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 18:40:23.647324 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.638051    2586 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 18:40:23.647324 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.638056    2586 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 18:40:23.647324 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.638061    2586 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 18:40:23.647324 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.638066    2586 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 18:40:23.647324 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.638071    2586 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 18:40:23.647324 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.638076    2586 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 18:40:23.647324 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.638080    2586 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 18:40:23.647324 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.638085    2586 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 18:40:23.647324 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.638091    2586 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 18:40:23.647324 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.638096    2586 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 18:40:23.647324 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.638100    2586 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 18:40:23.647324 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.638105    2586 flags.go:64] FLAG: --system-cgroups=""
Sep 16 18:40:23.647324 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.638110    2586 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 18:40:23.647324 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.638118    2586 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 18:40:23.647324 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.638123    2586 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 18:40:23.647324 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.638128    2586 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 18:40:23.647324 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.638135    2586 flags.go:64] FLAG: --tls-min-version=""
Sep 16 18:40:23.647324 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.638140    2586 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 18:40:23.647324 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.638144    2586 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 18:40:23.647324 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.638149    2586 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 18:40:23.647324 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.638153    2586 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 18:40:23.647324 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.638158    2586 flags.go:64] FLAG: --v="2"
Sep 16 18:40:23.647324 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.638165    2586 flags.go:64] FLAG: --version="false"
Sep 16 18:40:23.647324 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.638171    2586 flags.go:64] FLAG: --vmodule=""
Sep 16 18:40:23.647955 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.638177    2586 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 18:40:23.647955 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.638182    2586 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 18:40:23.647955 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638383    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:23.647955 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638390    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:23.647955 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638395    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:23.647955 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638400    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:23.647955 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638405    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:23.647955 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638410    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:23.647955 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638416    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:23.647955 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638420    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:23.647955 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638425    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:23.647955 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638429    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:23.647955 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638433    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:23.647955 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638437    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:23.647955 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638442    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:23.647955 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638448    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:23.647955 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638454    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:23.647955 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638459    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:23.647955 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638464    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:23.647955 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638468    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:23.648706 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638473    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:23.648706 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638478    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:23.648706 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638482    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:23.648706 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638486    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:23.648706 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638490    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:23.648706 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638496    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:23.648706 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638501    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:23.648706 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638505    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:23.648706 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638512    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:23.648706 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638517    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:23.648706 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638522    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:23.648706 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638527    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:23.648706 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638531    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:23.648706 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638535    2586 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:23.648706 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638539    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:23.648706 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638543    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:23.648706 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638548    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:23.648706 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638552    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:23.648706 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638556    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:23.648706 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638560    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:23.649583 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638567    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:23.649583 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638571    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:23.649583 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638575    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:23.649583 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638580    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:23.649583 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638584    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:23.649583 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638588    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:23.649583 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638593    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:23.649583 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638597    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:23.649583 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638601    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:23.649583 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638605    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:23.649583 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638609    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:23.649583 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638613    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:23.649583 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638619    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:23.649583 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638623    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:23.649583 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638628    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:23.649583 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638632    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:23.649583 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638637    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:23.649583 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638641    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:23.649583 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638646    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:23.649583 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638650    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:23.650500 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638654    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:23.650500 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638658    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:23.650500 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638664    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:23.650500 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638668    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:23.650500 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638673    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:23.650500 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638677    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:23.650500 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638681    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:23.650500 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638685    2586 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:23.650500 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638689    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:23.650500 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638694    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:23.650500 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638698    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:23.650500 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638701    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:23.650500 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638709    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:23.650500 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638714    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:23.650500 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638718    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:23.650500 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638722    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:23.650500 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638726    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:23.650500 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638730    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:23.650500 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638735    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:23.650500 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638739    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:23.651430 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638743    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:23.651430 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638747    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:23.651430 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638751    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:23.651430 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638755    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:23.651430 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638759    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:23.651430 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638763    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:23.651430 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638767    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:23.651430 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638772    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:23.651430 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638776    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:23.651430 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638780    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:23.651430 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638784    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:23.651430 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638788    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:23.651430 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638793    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:23.651430 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638797    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:23.651430 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638803    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:23.651430 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638807    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:23.651430 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638812    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:23.651430 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638832    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:23.651430 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638837    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:23.652120 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638841    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:23.652120 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.638845    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:23.652120 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.639468    2586 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:40:23.652120 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.651830    2586 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 18:40:23.652120 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.651848    2586 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 18:40:23.652656 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.652639    2586 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 18:40:23.652705 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.652664    2586 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 18:40:23.652856 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.652845    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:23.652856 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.652857    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:23.652962 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.652863    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:23.652962 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.652874    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:23.652962 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.652879    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:23.652962 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.652884    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:23.652962 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.652892    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:23.652962 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.652898    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:23.652962 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.652903    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:23.652962 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.652909    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:23.652962 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.652914    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:23.652962 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.652918    2586 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:23.652962 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.652923    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:23.652962 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.652928    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:23.652962 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.652933    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:23.652962 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.652951    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:23.652962 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.652956    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:23.652962 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.652964    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:23.652962 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.652970    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:23.653702 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.652975    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:23.653702 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.652980    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:23.653702 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.652985    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:23.653702 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.652990    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:23.653702 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.652995    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:23.653702 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653000    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:23.653702 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653004    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:23.653702 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653009    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:23.653702 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653018    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:23.653702 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653022    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:23.653702 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653027    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:23.653702 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653031    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:23.653702 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653036    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:23.653702 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653040    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:23.653702 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653045    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:23.653702 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653050    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:23.653702 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653054    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:23.653702 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653058    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:23.653702 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653062    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:23.653702 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653066    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:23.654659 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653071    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:23.654659 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653080    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:23.654659 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653085    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:23.654659 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653089    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:23.654659 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653094    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:23.654659 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653098    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:23.654659 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653102    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:23.654659 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653106    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:23.654659 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653112    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:23.654659 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653127    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:23.654659 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653132    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:23.654659 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653137    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:23.654659 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653141    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:23.654659 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653151    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:23.654659 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653156    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:23.654659 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653160    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:23.654659 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653165    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:23.654659 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653169    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:23.654659 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653174    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:23.655246 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653178    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:23.655246 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653182    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:23.655246 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653187    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:23.655246 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653191    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:23.655246 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653196    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:23.655246 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653200    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:23.655246 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653205    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:23.655246 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653214    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:23.655246 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653218    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:23.655246 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653223    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:23.655246 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653227    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:23.655246 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653231    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:23.655246 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653235    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:23.655246 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653240    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:23.655246 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653244    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:23.655246 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653249    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:23.655246 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653253    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:23.655246 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653258    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:23.655246 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653262    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:23.655980 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653267    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:23.655980 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653275    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:23.655980 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653280    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:23.655980 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653284    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:23.655980 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653297    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:23.655980 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653302    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:23.655980 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653307    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:23.655980 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653312    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:23.655980 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653316    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:23.655980 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653321    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:23.655980 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653325    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:23.655980 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653330    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:23.655980 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653335    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:23.655980 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653343    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:23.655980 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653348    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:23.655980 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653352    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:23.655980 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653357    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:23.655980 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653361    2586 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:23.655980 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653365    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:23.656463 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653370    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:23.656463 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653374    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:23.656463 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653379    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:23.656463 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.653387    2586 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:40:23.656463 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653858    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:23.656463 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653867    2586 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:23.656463 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653872    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:23.656463 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653877    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:23.656463 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653881    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:23.656463 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653890    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:23.656463 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653894    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:23.656463 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653900    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:23.656463 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653908    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:23.656463 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653912    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:23.656928 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653917    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:23.656928 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653922    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:23.656928 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653927    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:23.656928 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653932    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:23.656928 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653944    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:23.656928 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653949    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:23.656928 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653954    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:23.656928 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653964    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:23.656928 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653968    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:23.656928 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653973    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:23.656928 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653978    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:23.656928 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653982    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:23.656928 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653986    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:23.656928 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653990    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:23.656928 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653995    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:23.656928 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.653999    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:23.656928 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654003    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:23.656928 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654008    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:23.656928 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654012    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:23.656928 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654017    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:23.657466 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654026    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:23.657466 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654030    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:23.657466 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654034    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:23.657466 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654038    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:23.657466 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654043    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:23.657466 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654047    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:23.657466 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654052    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:23.657466 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654056    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:23.657466 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654060    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:23.657466 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654065    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:23.657466 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654069    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:23.657466 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654073    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:23.657466 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654078    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:23.657466 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654087    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:23.657466 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654091    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:23.657466 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654095    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:23.657466 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654100    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:23.657466 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654112    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:23.657466 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654117    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:23.657466 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654122    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:23.658445 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654127    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:23.658445 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654131    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:23.658445 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654136    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:23.658445 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654140    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:23.658445 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654145    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:23.658445 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654149    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:23.658445 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654158    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:23.658445 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654162    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:23.658445 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654167    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:23.658445 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654172    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:23.658445 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654176    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:23.658445 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654182    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:23.658445 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654188    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:23.658445 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654192    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:23.658445 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654196    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:23.658445 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654200    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:23.658445 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654205    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:23.658445 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654209    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:23.658445 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654218    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:23.659088 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654222    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:23.659088 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654227    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:23.659088 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654232    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:23.659088 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654236    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:23.659088 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654241    2586 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:23.659088 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654245    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:23.659088 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654250    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:23.659088 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654254    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:23.659088 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654258    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:23.659088 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654263    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:23.659088 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654267    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:23.659088 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654284    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:23.659088 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654290    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:23.659088 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654294    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:23.659088 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654299    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:23.659088 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654303    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:23.659088 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654308    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:23.659088 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654312    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:23.659088 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654317    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:23.659088 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654321    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:23.659717 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654325    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:23.659717 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654330    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:23.659717 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654334    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:23.659717 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654346    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:23.659717 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654352    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:23.659717 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654357    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:23.659717 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654362    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:23.659717 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654367    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:23.659717 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654371    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:23.659717 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:23.654376    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:23.659717 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.654385    2586 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:40:23.659717 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.654641    2586 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 18:40:23.659717 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.659355    2586 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 18:40:23.660681 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.660665    2586 server.go:1019] "Starting client certificate rotation"
Sep 16 18:40:23.660789 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.660774    2586 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:40:23.660859 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.660835    2586 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:40:23.664609 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.664593    2586 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:40:23.666594 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.666579    2586 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:40:23.674219 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.674202    2586 log.go:25] "Validated CRI v1 runtime API"
Sep 16 18:40:23.677523 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.677509    2586 log.go:25] "Validated CRI v1 image API"
Sep 16 18:40:23.681508 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.681494    2586 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 18:40:23.685686 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.685667    2586 fs.go:135] Filesystem UUIDs: map[2d2b0a46-6873-4c43-8748-53c952ed09ad:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d58371f6-e897-433c-9b1e-665a49ab15e8:/dev/nvme0n1p4]
Sep 16 18:40:23.685748 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.685684    2586 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 18:40:23.692139 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.692024    2586 manager.go:216] Machine: {Timestamp:2026-09-16 18:40:23.690691785 +0000 UTC m=+0.369964580 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098601 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e12b23764b507ebd29c22563991a8 SystemUUID:ec2e12b2-3764-b507-ebd2-9c22563991a8 BootID:97598fc0-e39a-4785-83fd-4a52d5f74bb1 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:cf:1b:58:d1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:cf:1b:58:d1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7a:c2:d9:19:ba:34 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 18:40:23.692139 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.692135    2586 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 18:40:23.692259 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.692219    2586 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 18:40:23.695618 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.695595    2586 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 18:40:23.695755 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.695619    2586 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-190.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 18:40:23.696305 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.696295    2586 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 18:40:23.696344 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.696308    2586 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 18:40:23.696344 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.696316    2586 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:40:23.696918 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.696908    2586 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:40:23.697736 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.697727    2586 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 18:40:23.700679 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.700664    2586 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:40:23.700809 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.700799    2586 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 18:40:23.700863 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.700841    2586 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 18:40:23.700863 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.700858    2586 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 18:40:23.700915 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.700874    2586 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 18:40:23.700915 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.700883    2586 kubelet.go:398] "Adding apiserver pod source"
Sep 16 18:40:23.700915 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.700894    2586 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 18:40:23.701392 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.701379    2586 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:40:23.701442 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.701399    2586 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:40:23.701442 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.701391    2586 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:40:23.703539 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.703520    2586 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 18:40:23.707396 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.707375    2586 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 18:40:23.707473 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.707451    2586 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 18:40:23.709230 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.709212    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 18:40:23.709342 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.709239    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 18:40:23.709342 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.709250    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 18:40:23.709342 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.709262    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 18:40:23.709342 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.709277    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 18:40:23.709342 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.709286    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 18:40:23.709342 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.709308    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 18:40:23.709342 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.709317    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 18:40:23.709342 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.709326    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 18:40:23.709342 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.709334    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 18:40:23.709342 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.709346    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 18:40:23.709650 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.709355    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 18:40:23.710581 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.710569    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 18:40:23.710640 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.710584    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 18:40:23.712025 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:23.712000    2586 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 18:40:23.712118 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:23.712047    2586 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-190.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 18:40:23.714367 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.714352    2586 server.go:1268] "Started kubelet"
Sep 16 18:40:23.714469 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.714421    2586 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 18:40:23.714599 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.714428    2586 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 18:40:23.714707 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.714619    2586 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 18:40:23.715312 ip-10-0-129-190 systemd[1]: Started Kubernetes Kubelet.
Sep 16 18:40:23.715850 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.715837    2586 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 18:40:23.717311 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.717288    2586 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 18:40:23.721188 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:23.721169    2586 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 18:40:23.721462 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.721443    2586 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 18:40:23.721977 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.721962    2586 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 18:40:23.722115 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:23.722075    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-190.ec2.internal\" not found"
Sep 16 18:40:23.722633 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.722605    2586 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 18:40:23.722633 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.722613    2586 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 18:40:23.722633 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.722622    2586 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 18:40:23.722782 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.722720    2586 factory.go:55] Registering systemd factory
Sep 16 18:40:23.722782 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.722739    2586 factory.go:223] Registration of the systemd container factory successfully
Sep 16 18:40:23.722782 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.722781    2586 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 18:40:23.722940 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.722790    2586 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 18:40:23.722995 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.722977    2586 factory.go:153] Registering CRI-O factory
Sep 16 18:40:23.722995 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.722992    2586 factory.go:223] Registration of the crio container factory successfully
Sep 16 18:40:23.723073 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.723055    2586 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 18:40:23.723122 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.723106    2586 factory.go:103] Registering Raw factory
Sep 16 18:40:23.723182 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.723128    2586 manager.go:1196] Started watching for new ooms in manager
Sep 16 18:40:23.723481 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.723469    2586 manager.go:349] Starting recovery of all containers
Sep 16 18:40:23.723896 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.723871    2586 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-190.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 18:40:23.724529 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:23.723554    2586 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-190.ec2.internal.18d5e1301d34e31d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-190.ec2.internal,UID:ip-10-0-129-190.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-190.ec2.internal,},FirstTimestamp:2026-09-16 18:40:23.714325277 +0000 UTC m=+0.393598071,LastTimestamp:2026-09-16 18:40:23.714325277 +0000 UTC m=+0.393598071,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-190.ec2.internal,}"
Sep 16 18:40:23.729007 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:23.728980    2586 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-190.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 18:40:23.729128 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:23.729106    2586 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 18:40:23.733676 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.733648    2586 manager.go:354] Recovery completed
Sep 16 18:40:23.737011 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.736992    2586 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nldqx"
Sep 16 18:40:23.739879 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.739867    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:23.742104 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.742019    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-190.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:23.742104 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.742054    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-190.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:23.742104 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.742067    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-190.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:23.742663 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.742639    2586 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 18:40:23.742663 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.742652    2586 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 18:40:23.742768 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.742667    2586 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:40:23.742768 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.742691    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:23.743471 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.743454    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-190.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:23.743549 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.743486    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-190.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:23.743549 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.743502    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-190.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:23.744140 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.744118    2586 policy_none.go:49] "None policy: Start"
Sep 16 18:40:23.744215 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.744138    2586 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 18:40:23.744215 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.744166    2586 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 18:40:23.744302 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:23.744167    2586 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-190.ec2.internal.18d5e1301edbb95b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-190.ec2.internal,UID:ip-10-0-129-190.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-190.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-190.ec2.internal,},FirstTimestamp:2026-09-16 18:40:23.742036315 +0000 UTC m=+0.421309113,LastTimestamp:2026-09-16 18:40:23.742036315 +0000 UTC m=+0.421309113,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-190.ec2.internal,}"
Sep 16 18:40:23.745323 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.745192    2586 policy_none.go:47] "Start"
Sep 16 18:40:23.746495 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.746474    2586 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nldqx"
Sep 16 18:40:23.755198 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:23.755108    2586 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-190.ec2.internal.18d5e1301edc16f2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-190.ec2.internal,UID:ip-10-0-129-190.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-190.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-190.ec2.internal,},FirstTimestamp:2026-09-16 18:40:23.742060274 +0000 UTC m=+0.421333069,LastTimestamp:2026-09-16 18:40:23.742060274 +0000 UTC m=+0.421333069,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-190.ec2.internal,}"
Sep 16 18:40:23.765166 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:23.765097    2586 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-190.ec2.internal.18d5e1301edc46b2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-190.ec2.internal,UID:ip-10-0-129-190.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-129-190.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-129-190.ec2.internal,},FirstTimestamp:2026-09-16 18:40:23.742072498 +0000 UTC m=+0.421345296,LastTimestamp:2026-09-16 18:40:23.742072498 +0000 UTC m=+0.421345296,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-190.ec2.internal,}"
Sep 16 18:40:23.772509 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.772492    2586 manager.go:337] "Starting Device Plugin manager"
Sep 16 18:40:23.895105 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:23.772533    2586 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 18:40:23.895105 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.772544    2586 server.go:87] "Starting device plugin registration server"
Sep 16 18:40:23.895105 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.772879    2586 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 18:40:23.895105 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.772892    2586 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 18:40:23.895105 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.773065    2586 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 18:40:23.895105 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.773140    2586 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 18:40:23.895105 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.773149    2586 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 18:40:23.895105 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:23.773590    2586 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 18:40:23.895105 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:23.773641    2586 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-190.ec2.internal\" not found"
Sep 16 18:40:23.895105 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:23.778726    2586 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-190.ec2.internal.18d5e1301edbb95b\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-190.ec2.internal.18d5e1301edbb95b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-190.ec2.internal,UID:ip-10-0-129-190.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-190.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-190.ec2.internal,},FirstTimestamp:2026-09-16 18:40:23.742036315 +0000 UTC m=+0.421309113,LastTimestamp:2026-09-16 18:40:23.743469719 +0000 UTC m=+0.422742519,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-190.ec2.internal,}"
Sep 16 18:40:23.895105 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:23.803920    2586 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-190.ec2.internal.18d5e1301edc16f2\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-190.ec2.internal.18d5e1301edc16f2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-190.ec2.internal,UID:ip-10-0-129-190.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-190.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-190.ec2.internal,},FirstTimestamp:2026-09-16 18:40:23.742060274 +0000 UTC m=+0.421333069,LastTimestamp:2026-09-16 18:40:23.743493595 +0000 UTC m=+0.422766391,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-190.ec2.internal,}"
Sep 16 18:40:23.895105 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:23.815686    2586 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-190.ec2.internal.18d5e1301edc46b2\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-190.ec2.internal.18d5e1301edc46b2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-190.ec2.internal,UID:ip-10-0-129-190.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-129-190.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-129-190.ec2.internal,},FirstTimestamp:2026-09-16 18:40:23.742072498 +0000 UTC m=+0.421345296,LastTimestamp:2026-09-16 18:40:23.74350774 +0000 UTC m=+0.422780536,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-190.ec2.internal,}"
Sep 16 18:40:23.895105 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:23.824602    2586 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-190.ec2.internal.18d5e13020d16108  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-190.ec2.internal,UID:ip-10-0-129-190.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-129-190.ec2.internal,},FirstTimestamp:2026-09-16 18:40:23.774912776 +0000 UTC m=+0.454185561,LastTimestamp:2026-09-16 18:40:23.774912776 +0000 UTC m=+0.454185561,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-190.ec2.internal,}"
Sep 16 18:40:23.895105 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.849309    2586 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 18:40:23.895105 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.850516    2586 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 18:40:23.895105 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.850573    2586 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 18:40:23.895105 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.850600    2586 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 18:40:23.895105 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:23.850696    2586 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 18:40:23.895704 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:23.860525    2586 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 16 18:40:23.895704 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.873644    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:23.895704 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.874501    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-190.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:23.895704 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.874524    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-190.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:23.895704 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.874536    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-190.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:23.895704 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.874557    2586 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-190.ec2.internal"
Sep 16 18:40:23.895704 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:23.883340    2586 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-190.ec2.internal.18d5e1301edbb95b\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-190.ec2.internal.18d5e1301edbb95b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-190.ec2.internal,UID:ip-10-0-129-190.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-190.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-190.ec2.internal,},FirstTimestamp:2026-09-16 18:40:23.742036315 +0000 UTC m=+0.421309113,LastTimestamp:2026-09-16 18:40:23.874511343 +0000 UTC m=+0.553784138,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-190.ec2.internal,}"
Sep 16 18:40:23.895704 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:23.885017    2586 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-129-190.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-129-190.ec2.internal"
Sep 16 18:40:23.895704 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:23.885788    2586 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-190.ec2.internal.18d5e1301edc16f2\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-190.ec2.internal.18d5e1301edc16f2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-190.ec2.internal,UID:ip-10-0-129-190.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-190.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-190.ec2.internal,},FirstTimestamp:2026-09-16 18:40:23.742060274 +0000 UTC m=+0.421333069,LastTimestamp:2026-09-16 18:40:23.8745292 +0000 UTC m=+0.553801995,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-190.ec2.internal,}"
Sep 16 18:40:23.895704 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:23.894837    2586 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-190.ec2.internal.18d5e1301edc46b2\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-190.ec2.internal.18d5e1301edc46b2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-190.ec2.internal,UID:ip-10-0-129-190.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-129-190.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-129-190.ec2.internal,},FirstTimestamp:2026-09-16 18:40:23.742072498 +0000 UTC m=+0.421345296,LastTimestamp:2026-09-16 18:40:23.874539756 +0000 UTC m=+0.553812551,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-190.ec2.internal,}"
Sep 16 18:40:23.945291 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:23.945264    2586 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-129-190.ec2.internal\" not found" node="ip-10-0-129-190.ec2.internal"
Sep 16 18:40:23.951516 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.951498    2586 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-190.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-190.ec2.internal"]
Sep 16 18:40:23.951565 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.951559    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:23.952410 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.952394    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-190.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:23.952502 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.952422    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-190.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:23.952502 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.952437    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-190.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:23.953756 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.953744    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:23.953899 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.953885    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-190.ec2.internal"
Sep 16 18:40:23.953933 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.953912    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:23.954430 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.954402    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-190.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:23.954430 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.954430    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-190.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:23.954567 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.954444    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-190.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:23.954567 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.954449    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-190.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:23.954567 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.954470    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-190.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:23.954567 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.954479    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-190.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:23.955672 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.955659    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-190.ec2.internal"
Sep 16 18:40:23.955733 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.955688    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:23.956342 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.956327    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-190.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:23.956424 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.956356    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-190.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:23.956424 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:23.956370    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-190.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:23.969358 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:23.969339    2586 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-190.ec2.internal\" not found" node="ip-10-0-129-190.ec2.internal"
Sep 16 18:40:23.973307 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:23.973291    2586 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-190.ec2.internal\" not found" node="ip-10-0-129-190.ec2.internal"
Sep 16 18:40:24.025051 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:24.025016    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/aacec50d8c91383d21cd8b8f8d4aaf7a-config\") pod \"kube-apiserver-proxy-ip-10-0-129-190.ec2.internal\" (UID: \"aacec50d8c91383d21cd8b8f8d4aaf7a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-190.ec2.internal"
Sep 16 18:40:24.025051 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:24.025052    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fd4c464e8a7b0afa02a1b06875f0d90d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-190.ec2.internal\" (UID: \"fd4c464e8a7b0afa02a1b06875f0d90d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-190.ec2.internal"
Sep 16 18:40:24.025215 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:24.025069    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fd4c464e8a7b0afa02a1b06875f0d90d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-190.ec2.internal\" (UID: \"fd4c464e8a7b0afa02a1b06875f0d90d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-190.ec2.internal"
Sep 16 18:40:24.085765 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:24.085738    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:24.087203 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:24.087186    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-190.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:24.087300 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:24.087223    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-190.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:24.087300 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:24.087241    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-190.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:24.087300 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:24.087273    2586 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-190.ec2.internal"
Sep 16 18:40:24.096037 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:24.096022    2586 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-190.ec2.internal"
Sep 16 18:40:24.096085 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:24.096045    2586 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-190.ec2.internal\": node \"ip-10-0-129-190.ec2.internal\" not found"
Sep 16 18:40:24.109321 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:24.109298    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-190.ec2.internal\" not found"
Sep 16 18:40:24.125888 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:24.125868    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/aacec50d8c91383d21cd8b8f8d4aaf7a-config\") pod \"kube-apiserver-proxy-ip-10-0-129-190.ec2.internal\" (UID: \"aacec50d8c91383d21cd8b8f8d4aaf7a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-190.ec2.internal"
Sep 16 18:40:24.125950 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:24.125898    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/aacec50d8c91383d21cd8b8f8d4aaf7a-config\") pod \"kube-apiserver-proxy-ip-10-0-129-190.ec2.internal\" (UID: \"aacec50d8c91383d21cd8b8f8d4aaf7a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-190.ec2.internal"
Sep 16 18:40:24.125950 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:24.125937    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fd4c464e8a7b0afa02a1b06875f0d90d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-190.ec2.internal\" (UID: \"fd4c464e8a7b0afa02a1b06875f0d90d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-190.ec2.internal"
Sep 16 18:40:24.126023 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:24.125952    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fd4c464e8a7b0afa02a1b06875f0d90d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-190.ec2.internal\" (UID: \"fd4c464e8a7b0afa02a1b06875f0d90d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-190.ec2.internal"
Sep 16 18:40:24.126023 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:24.125976    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fd4c464e8a7b0afa02a1b06875f0d90d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-190.ec2.internal\" (UID: \"fd4c464e8a7b0afa02a1b06875f0d90d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-190.ec2.internal"
Sep 16 18:40:24.126023 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:24.126006    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fd4c464e8a7b0afa02a1b06875f0d90d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-190.ec2.internal\" (UID: \"fd4c464e8a7b0afa02a1b06875f0d90d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-190.ec2.internal"
Sep 16 18:40:24.210325 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:24.210260    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-190.ec2.internal\" not found"
Sep 16 18:40:24.271845 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:24.271793    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-190.ec2.internal"
Sep 16 18:40:24.276109 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:24.276092    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-190.ec2.internal"
Sep 16 18:40:25.546454 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:25.546430    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-190.ec2.internal\" not found"
Sep 16 18:40:25.646961 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:25.646918    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-190.ec2.internal\" not found"
Sep 16 18:40:25.747463 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:25.747396    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-190.ec2.internal\" not found"
Sep 16 18:40:25.846173 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:25.846142    2586 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:25.847731 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:25.847701    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-190.ec2.internal\" not found"
Sep 16 18:40:25.895678 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:25.895643    2586 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 18:40:25.896086 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:25.895807    2586 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:40:25.948223 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:25.948188    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-190.ec2.internal\" not found"
Sep 16 18:40:26.021030 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:26.020999    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaacec50d8c91383d21cd8b8f8d4aaf7a.slice/crio-1d775b20cc66ce5671121eb266fdabb94de54bdf40e709b74479ea59c4acbe68 WatchSource:0}: Error finding container 1d775b20cc66ce5671121eb266fdabb94de54bdf40e709b74479ea59c4acbe68: Status 404 returned error can't find the container with id 1d775b20cc66ce5671121eb266fdabb94de54bdf40e709b74479ea59c4acbe68
Sep 16 18:40:26.021276 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:26.021259    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd4c464e8a7b0afa02a1b06875f0d90d.slice/crio-120ddb967e2aa0f5959e8c8a8b49b154992789b399b438fe36f9943377b01790 WatchSource:0}: Error finding container 120ddb967e2aa0f5959e8c8a8b49b154992789b399b438fe36f9943377b01790: Status 404 returned error can't find the container with id 120ddb967e2aa0f5959e8c8a8b49b154992789b399b438fe36f9943377b01790
Sep 16 18:40:26.027035 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.027015    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:40:26.048326 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:26.048289    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-190.ec2.internal\" not found"
Sep 16 18:40:26.063031 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.063006    2586 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:26.077006 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.076982    2586 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:26.090076 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.090024    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-190.ec2.internal" event={"ID":"fd4c464e8a7b0afa02a1b06875f0d90d","Type":"ContainerStarted","Data":"120ddb967e2aa0f5959e8c8a8b49b154992789b399b438fe36f9943377b01790"}
Sep 16 18:40:26.091171 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.091145    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-190.ec2.internal" event={"ID":"aacec50d8c91383d21cd8b8f8d4aaf7a","Type":"ContainerStarted","Data":"1d775b20cc66ce5671121eb266fdabb94de54bdf40e709b74479ea59c4acbe68"}
Sep 16 18:40:26.133396 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.133346    2586 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 18:35:24 +0000 UTC" deadline="2028-05-13 16:33:02.6342006 +0000 UTC"
Sep 16 18:40:26.133396 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.133391    2586 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14517h52m36.500814564s"
Sep 16 18:40:26.148517 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:26.148494    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-190.ec2.internal\" not found"
Sep 16 18:40:26.170428 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.170401    2586 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:26.258302 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.258279    2586 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-190.ec2.internal"
Sep 16 18:40:26.271041 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.270991    2586 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:40:26.272057 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.272043    2586 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-190.ec2.internal"
Sep 16 18:40:26.286508 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.286490    2586 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:40:26.937808 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.937773    2586 apiserver.go:52] "Watching apiserver"
Sep 16 18:40:26.945500 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.945477    2586 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 18:40:26.945951 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.945927    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-cc7hj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-190.ec2.internal","openshift-multus/multus-additional-cni-plugins-9s7jf","openshift-network-operator/iptables-alerter-vktwg","kube-system/global-pull-secret-syncer-8ktbc","kube-system/kube-apiserver-proxy-ip-10-0-129-190.ec2.internal","openshift-image-registry/node-ca-6298x","openshift-multus/multus-wbl55","openshift-multus/network-metrics-daemon-9vpbt","openshift-network-diagnostics/network-check-target-kmqvn","openshift-ovn-kubernetes/ovnkube-node-wxhg2","kube-system/konnectivity-agent-7n8dt","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbvbq","openshift-cluster-node-tuning-operator/tuned-6kjpc"]
Sep 16 18:40:26.947426 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.947407    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6298x"
Sep 16 18:40:26.948636 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.948614    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbvbq"
Sep 16 18:40:26.949972 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.949928    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9s7jf"
Sep 16 18:40:26.950187 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.950149    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xxwm4\""
Sep 16 18:40:26.950331 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.950271    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 18:40:26.950549 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.950504    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 18:40:26.950675 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.950648    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 18:40:26.950910 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.950883    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 18:40:26.950974 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.950954    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-m77st\""
Sep 16 18:40:26.951665 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.951396    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vktwg"
Sep 16 18:40:26.952550 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.952507    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 18:40:26.952550 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.952520    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ktbc"
Sep 16 18:40:26.952741 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:26.952597    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ktbc" podUID="8c42d544-c24c-4b4b-b88d-13cc74d9fa87"
Sep 16 18:40:26.952869 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.952852    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 18:40:26.953862 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.953834    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6kjpc"
Sep 16 18:40:26.955136 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.955023    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vpbt"
Sep 16 18:40:26.955136 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:26.955093    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vpbt" podUID="1e8169e6-6afe-417b-90f0-5a55bf294956"
Sep 16 18:40:26.956479 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.955981    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:40:26.956479 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.955990    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kxdmp\""
Sep 16 18:40:26.956479 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.956297    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 18:40:26.956479 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.956355    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:40:26.956479 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.956355    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-49t4m\""
Sep 16 18:40:26.956479 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.956397    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 18:40:26.956772 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.956490    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7wwdn\""
Sep 16 18:40:26.956772 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.956553    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cc7hj"
Sep 16 18:40:26.957100 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.957082    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 18:40:26.957331 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.957314    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 18:40:26.957583 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.957563    2586 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 18:40:26.957681 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.957570    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 18:40:26.957681 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.957672    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 18:40:26.958106 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.958086    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wbl55"
Sep 16 18:40:26.960170 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.960026    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmqvn"
Sep 16 18:40:26.960170 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:26.960121    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kmqvn" podUID="cbd2c59d-9acb-4f04-8068-c0e2d13e5d41"
Sep 16 18:40:26.961839 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.961762    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-mdj2f\""
Sep 16 18:40:26.961839 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.961771    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 18:40:26.962087 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.962061    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 18:40:26.963212 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.962159    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 18:40:26.963212 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.962395    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 18:40:26.963212 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.962401    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 18:40:26.963212 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.962622    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kwhz8\""
Sep 16 18:40:26.965259 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.964592    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:26.966736 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.966525    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7n8dt"
Sep 16 18:40:26.967656 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.967640    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 18:40:26.967886 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.967869    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 18:40:26.968156 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.968142    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 18:40:26.968261 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.968165    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 18:40:26.968261 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.968166    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 18:40:26.968479 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.968464    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 18:40:26.968751 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.968715    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-hdgp7\""
Sep 16 18:40:26.969021 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.969002    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 18:40:26.969021 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.969014    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6hlsl\""
Sep 16 18:40:26.969641 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.969618    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/79f801e9-77ff-4d5d-8d54-ffb25be9dd3a-sys-fs\") pod \"aws-ebs-csi-driver-node-kbvbq\" (UID: \"79f801e9-77ff-4d5d-8d54-ffb25be9dd3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbvbq"
Sep 16 18:40:26.969730 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.969656    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9716763a-ff84-4735-b478-deb614be27c4-cni-binary-copy\") pod \"multus-additional-cni-plugins-9s7jf\" (UID: \"9716763a-ff84-4735-b478-deb614be27c4\") " pod="openshift-multus/multus-additional-cni-plugins-9s7jf"
Sep 16 18:40:26.969730 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.969670    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 18:40:26.969730 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.969678    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2807fdd0-b58d-446d-ac9b-84a50d9b9ac6-sys\") pod \"tuned-6kjpc\" (UID: \"2807fdd0-b58d-446d-ac9b-84a50d9b9ac6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kjpc"
Sep 16 18:40:26.969921 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.969732    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2807fdd0-b58d-446d-ac9b-84a50d9b9ac6-host\") pod \"tuned-6kjpc\" (UID: \"2807fdd0-b58d-446d-ac9b-84a50d9b9ac6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kjpc"
Sep 16 18:40:26.969921 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.969784    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7fe5b556-37c9-40c7-a09b-5c6116b43e6f-host\") pod \"node-ca-6298x\" (UID: \"7fe5b556-37c9-40c7-a09b-5c6116b43e6f\") " pod="openshift-image-registry/node-ca-6298x"
Sep 16 18:40:26.969921 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.969816    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j227\" (UniqueName: \"kubernetes.io/projected/79f801e9-77ff-4d5d-8d54-ffb25be9dd3a-kube-api-access-7j227\") pod \"aws-ebs-csi-driver-node-kbvbq\" (UID: \"79f801e9-77ff-4d5d-8d54-ffb25be9dd3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbvbq"
Sep 16 18:40:26.969921 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.969849    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8c42d544-c24c-4b4b-b88d-13cc74d9fa87-kubelet-config\") pod \"global-pull-secret-syncer-8ktbc\" (UID: \"8c42d544-c24c-4b4b-b88d-13cc74d9fa87\") " pod="kube-system/global-pull-secret-syncer-8ktbc"
Sep 16 18:40:26.969921 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.969878    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c42d544-c24c-4b4b-b88d-13cc74d9fa87-original-pull-secret\") pod \"global-pull-secret-syncer-8ktbc\" (UID: \"8c42d544-c24c-4b4b-b88d-13cc74d9fa87\") " pod="kube-system/global-pull-secret-syncer-8ktbc"
Sep 16 18:40:26.969921 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.969900    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2807fdd0-b58d-446d-ac9b-84a50d9b9ac6-tmp\") pod \"tuned-6kjpc\" (UID: \"2807fdd0-b58d-446d-ac9b-84a50d9b9ac6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kjpc"
Sep 16 18:40:26.969921 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.969918    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b58ed079-b715-420e-a11f-2023c52e1321-tmp-dir\") pod \"node-resolver-cc7hj\" (UID: \"b58ed079-b715-420e-a11f-2023c52e1321\") " pod="openshift-dns/node-resolver-cc7hj"
Sep 16 18:40:26.970289 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.969941    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/66e76bc5-f7b3-4952-b67c-0fb014f21ecb-host-run-netns\") pod \"multus-wbl55\" (UID: \"66e76bc5-f7b3-4952-b67c-0fb014f21ecb\") " pod="openshift-multus/multus-wbl55"
Sep 16 18:40:26.970289 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.969965    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66e76bc5-f7b3-4952-b67c-0fb014f21ecb-etc-kubernetes\") pod \"multus-wbl55\" (UID: \"66e76bc5-f7b3-4952-b67c-0fb014f21ecb\") " pod="openshift-multus/multus-wbl55"
Sep 16 18:40:26.970289 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.969987    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e8169e6-6afe-417b-90f0-5a55bf294956-metrics-certs\") pod \"network-metrics-daemon-9vpbt\" (UID: \"1e8169e6-6afe-417b-90f0-5a55bf294956\") " pod="openshift-multus/network-metrics-daemon-9vpbt"
Sep 16 18:40:26.970289 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.970010    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/21e31a4b-fd39-4863-ada4-4c87f208a134-iptables-alerter-script\") pod \"iptables-alerter-vktwg\" (UID: \"21e31a4b-fd39-4863-ada4-4c87f208a134\") " pod="openshift-network-operator/iptables-alerter-vktwg"
Sep 16 18:40:26.970289 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.970033    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnzbm\" (UniqueName: \"kubernetes.io/projected/21e31a4b-fd39-4863-ada4-4c87f208a134-kube-api-access-cnzbm\") pod \"iptables-alerter-vktwg\" (UID: \"21e31a4b-fd39-4863-ada4-4c87f208a134\") " pod="openshift-network-operator/iptables-alerter-vktwg"
Sep 16 18:40:26.970289 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.970055    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2807fdd0-b58d-446d-ac9b-84a50d9b9ac6-etc-sysctl-conf\") pod \"tuned-6kjpc\" (UID: \"2807fdd0-b58d-446d-ac9b-84a50d9b9ac6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kjpc"
Sep 16 18:40:26.970289 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.970077    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b58ed079-b715-420e-a11f-2023c52e1321-hosts-file\") pod \"node-resolver-cc7hj\" (UID: \"b58ed079-b715-420e-a11f-2023c52e1321\") " pod="openshift-dns/node-resolver-cc7hj"
Sep 16 18:40:26.970289 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.970131    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66e76bc5-f7b3-4952-b67c-0fb014f21ecb-multus-cni-dir\") pod \"multus-wbl55\" (UID: \"66e76bc5-f7b3-4952-b67c-0fb014f21ecb\") " pod="openshift-multus/multus-wbl55"
Sep 16 18:40:26.970289 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.970157    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2807fdd0-b58d-446d-ac9b-84a50d9b9ac6-etc-modprobe-d\") pod \"tuned-6kjpc\" (UID: \"2807fdd0-b58d-446d-ac9b-84a50d9b9ac6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kjpc"
Sep 16 18:40:26.970289 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.970181    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkvjt\" (UniqueName: \"kubernetes.io/projected/9716763a-ff84-4735-b478-deb614be27c4-kube-api-access-gkvjt\") pod \"multus-additional-cni-plugins-9s7jf\" (UID: \"9716763a-ff84-4735-b478-deb614be27c4\") " pod="openshift-multus/multus-additional-cni-plugins-9s7jf"
Sep 16 18:40:26.970289 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.970199    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2807fdd0-b58d-446d-ac9b-84a50d9b9ac6-etc-sysconfig\") pod \"tuned-6kjpc\" (UID: \"2807fdd0-b58d-446d-ac9b-84a50d9b9ac6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kjpc"
Sep 16 18:40:26.970289 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.970224    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/79f801e9-77ff-4d5d-8d54-ffb25be9dd3a-device-dir\") pod \"aws-ebs-csi-driver-node-kbvbq\" (UID: \"79f801e9-77ff-4d5d-8d54-ffb25be9dd3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbvbq"
Sep 16 18:40:26.970289 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.970263    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/66e76bc5-f7b3-4952-b67c-0fb014f21ecb-host-run-k8s-cni-cncf-io\") pod \"multus-wbl55\" (UID: \"66e76bc5-f7b3-4952-b67c-0fb014f21ecb\") " pod="openshift-multus/multus-wbl55"
Sep 16 18:40:26.970289 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.970283    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-564kw\" (UniqueName: \"kubernetes.io/projected/b58ed079-b715-420e-a11f-2023c52e1321-kube-api-access-564kw\") pod \"node-resolver-cc7hj\" (UID: \"b58ed079-b715-420e-a11f-2023c52e1321\") " pod="openshift-dns/node-resolver-cc7hj"
Sep 16 18:40:26.970289 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.970298    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4jzq\" (UniqueName: \"kubernetes.io/projected/cbd2c59d-9acb-4f04-8068-c0e2d13e5d41-kube-api-access-k4jzq\") pod \"network-check-target-kmqvn\" (UID: \"cbd2c59d-9acb-4f04-8068-c0e2d13e5d41\") " pod="openshift-network-diagnostics/network-check-target-kmqvn"
Sep 16 18:40:26.971045 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.970311    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/79f801e9-77ff-4d5d-8d54-ffb25be9dd3a-etc-selinux\") pod \"aws-ebs-csi-driver-node-kbvbq\" (UID: \"79f801e9-77ff-4d5d-8d54-ffb25be9dd3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbvbq"
Sep 16 18:40:26.971045 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.970326    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/21e31a4b-fd39-4863-ada4-4c87f208a134-host-slash\") pod \"iptables-alerter-vktwg\" (UID: \"21e31a4b-fd39-4863-ada4-4c87f208a134\") " pod="openshift-network-operator/iptables-alerter-vktwg"
Sep 16 18:40:26.971045 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.970351    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/66e76bc5-f7b3-4952-b67c-0fb014f21ecb-multus-daemon-config\") pod \"multus-wbl55\" (UID: \"66e76bc5-f7b3-4952-b67c-0fb014f21ecb\") " pod="openshift-multus/multus-wbl55"
Sep 16 18:40:26.971045 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.970376    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67cdx\" (UniqueName: \"kubernetes.io/projected/7fe5b556-37c9-40c7-a09b-5c6116b43e6f-kube-api-access-67cdx\") pod \"node-ca-6298x\" (UID: \"7fe5b556-37c9-40c7-a09b-5c6116b43e6f\") " pod="openshift-image-registry/node-ca-6298x"
Sep 16 18:40:26.971045 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.970391    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2807fdd0-b58d-446d-ac9b-84a50d9b9ac6-var-lib-kubelet\") pod \"tuned-6kjpc\" (UID: \"2807fdd0-b58d-446d-ac9b-84a50d9b9ac6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kjpc"
Sep 16 18:40:26.971045 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.970408    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/66e76bc5-f7b3-4952-b67c-0fb014f21ecb-cni-binary-copy\") pod \"multus-wbl55\" (UID: \"66e76bc5-f7b3-4952-b67c-0fb014f21ecb\") " pod="openshift-multus/multus-wbl55"
Sep 16 18:40:26.971045 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.970428    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9716763a-ff84-4735-b478-deb614be27c4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9s7jf\" (UID: \"9716763a-ff84-4735-b478-deb614be27c4\") " pod="openshift-multus/multus-additional-cni-plugins-9s7jf"
Sep 16 18:40:26.971045 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.970449    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/79f801e9-77ff-4d5d-8d54-ffb25be9dd3a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kbvbq\" (UID: \"79f801e9-77ff-4d5d-8d54-ffb25be9dd3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbvbq"
Sep 16 18:40:26.971045 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.970481    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66e76bc5-f7b3-4952-b67c-0fb014f21ecb-system-cni-dir\") pod \"multus-wbl55\" (UID: \"66e76bc5-f7b3-4952-b67c-0fb014f21ecb\") " pod="openshift-multus/multus-wbl55"
Sep 16 18:40:26.971045 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.970530    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/66e76bc5-f7b3-4952-b67c-0fb014f21ecb-host-run-multus-certs\") pod \"multus-wbl55\" (UID: \"66e76bc5-f7b3-4952-b67c-0fb014f21ecb\") " pod="openshift-multus/multus-wbl55"
Sep 16 18:40:26.971045 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.970552    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8c42d544-c24c-4b4b-b88d-13cc74d9fa87-dbus\") pod \"global-pull-secret-syncer-8ktbc\" (UID: \"8c42d544-c24c-4b4b-b88d-13cc74d9fa87\") " pod="kube-system/global-pull-secret-syncer-8ktbc"
Sep 16 18:40:26.971045 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.970575    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2807fdd0-b58d-446d-ac9b-84a50d9b9ac6-etc-sysctl-d\") pod \"tuned-6kjpc\" (UID: \"2807fdd0-b58d-446d-ac9b-84a50d9b9ac6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kjpc"
Sep 16 18:40:26.971045 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.970597    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/66e76bc5-f7b3-4952-b67c-0fb014f21ecb-host-var-lib-cni-bin\") pod \"multus-wbl55\" (UID: \"66e76bc5-f7b3-4952-b67c-0fb014f21ecb\") " pod="openshift-multus/multus-wbl55"
Sep 16 18:40:26.971045 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.970622    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/66e76bc5-f7b3-4952-b67c-0fb014f21ecb-multus-conf-dir\") pod \"multus-wbl55\" (UID: \"66e76bc5-f7b3-4952-b67c-0fb014f21ecb\") " pod="openshift-multus/multus-wbl55"
Sep 16 18:40:26.971045 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.970643    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scxfj\" (UniqueName: \"kubernetes.io/projected/66e76bc5-f7b3-4952-b67c-0fb014f21ecb-kube-api-access-scxfj\") pod \"multus-wbl55\" (UID: \"66e76bc5-f7b3-4952-b67c-0fb014f21ecb\") " pod="openshift-multus/multus-wbl55"
Sep 16 18:40:26.971045 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.970666    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwdgz\" (UniqueName: \"kubernetes.io/projected/1e8169e6-6afe-417b-90f0-5a55bf294956-kube-api-access-lwdgz\") pod \"network-metrics-daemon-9vpbt\" (UID: \"1e8169e6-6afe-417b-90f0-5a55bf294956\") " pod="openshift-multus/network-metrics-daemon-9vpbt"
Sep 16 18:40:26.971725 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.970726    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9716763a-ff84-4735-b478-deb614be27c4-system-cni-dir\") pod \"multus-additional-cni-plugins-9s7jf\" (UID: \"9716763a-ff84-4735-b478-deb614be27c4\") " pod="openshift-multus/multus-additional-cni-plugins-9s7jf"
Sep 16 18:40:26.971725 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.970789    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/79f801e9-77ff-4d5d-8d54-ffb25be9dd3a-socket-dir\") pod \"aws-ebs-csi-driver-node-kbvbq\" (UID: \"79f801e9-77ff-4d5d-8d54-ffb25be9dd3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbvbq"
Sep 16 18:40:26.971725 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.970814    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9716763a-ff84-4735-b478-deb614be27c4-cnibin\") pod \"multus-additional-cni-plugins-9s7jf\" (UID: \"9716763a-ff84-4735-b478-deb614be27c4\") " pod="openshift-multus/multus-additional-cni-plugins-9s7jf"
Sep 16 18:40:26.971725 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.970840    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2807fdd0-b58d-446d-ac9b-84a50d9b9ac6-etc-kubernetes\") pod \"tuned-6kjpc\" (UID: \"2807fdd0-b58d-446d-ac9b-84a50d9b9ac6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kjpc"
Sep 16 18:40:26.971725 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.970885    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2807fdd0-b58d-446d-ac9b-84a50d9b9ac6-etc-systemd\") pod \"tuned-6kjpc\" (UID: \"2807fdd0-b58d-446d-ac9b-84a50d9b9ac6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kjpc"
Sep 16 18:40:26.971725 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.970925    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2807fdd0-b58d-446d-ac9b-84a50d9b9ac6-run\") pod \"tuned-6kjpc\" (UID: \"2807fdd0-b58d-446d-ac9b-84a50d9b9ac6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kjpc"
Sep 16 18:40:26.971725 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.970955    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2807fdd0-b58d-446d-ac9b-84a50d9b9ac6-lib-modules\") pod \"tuned-6kjpc\" (UID: \"2807fdd0-b58d-446d-ac9b-84a50d9b9ac6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kjpc"
Sep 16 18:40:26.971725 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.970984    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7fe5b556-37c9-40c7-a09b-5c6116b43e6f-serviceca\") pod \"node-ca-6298x\" (UID: \"7fe5b556-37c9-40c7-a09b-5c6116b43e6f\") " pod="openshift-image-registry/node-ca-6298x"
Sep 16 18:40:26.971725 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.971008    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/66e76bc5-f7b3-4952-b67c-0fb014f21ecb-host-var-lib-cni-multus\") pod \"multus-wbl55\" (UID: \"66e76bc5-f7b3-4952-b67c-0fb014f21ecb\") " pod="openshift-multus/multus-wbl55"
Sep 16 18:40:26.971725 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.971081    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/66e76bc5-f7b3-4952-b67c-0fb014f21ecb-host-var-lib-kubelet\") pod \"multus-wbl55\" (UID: \"66e76bc5-f7b3-4952-b67c-0fb014f21ecb\") " pod="openshift-multus/multus-wbl55"
Sep 16 18:40:26.971725 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.971106    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/66e76bc5-f7b3-4952-b67c-0fb014f21ecb-hostroot\") pod \"multus-wbl55\" (UID: \"66e76bc5-f7b3-4952-b67c-0fb014f21ecb\") " pod="openshift-multus/multus-wbl55"
Sep 16 18:40:26.971725 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.971132    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9716763a-ff84-4735-b478-deb614be27c4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9s7jf\" (UID: \"9716763a-ff84-4735-b478-deb614be27c4\") " pod="openshift-multus/multus-additional-cni-plugins-9s7jf"
Sep 16 18:40:26.971725 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.971154    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/66e76bc5-f7b3-4952-b67c-0fb014f21ecb-cnibin\") pod \"multus-wbl55\" (UID: \"66e76bc5-f7b3-4952-b67c-0fb014f21ecb\") " pod="openshift-multus/multus-wbl55"
Sep 16 18:40:26.971725 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.971169    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/79f801e9-77ff-4d5d-8d54-ffb25be9dd3a-registration-dir\") pod \"aws-ebs-csi-driver-node-kbvbq\" (UID: \"79f801e9-77ff-4d5d-8d54-ffb25be9dd3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbvbq"
Sep 16 18:40:26.971725 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.971210    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9716763a-ff84-4735-b478-deb614be27c4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9s7jf\" (UID: \"9716763a-ff84-4735-b478-deb614be27c4\") " pod="openshift-multus/multus-additional-cni-plugins-9s7jf"
Sep 16 18:40:26.971725 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.971271    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2807fdd0-b58d-446d-ac9b-84a50d9b9ac6-etc-tuned\") pod \"tuned-6kjpc\" (UID: \"2807fdd0-b58d-446d-ac9b-84a50d9b9ac6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kjpc"
Sep 16 18:40:26.972284 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.971298    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/66e76bc5-f7b3-4952-b67c-0fb014f21ecb-multus-socket-dir-parent\") pod \"multus-wbl55\" (UID: \"66e76bc5-f7b3-4952-b67c-0fb014f21ecb\") " pod="openshift-multus/multus-wbl55"
Sep 16 18:40:26.972284 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.971323    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t4pz\" (UniqueName: \"kubernetes.io/projected/2807fdd0-b58d-446d-ac9b-84a50d9b9ac6-kube-api-access-5t4pz\") pod \"tuned-6kjpc\" (UID: \"2807fdd0-b58d-446d-ac9b-84a50d9b9ac6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kjpc"
Sep 16 18:40:26.972592 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.972574    2586 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:40:27.000004 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:26.999971    2586 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-r4jlw"
Sep 16 18:40:27.019084 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.019055    2586 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-r4jlw"
Sep 16 18:40:27.059731 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.059702    2586 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 18:40:27.071495 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.071466    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/21e31a4b-fd39-4863-ada4-4c87f208a134-host-slash\") pod \"iptables-alerter-vktwg\" (UID: \"21e31a4b-fd39-4863-ada4-4c87f208a134\") " pod="openshift-network-operator/iptables-alerter-vktwg"
Sep 16 18:40:27.071658 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.071502    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/66e76bc5-f7b3-4952-b67c-0fb014f21ecb-multus-daemon-config\") pod \"multus-wbl55\" (UID: \"66e76bc5-f7b3-4952-b67c-0fb014f21ecb\") " pod="openshift-multus/multus-wbl55"
Sep 16 18:40:27.071658 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.071528    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67cdx\" (UniqueName: \"kubernetes.io/projected/7fe5b556-37c9-40c7-a09b-5c6116b43e6f-kube-api-access-67cdx\") pod \"node-ca-6298x\" (UID: \"7fe5b556-37c9-40c7-a09b-5c6116b43e6f\") " pod="openshift-image-registry/node-ca-6298x"
Sep 16 18:40:27.071658 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.071549    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2807fdd0-b58d-446d-ac9b-84a50d9b9ac6-var-lib-kubelet\") pod \"tuned-6kjpc\" (UID: \"2807fdd0-b58d-446d-ac9b-84a50d9b9ac6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kjpc"
Sep 16 18:40:27.071658 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.071575    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f76a1a1e-3387-4aae-8964-63c913eb270a-var-lib-openvswitch\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.071658 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.071593    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/21e31a4b-fd39-4863-ada4-4c87f208a134-host-slash\") pod \"iptables-alerter-vktwg\" (UID: \"21e31a4b-fd39-4863-ada4-4c87f208a134\") " pod="openshift-network-operator/iptables-alerter-vktwg"
Sep 16 18:40:27.071658 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.071598    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/66e76bc5-f7b3-4952-b67c-0fb014f21ecb-cni-binary-copy\") pod \"multus-wbl55\" (UID: \"66e76bc5-f7b3-4952-b67c-0fb014f21ecb\") " pod="openshift-multus/multus-wbl55"
Sep 16 18:40:27.071964 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.071660    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9716763a-ff84-4735-b478-deb614be27c4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9s7jf\" (UID: \"9716763a-ff84-4735-b478-deb614be27c4\") " pod="openshift-multus/multus-additional-cni-plugins-9s7jf"
Sep 16 18:40:27.071964 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.071693    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f76a1a1e-3387-4aae-8964-63c913eb270a-etc-openvswitch\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.071964 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.071717    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f76a1a1e-3387-4aae-8964-63c913eb270a-node-log\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.071964 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.071740    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f76a1a1e-3387-4aae-8964-63c913eb270a-log-socket\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.071964 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.071765    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/79f801e9-77ff-4d5d-8d54-ffb25be9dd3a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kbvbq\" (UID: \"79f801e9-77ff-4d5d-8d54-ffb25be9dd3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbvbq"
Sep 16 18:40:27.071964 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.071803    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66e76bc5-f7b3-4952-b67c-0fb014f21ecb-system-cni-dir\") pod \"multus-wbl55\" (UID: \"66e76bc5-f7b3-4952-b67c-0fb014f21ecb\") " pod="openshift-multus/multus-wbl55"
Sep 16 18:40:27.071964 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.071838    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/66e76bc5-f7b3-4952-b67c-0fb014f21ecb-host-run-multus-certs\") pod \"multus-wbl55\" (UID: \"66e76bc5-f7b3-4952-b67c-0fb014f21ecb\") " pod="openshift-multus/multus-wbl55"
Sep 16 18:40:27.071964 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.071861    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8c42d544-c24c-4b4b-b88d-13cc74d9fa87-dbus\") pod \"global-pull-secret-syncer-8ktbc\" (UID: \"8c42d544-c24c-4b4b-b88d-13cc74d9fa87\") " pod="kube-system/global-pull-secret-syncer-8ktbc"
Sep 16 18:40:27.071964 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.071884    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2807fdd0-b58d-446d-ac9b-84a50d9b9ac6-etc-sysctl-d\") pod \"tuned-6kjpc\" (UID: \"2807fdd0-b58d-446d-ac9b-84a50d9b9ac6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kjpc"
Sep 16 18:40:27.071964 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.071910    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f76a1a1e-3387-4aae-8964-63c913eb270a-host-run-netns\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.071964 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.071941    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/66e76bc5-f7b3-4952-b67c-0fb014f21ecb-host-var-lib-cni-bin\") pod \"multus-wbl55\" (UID: \"66e76bc5-f7b3-4952-b67c-0fb014f21ecb\") " pod="openshift-multus/multus-wbl55"
Sep 16 18:40:27.071964 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.071966    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/66e76bc5-f7b3-4952-b67c-0fb014f21ecb-multus-conf-dir\") pod \"multus-wbl55\" (UID: \"66e76bc5-f7b3-4952-b67c-0fb014f21ecb\") " pod="openshift-multus/multus-wbl55"
Sep 16 18:40:27.072535 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.071992    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scxfj\" (UniqueName: \"kubernetes.io/projected/66e76bc5-f7b3-4952-b67c-0fb014f21ecb-kube-api-access-scxfj\") pod \"multus-wbl55\" (UID: \"66e76bc5-f7b3-4952-b67c-0fb014f21ecb\") " pod="openshift-multus/multus-wbl55"
Sep 16 18:40:27.072535 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.072019    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwdgz\" (UniqueName: \"kubernetes.io/projected/1e8169e6-6afe-417b-90f0-5a55bf294956-kube-api-access-lwdgz\") pod \"network-metrics-daemon-9vpbt\" (UID: \"1e8169e6-6afe-417b-90f0-5a55bf294956\") " pod="openshift-multus/network-metrics-daemon-9vpbt"
Sep 16 18:40:27.072535 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.072047    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9716763a-ff84-4735-b478-deb614be27c4-system-cni-dir\") pod \"multus-additional-cni-plugins-9s7jf\" (UID: \"9716763a-ff84-4735-b478-deb614be27c4\") " pod="openshift-multus/multus-additional-cni-plugins-9s7jf"
Sep 16 18:40:27.072535 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.072070    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f76a1a1e-3387-4aae-8964-63c913eb270a-systemd-units\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.072535 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.072086    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/66e76bc5-f7b3-4952-b67c-0fb014f21ecb-host-run-multus-certs\") pod \"multus-wbl55\" (UID: \"66e76bc5-f7b3-4952-b67c-0fb014f21ecb\") " pod="openshift-multus/multus-wbl55"
Sep 16 18:40:27.072535 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.072094    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f76a1a1e-3387-4aae-8964-63c913eb270a-host-run-ovn-kubernetes\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.072535 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.072142    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2807fdd0-b58d-446d-ac9b-84a50d9b9ac6-var-lib-kubelet\") pod \"tuned-6kjpc\" (UID: \"2807fdd0-b58d-446d-ac9b-84a50d9b9ac6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kjpc"
Sep 16 18:40:27.072535 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.072134    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6f68d3ed-0908-4d8a-a879-5b002a923ca9-agent-certs\") pod \"konnectivity-agent-7n8dt\" (UID: \"6f68d3ed-0908-4d8a-a879-5b002a923ca9\") " pod="kube-system/konnectivity-agent-7n8dt"
Sep 16 18:40:27.072535 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.072164    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/79f801e9-77ff-4d5d-8d54-ffb25be9dd3a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kbvbq\" (UID: \"79f801e9-77ff-4d5d-8d54-ffb25be9dd3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbvbq"
Sep 16 18:40:27.072535 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.072185    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/79f801e9-77ff-4d5d-8d54-ffb25be9dd3a-socket-dir\") pod \"aws-ebs-csi-driver-node-kbvbq\" (UID: \"79f801e9-77ff-4d5d-8d54-ffb25be9dd3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbvbq"
Sep 16 18:40:27.072535 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.072203    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9716763a-ff84-4735-b478-deb614be27c4-cnibin\") pod \"multus-additional-cni-plugins-9s7jf\" (UID: \"9716763a-ff84-4735-b478-deb614be27c4\") " pod="openshift-multus/multus-additional-cni-plugins-9s7jf"
Sep 16 18:40:27.072535 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.072272    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/66e76bc5-f7b3-4952-b67c-0fb014f21ecb-cni-binary-copy\") pod \"multus-wbl55\" (UID: \"66e76bc5-f7b3-4952-b67c-0fb014f21ecb\") " pod="openshift-multus/multus-wbl55"
Sep 16 18:40:27.072535 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.072247    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2807fdd0-b58d-446d-ac9b-84a50d9b9ac6-etc-kubernetes\") pod \"tuned-6kjpc\" (UID: \"2807fdd0-b58d-446d-ac9b-84a50d9b9ac6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kjpc"
Sep 16 18:40:27.072535 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.072288    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66e76bc5-f7b3-4952-b67c-0fb014f21ecb-system-cni-dir\") pod \"multus-wbl55\" (UID: \"66e76bc5-f7b3-4952-b67c-0fb014f21ecb\") " pod="openshift-multus/multus-wbl55"
Sep 16 18:40:27.072535 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.072302    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2807fdd0-b58d-446d-ac9b-84a50d9b9ac6-etc-systemd\") pod \"tuned-6kjpc\" (UID: \"2807fdd0-b58d-446d-ac9b-84a50d9b9ac6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kjpc"
Sep 16 18:40:27.072535 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.072332    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f76a1a1e-3387-4aae-8964-63c913eb270a-host-kubelet\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.072535 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.072345    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2807fdd0-b58d-446d-ac9b-84a50d9b9ac6-etc-systemd\") pod \"tuned-6kjpc\" (UID: \"2807fdd0-b58d-446d-ac9b-84a50d9b9ac6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kjpc"
Sep 16 18:40:27.073371 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.072363    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f76a1a1e-3387-4aae-8964-63c913eb270a-ovn-node-metrics-cert\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.073371 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.072382    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8c42d544-c24c-4b4b-b88d-13cc74d9fa87-dbus\") pod \"global-pull-secret-syncer-8ktbc\" (UID: \"8c42d544-c24c-4b4b-b88d-13cc74d9fa87\") " pod="kube-system/global-pull-secret-syncer-8ktbc"
Sep 16 18:40:27.073371 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.072414    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2807fdd0-b58d-446d-ac9b-84a50d9b9ac6-run\") pod \"tuned-6kjpc\" (UID: \"2807fdd0-b58d-446d-ac9b-84a50d9b9ac6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kjpc"
Sep 16 18:40:27.073371 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.072419    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9716763a-ff84-4735-b478-deb614be27c4-system-cni-dir\") pod \"multus-additional-cni-plugins-9s7jf\" (UID: \"9716763a-ff84-4735-b478-deb614be27c4\") " pod="openshift-multus/multus-additional-cni-plugins-9s7jf"
Sep 16 18:40:27.073371 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.072440    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2807fdd0-b58d-446d-ac9b-84a50d9b9ac6-lib-modules\") pod \"tuned-6kjpc\" (UID: \"2807fdd0-b58d-446d-ac9b-84a50d9b9ac6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kjpc"
Sep 16 18:40:27.073371 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.072489    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2807fdd0-b58d-446d-ac9b-84a50d9b9ac6-etc-kubernetes\") pod \"tuned-6kjpc\" (UID: \"2807fdd0-b58d-446d-ac9b-84a50d9b9ac6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kjpc"
Sep 16 18:40:27.073371 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.072499    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/79f801e9-77ff-4d5d-8d54-ffb25be9dd3a-socket-dir\") pod \"aws-ebs-csi-driver-node-kbvbq\" (UID: \"79f801e9-77ff-4d5d-8d54-ffb25be9dd3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbvbq"
Sep 16 18:40:27.073371 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.072505    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2807fdd0-b58d-446d-ac9b-84a50d9b9ac6-etc-sysctl-d\") pod \"tuned-6kjpc\" (UID: \"2807fdd0-b58d-446d-ac9b-84a50d9b9ac6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kjpc"
Sep 16 18:40:27.073371 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.072506    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/66e76bc5-f7b3-4952-b67c-0fb014f21ecb-host-var-lib-cni-bin\") pod \"multus-wbl55\" (UID: \"66e76bc5-f7b3-4952-b67c-0fb014f21ecb\") " pod="openshift-multus/multus-wbl55"
Sep 16 18:40:27.073371 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.072517    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7fe5b556-37c9-40c7-a09b-5c6116b43e6f-serviceca\") pod \"node-ca-6298x\" (UID: \"7fe5b556-37c9-40c7-a09b-5c6116b43e6f\") " pod="openshift-image-registry/node-ca-6298x"
Sep 16 18:40:27.073371 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.072512    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2807fdd0-b58d-446d-ac9b-84a50d9b9ac6-run\") pod \"tuned-6kjpc\" (UID: \"2807fdd0-b58d-446d-ac9b-84a50d9b9ac6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kjpc"
Sep 16 18:40:27.073371 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.072556    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/66e76bc5-f7b3-4952-b67c-0fb014f21ecb-host-var-lib-cni-multus\") pod \"multus-wbl55\" (UID: \"66e76bc5-f7b3-4952-b67c-0fb014f21ecb\") " pod="openshift-multus/multus-wbl55"
Sep 16 18:40:27.073371 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.072595    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2807fdd0-b58d-446d-ac9b-84a50d9b9ac6-lib-modules\") pod \"tuned-6kjpc\" (UID: \"2807fdd0-b58d-446d-ac9b-84a50d9b9ac6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kjpc"
Sep 16 18:40:27.073371 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.072670    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/66e76bc5-f7b3-4952-b67c-0fb014f21ecb-host-var-lib-cni-multus\") pod \"multus-wbl55\" (UID: \"66e76bc5-f7b3-4952-b67c-0fb014f21ecb\") " pod="openshift-multus/multus-wbl55"
Sep 16 18:40:27.073371 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.072638    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/66e76bc5-f7b3-4952-b67c-0fb014f21ecb-host-var-lib-kubelet\") pod \"multus-wbl55\" (UID: \"66e76bc5-f7b3-4952-b67c-0fb014f21ecb\") " pod="openshift-multus/multus-wbl55"
Sep 16 18:40:27.073371 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.072714    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9716763a-ff84-4735-b478-deb614be27c4-cnibin\") pod \"multus-additional-cni-plugins-9s7jf\" (UID: \"9716763a-ff84-4735-b478-deb614be27c4\") " pod="openshift-multus/multus-additional-cni-plugins-9s7jf"
Sep 16 18:40:27.073371 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.072726    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/66e76bc5-f7b3-4952-b67c-0fb014f21ecb-multus-conf-dir\") pod \"multus-wbl55\" (UID: \"66e76bc5-f7b3-4952-b67c-0fb014f21ecb\") " pod="openshift-multus/multus-wbl55"
Sep 16 18:40:27.073371 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.072777    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/66e76bc5-f7b3-4952-b67c-0fb014f21ecb-host-var-lib-kubelet\") pod \"multus-wbl55\" (UID: \"66e76bc5-f7b3-4952-b67c-0fb014f21ecb\") " pod="openshift-multus/multus-wbl55"
Sep 16 18:40:27.074196 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.072765    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/66e76bc5-f7b3-4952-b67c-0fb014f21ecb-hostroot\") pod \"multus-wbl55\" (UID: \"66e76bc5-f7b3-4952-b67c-0fb014f21ecb\") " pod="openshift-multus/multus-wbl55"
Sep 16 18:40:27.074196 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.072817    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/66e76bc5-f7b3-4952-b67c-0fb014f21ecb-hostroot\") pod \"multus-wbl55\" (UID: \"66e76bc5-f7b3-4952-b67c-0fb014f21ecb\") " pod="openshift-multus/multus-wbl55"
Sep 16 18:40:27.074196 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.072824    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9716763a-ff84-4735-b478-deb614be27c4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9s7jf\" (UID: \"9716763a-ff84-4735-b478-deb614be27c4\") " pod="openshift-multus/multus-additional-cni-plugins-9s7jf"
Sep 16 18:40:27.074196 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.072865    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f76a1a1e-3387-4aae-8964-63c913eb270a-host-cni-netd\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.074196 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.072906    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/66e76bc5-f7b3-4952-b67c-0fb014f21ecb-cnibin\") pod \"multus-wbl55\" (UID: \"66e76bc5-f7b3-4952-b67c-0fb014f21ecb\") " pod="openshift-multus/multus-wbl55"
Sep 16 18:40:27.074196 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.072943    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/79f801e9-77ff-4d5d-8d54-ffb25be9dd3a-registration-dir\") pod \"aws-ebs-csi-driver-node-kbvbq\" (UID: \"79f801e9-77ff-4d5d-8d54-ffb25be9dd3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbvbq"
Sep 16 18:40:27.074196 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.072979    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7fe5b556-37c9-40c7-a09b-5c6116b43e6f-serviceca\") pod \"node-ca-6298x\" (UID: \"7fe5b556-37c9-40c7-a09b-5c6116b43e6f\") " pod="openshift-image-registry/node-ca-6298x"
Sep 16 18:40:27.074196 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.072968    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9716763a-ff84-4735-b478-deb614be27c4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9s7jf\" (UID: \"9716763a-ff84-4735-b478-deb614be27c4\") " pod="openshift-multus/multus-additional-cni-plugins-9s7jf"
Sep 16 18:40:27.074196 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.073012    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/66e76bc5-f7b3-4952-b67c-0fb014f21ecb-cnibin\") pod \"multus-wbl55\" (UID: \"66e76bc5-f7b3-4952-b67c-0fb014f21ecb\") " pod="openshift-multus/multus-wbl55"
Sep 16 18:40:27.074196 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.073070    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2807fdd0-b58d-446d-ac9b-84a50d9b9ac6-etc-tuned\") pod \"tuned-6kjpc\" (UID: \"2807fdd0-b58d-446d-ac9b-84a50d9b9ac6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kjpc"
Sep 16 18:40:27.074196 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.073079    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9716763a-ff84-4735-b478-deb614be27c4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9s7jf\" (UID: \"9716763a-ff84-4735-b478-deb614be27c4\") " pod="openshift-multus/multus-additional-cni-plugins-9s7jf"
Sep 16 18:40:27.074196 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.073080    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9716763a-ff84-4735-b478-deb614be27c4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9s7jf\" (UID: \"9716763a-ff84-4735-b478-deb614be27c4\") " pod="openshift-multus/multus-additional-cni-plugins-9s7jf"
Sep 16 18:40:27.074196 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.073101    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f76a1a1e-3387-4aae-8964-63c913eb270a-run-ovn\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.074196 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.073071    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/66e76bc5-f7b3-4952-b67c-0fb014f21ecb-multus-daemon-config\") pod \"multus-wbl55\" (UID: \"66e76bc5-f7b3-4952-b67c-0fb014f21ecb\") " pod="openshift-multus/multus-wbl55"
Sep 16 18:40:27.074196 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.073116    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/79f801e9-77ff-4d5d-8d54-ffb25be9dd3a-registration-dir\") pod \"aws-ebs-csi-driver-node-kbvbq\" (UID: \"79f801e9-77ff-4d5d-8d54-ffb25be9dd3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbvbq"
Sep 16 18:40:27.074196 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.073214    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f76a1a1e-3387-4aae-8964-63c913eb270a-ovnkube-config\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.074196 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.073258    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f76a1a1e-3387-4aae-8964-63c913eb270a-ovnkube-script-lib\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.075024 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.073287    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/66e76bc5-f7b3-4952-b67c-0fb014f21ecb-multus-socket-dir-parent\") pod \"multus-wbl55\" (UID: \"66e76bc5-f7b3-4952-b67c-0fb014f21ecb\") " pod="openshift-multus/multus-wbl55"
Sep 16 18:40:27.075024 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.073323    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t4pz\" (UniqueName: \"kubernetes.io/projected/2807fdd0-b58d-446d-ac9b-84a50d9b9ac6-kube-api-access-5t4pz\") pod \"tuned-6kjpc\" (UID: \"2807fdd0-b58d-446d-ac9b-84a50d9b9ac6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kjpc"
Sep 16 18:40:27.075024 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.073353    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/66e76bc5-f7b3-4952-b67c-0fb014f21ecb-multus-socket-dir-parent\") pod \"multus-wbl55\" (UID: \"66e76bc5-f7b3-4952-b67c-0fb014f21ecb\") " pod="openshift-multus/multus-wbl55"
Sep 16 18:40:27.075024 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.073360    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f76a1a1e-3387-4aae-8964-63c913eb270a-run-systemd\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.075024 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.073393    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f76a1a1e-3387-4aae-8964-63c913eb270a-host-cni-bin\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.075024 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.073452    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9716763a-ff84-4735-b478-deb614be27c4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9s7jf\" (UID: \"9716763a-ff84-4735-b478-deb614be27c4\") " pod="openshift-multus/multus-additional-cni-plugins-9s7jf"
Sep 16 18:40:27.075024 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.073534    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/79f801e9-77ff-4d5d-8d54-ffb25be9dd3a-sys-fs\") pod \"aws-ebs-csi-driver-node-kbvbq\" (UID: \"79f801e9-77ff-4d5d-8d54-ffb25be9dd3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbvbq"
Sep 16 18:40:27.075024 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.073572    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9716763a-ff84-4735-b478-deb614be27c4-cni-binary-copy\") pod \"multus-additional-cni-plugins-9s7jf\" (UID: \"9716763a-ff84-4735-b478-deb614be27c4\") " pod="openshift-multus/multus-additional-cni-plugins-9s7jf"
Sep 16 18:40:27.075024 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.073597    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2807fdd0-b58d-446d-ac9b-84a50d9b9ac6-sys\") pod \"tuned-6kjpc\" (UID: \"2807fdd0-b58d-446d-ac9b-84a50d9b9ac6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kjpc"
Sep 16 18:40:27.075024 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.073614    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f76a1a1e-3387-4aae-8964-63c913eb270a-host-slash\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.075024 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.073627    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2807fdd0-b58d-446d-ac9b-84a50d9b9ac6-host\") pod \"tuned-6kjpc\" (UID: \"2807fdd0-b58d-446d-ac9b-84a50d9b9ac6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kjpc"
Sep 16 18:40:27.075024 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.073636    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/79f801e9-77ff-4d5d-8d54-ffb25be9dd3a-sys-fs\") pod \"aws-ebs-csi-driver-node-kbvbq\" (UID: \"79f801e9-77ff-4d5d-8d54-ffb25be9dd3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbvbq"
Sep 16 18:40:27.075024 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.073640    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7fe5b556-37c9-40c7-a09b-5c6116b43e6f-host\") pod \"node-ca-6298x\" (UID: \"7fe5b556-37c9-40c7-a09b-5c6116b43e6f\") " pod="openshift-image-registry/node-ca-6298x"
Sep 16 18:40:27.075024 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.073673    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7fe5b556-37c9-40c7-a09b-5c6116b43e6f-host\") pod \"node-ca-6298x\" (UID: \"7fe5b556-37c9-40c7-a09b-5c6116b43e6f\") " pod="openshift-image-registry/node-ca-6298x"
Sep 16 18:40:27.075024 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.073688    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j227\" (UniqueName: \"kubernetes.io/projected/79f801e9-77ff-4d5d-8d54-ffb25be9dd3a-kube-api-access-7j227\") pod \"aws-ebs-csi-driver-node-kbvbq\" (UID: \"79f801e9-77ff-4d5d-8d54-ffb25be9dd3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbvbq"
Sep 16 18:40:27.075024 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.073727    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2807fdd0-b58d-446d-ac9b-84a50d9b9ac6-sys\") pod \"tuned-6kjpc\" (UID: \"2807fdd0-b58d-446d-ac9b-84a50d9b9ac6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kjpc"
Sep 16 18:40:27.075024 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.073717    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8c42d544-c24c-4b4b-b88d-13cc74d9fa87-kubelet-config\") pod \"global-pull-secret-syncer-8ktbc\" (UID: \"8c42d544-c24c-4b4b-b88d-13cc74d9fa87\") " pod="kube-system/global-pull-secret-syncer-8ktbc"
Sep 16 18:40:27.075772 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.073772    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c42d544-c24c-4b4b-b88d-13cc74d9fa87-original-pull-secret\") pod \"global-pull-secret-syncer-8ktbc\" (UID: \"8c42d544-c24c-4b4b-b88d-13cc74d9fa87\") " pod="kube-system/global-pull-secret-syncer-8ktbc"
Sep 16 18:40:27.075772 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.073800    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2807fdd0-b58d-446d-ac9b-84a50d9b9ac6-tmp\") pod \"tuned-6kjpc\" (UID: \"2807fdd0-b58d-446d-ac9b-84a50d9b9ac6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kjpc"
Sep 16 18:40:27.075772 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.073794    2586 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 18:40:27.075772 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.073830    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b58ed079-b715-420e-a11f-2023c52e1321-tmp-dir\") pod \"node-resolver-cc7hj\" (UID: \"b58ed079-b715-420e-a11f-2023c52e1321\") " pod="openshift-dns/node-resolver-cc7hj"
Sep 16 18:40:27.075772 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.073863    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f76a1a1e-3387-4aae-8964-63c913eb270a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.075772 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.073887    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/66e76bc5-f7b3-4952-b67c-0fb014f21ecb-host-run-netns\") pod \"multus-wbl55\" (UID: \"66e76bc5-f7b3-4952-b67c-0fb014f21ecb\") " pod="openshift-multus/multus-wbl55"
Sep 16 18:40:27.075772 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.073912    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66e76bc5-f7b3-4952-b67c-0fb014f21ecb-etc-kubernetes\") pod \"multus-wbl55\" (UID: \"66e76bc5-f7b3-4952-b67c-0fb014f21ecb\") " pod="openshift-multus/multus-wbl55"
Sep 16 18:40:27.075772 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.073936    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e8169e6-6afe-417b-90f0-5a55bf294956-metrics-certs\") pod \"network-metrics-daemon-9vpbt\" (UID: \"1e8169e6-6afe-417b-90f0-5a55bf294956\") " pod="openshift-multus/network-metrics-daemon-9vpbt"
Sep 16 18:40:27.075772 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.073938    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2807fdd0-b58d-446d-ac9b-84a50d9b9ac6-host\") pod \"tuned-6kjpc\" (UID: \"2807fdd0-b58d-446d-ac9b-84a50d9b9ac6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kjpc"
Sep 16 18:40:27.075772 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.073959    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f76a1a1e-3387-4aae-8964-63c913eb270a-env-overrides\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.075772 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.074018    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjb4h\" (UniqueName: \"kubernetes.io/projected/f76a1a1e-3387-4aae-8964-63c913eb270a-kube-api-access-tjb4h\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.075772 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.074059    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9716763a-ff84-4735-b478-deb614be27c4-cni-binary-copy\") pod \"multus-additional-cni-plugins-9s7jf\" (UID: \"9716763a-ff84-4735-b478-deb614be27c4\") " pod="openshift-multus/multus-additional-cni-plugins-9s7jf"
Sep 16 18:40:27.075772 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:27.074104    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:27.075772 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.074114    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6f68d3ed-0908-4d8a-a879-5b002a923ca9-konnectivity-ca\") pod \"konnectivity-agent-7n8dt\" (UID: \"6f68d3ed-0908-4d8a-a879-5b002a923ca9\") " pod="kube-system/konnectivity-agent-7n8dt"
Sep 16 18:40:27.075772 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.074162    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/21e31a4b-fd39-4863-ada4-4c87f208a134-iptables-alerter-script\") pod \"iptables-alerter-vktwg\" (UID: \"21e31a4b-fd39-4863-ada4-4c87f208a134\") " pod="openshift-network-operator/iptables-alerter-vktwg"
Sep 16 18:40:27.075772 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:27.074196    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c42d544-c24c-4b4b-b88d-13cc74d9fa87-original-pull-secret podName:8c42d544-c24c-4b4b-b88d-13cc74d9fa87 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:27.574154808 +0000 UTC m=+3.018016380 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c42d544-c24c-4b4b-b88d-13cc74d9fa87-original-pull-secret") pod "global-pull-secret-syncer-8ktbc" (UID: "8c42d544-c24c-4b4b-b88d-13cc74d9fa87") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:27.075772 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.074251    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnzbm\" (UniqueName: \"kubernetes.io/projected/21e31a4b-fd39-4863-ada4-4c87f208a134-kube-api-access-cnzbm\") pod \"iptables-alerter-vktwg\" (UID: \"21e31a4b-fd39-4863-ada4-4c87f208a134\") " pod="openshift-network-operator/iptables-alerter-vktwg"
Sep 16 18:40:27.076549 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.074268    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b58ed079-b715-420e-a11f-2023c52e1321-tmp-dir\") pod \"node-resolver-cc7hj\" (UID: \"b58ed079-b715-420e-a11f-2023c52e1321\") " pod="openshift-dns/node-resolver-cc7hj"
Sep 16 18:40:27.076549 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:27.074283    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:27.076549 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.074333    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2807fdd0-b58d-446d-ac9b-84a50d9b9ac6-etc-sysctl-conf\") pod \"tuned-6kjpc\" (UID: \"2807fdd0-b58d-446d-ac9b-84a50d9b9ac6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kjpc"
Sep 16 18:40:27.076549 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.074349    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66e76bc5-f7b3-4952-b67c-0fb014f21ecb-etc-kubernetes\") pod \"multus-wbl55\" (UID: \"66e76bc5-f7b3-4952-b67c-0fb014f21ecb\") " pod="openshift-multus/multus-wbl55"
Sep 16 18:40:27.076549 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.074364    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b58ed079-b715-420e-a11f-2023c52e1321-hosts-file\") pod \"node-resolver-cc7hj\" (UID: \"b58ed079-b715-420e-a11f-2023c52e1321\") " pod="openshift-dns/node-resolver-cc7hj"
Sep 16 18:40:27.076549 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.074408    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b58ed079-b715-420e-a11f-2023c52e1321-hosts-file\") pod \"node-resolver-cc7hj\" (UID: \"b58ed079-b715-420e-a11f-2023c52e1321\") " pod="openshift-dns/node-resolver-cc7hj"
Sep 16 18:40:27.076549 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.074428    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66e76bc5-f7b3-4952-b67c-0fb014f21ecb-multus-cni-dir\") pod \"multus-wbl55\" (UID: \"66e76bc5-f7b3-4952-b67c-0fb014f21ecb\") " pod="openshift-multus/multus-wbl55"
Sep 16 18:40:27.076549 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.074464    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/66e76bc5-f7b3-4952-b67c-0fb014f21ecb-host-run-netns\") pod \"multus-wbl55\" (UID: \"66e76bc5-f7b3-4952-b67c-0fb014f21ecb\") " pod="openshift-multus/multus-wbl55"
Sep 16 18:40:27.076549 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.074507    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2807fdd0-b58d-446d-ac9b-84a50d9b9ac6-etc-modprobe-d\") pod \"tuned-6kjpc\" (UID: \"2807fdd0-b58d-446d-ac9b-84a50d9b9ac6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kjpc"
Sep 16 18:40:27.076549 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.074532    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f76a1a1e-3387-4aae-8964-63c913eb270a-run-openvswitch\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.076549 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.074559    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkvjt\" (UniqueName: \"kubernetes.io/projected/9716763a-ff84-4735-b478-deb614be27c4-kube-api-access-gkvjt\") pod \"multus-additional-cni-plugins-9s7jf\" (UID: \"9716763a-ff84-4735-b478-deb614be27c4\") " pod="openshift-multus/multus-additional-cni-plugins-9s7jf"
Sep 16 18:40:27.076549 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.074604    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2807fdd0-b58d-446d-ac9b-84a50d9b9ac6-etc-sysconfig\") pod \"tuned-6kjpc\" (UID: \"2807fdd0-b58d-446d-ac9b-84a50d9b9ac6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kjpc"
Sep 16 18:40:27.076549 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.074674    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8c42d544-c24c-4b4b-b88d-13cc74d9fa87-kubelet-config\") pod \"global-pull-secret-syncer-8ktbc\" (UID: \"8c42d544-c24c-4b4b-b88d-13cc74d9fa87\") " pod="kube-system/global-pull-secret-syncer-8ktbc"
Sep 16 18:40:27.076549 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.074708    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/79f801e9-77ff-4d5d-8d54-ffb25be9dd3a-device-dir\") pod \"aws-ebs-csi-driver-node-kbvbq\" (UID: \"79f801e9-77ff-4d5d-8d54-ffb25be9dd3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbvbq"
Sep 16 18:40:27.076549 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.074712    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/21e31a4b-fd39-4863-ada4-4c87f208a134-iptables-alerter-script\") pod \"iptables-alerter-vktwg\" (UID: \"21e31a4b-fd39-4863-ada4-4c87f208a134\") " pod="openshift-network-operator/iptables-alerter-vktwg"
Sep 16 18:40:27.076549 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.074677    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2807fdd0-b58d-446d-ac9b-84a50d9b9ac6-etc-sysctl-conf\") pod \"tuned-6kjpc\" (UID: \"2807fdd0-b58d-446d-ac9b-84a50d9b9ac6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kjpc"
Sep 16 18:40:27.076549 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:27.074741    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e8169e6-6afe-417b-90f0-5a55bf294956-metrics-certs podName:1e8169e6-6afe-417b-90f0-5a55bf294956 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:27.574722358 +0000 UTC m=+3.018583929 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e8169e6-6afe-417b-90f0-5a55bf294956-metrics-certs") pod "network-metrics-daemon-9vpbt" (UID: "1e8169e6-6afe-417b-90f0-5a55bf294956") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:27.077281 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.074765    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2807fdd0-b58d-446d-ac9b-84a50d9b9ac6-etc-sysconfig\") pod \"tuned-6kjpc\" (UID: \"2807fdd0-b58d-446d-ac9b-84a50d9b9ac6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kjpc"
Sep 16 18:40:27.077281 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.074773    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/79f801e9-77ff-4d5d-8d54-ffb25be9dd3a-device-dir\") pod \"aws-ebs-csi-driver-node-kbvbq\" (UID: \"79f801e9-77ff-4d5d-8d54-ffb25be9dd3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbvbq"
Sep 16 18:40:27.077281 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.074786    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/66e76bc5-f7b3-4952-b67c-0fb014f21ecb-host-run-k8s-cni-cncf-io\") pod \"multus-wbl55\" (UID: \"66e76bc5-f7b3-4952-b67c-0fb014f21ecb\") " pod="openshift-multus/multus-wbl55"
Sep 16 18:40:27.077281 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.074815    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-564kw\" (UniqueName: \"kubernetes.io/projected/b58ed079-b715-420e-a11f-2023c52e1321-kube-api-access-564kw\") pod \"node-resolver-cc7hj\" (UID: \"b58ed079-b715-420e-a11f-2023c52e1321\") " pod="openshift-dns/node-resolver-cc7hj"
Sep 16 18:40:27.077281 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.074837    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/66e76bc5-f7b3-4952-b67c-0fb014f21ecb-host-run-k8s-cni-cncf-io\") pod \"multus-wbl55\" (UID: \"66e76bc5-f7b3-4952-b67c-0fb014f21ecb\") " pod="openshift-multus/multus-wbl55"
Sep 16 18:40:27.077281 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.074853    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4jzq\" (UniqueName: \"kubernetes.io/projected/cbd2c59d-9acb-4f04-8068-c0e2d13e5d41-kube-api-access-k4jzq\") pod \"network-check-target-kmqvn\" (UID: \"cbd2c59d-9acb-4f04-8068-c0e2d13e5d41\") " pod="openshift-network-diagnostics/network-check-target-kmqvn"
Sep 16 18:40:27.077281 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.074878    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/79f801e9-77ff-4d5d-8d54-ffb25be9dd3a-etc-selinux\") pod \"aws-ebs-csi-driver-node-kbvbq\" (UID: \"79f801e9-77ff-4d5d-8d54-ffb25be9dd3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbvbq"
Sep 16 18:40:27.077281 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.074888    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2807fdd0-b58d-446d-ac9b-84a50d9b9ac6-etc-modprobe-d\") pod \"tuned-6kjpc\" (UID: \"2807fdd0-b58d-446d-ac9b-84a50d9b9ac6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kjpc"
Sep 16 18:40:27.077281 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.074942    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/66e76bc5-f7b3-4952-b67c-0fb014f21ecb-multus-cni-dir\") pod \"multus-wbl55\" (UID: \"66e76bc5-f7b3-4952-b67c-0fb014f21ecb\") " pod="openshift-multus/multus-wbl55"
Sep 16 18:40:27.077281 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.075002    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/79f801e9-77ff-4d5d-8d54-ffb25be9dd3a-etc-selinux\") pod \"aws-ebs-csi-driver-node-kbvbq\" (UID: \"79f801e9-77ff-4d5d-8d54-ffb25be9dd3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbvbq"
Sep 16 18:40:27.077922 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.077899    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2807fdd0-b58d-446d-ac9b-84a50d9b9ac6-etc-tuned\") pod \"tuned-6kjpc\" (UID: \"2807fdd0-b58d-446d-ac9b-84a50d9b9ac6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kjpc"
Sep 16 18:40:27.077922 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.077918    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2807fdd0-b58d-446d-ac9b-84a50d9b9ac6-tmp\") pod \"tuned-6kjpc\" (UID: \"2807fdd0-b58d-446d-ac9b-84a50d9b9ac6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kjpc"
Sep 16 18:40:27.093698 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.093672    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scxfj\" (UniqueName: \"kubernetes.io/projected/66e76bc5-f7b3-4952-b67c-0fb014f21ecb-kube-api-access-scxfj\") pod \"multus-wbl55\" (UID: \"66e76bc5-f7b3-4952-b67c-0fb014f21ecb\") " pod="openshift-multus/multus-wbl55"
Sep 16 18:40:27.105642 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:27.105601    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:27.105642 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:27.105629    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:27.105642 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:27.105643    2586 projected.go:196] Error preparing data for projected volume kube-api-access-k4jzq for pod openshift-network-diagnostics/network-check-target-kmqvn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:27.105894 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:27.105714    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cbd2c59d-9acb-4f04-8068-c0e2d13e5d41-kube-api-access-k4jzq podName:cbd2c59d-9acb-4f04-8068-c0e2d13e5d41 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:27.605694952 +0000 UTC m=+3.049556528 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-k4jzq" (UniqueName: "kubernetes.io/projected/cbd2c59d-9acb-4f04-8068-c0e2d13e5d41-kube-api-access-k4jzq") pod "network-check-target-kmqvn" (UID: "cbd2c59d-9acb-4f04-8068-c0e2d13e5d41") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:27.108277 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.108213    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwdgz\" (UniqueName: \"kubernetes.io/projected/1e8169e6-6afe-417b-90f0-5a55bf294956-kube-api-access-lwdgz\") pod \"network-metrics-daemon-9vpbt\" (UID: \"1e8169e6-6afe-417b-90f0-5a55bf294956\") " pod="openshift-multus/network-metrics-daemon-9vpbt"
Sep 16 18:40:27.108465 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.108440    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-564kw\" (UniqueName: \"kubernetes.io/projected/b58ed079-b715-420e-a11f-2023c52e1321-kube-api-access-564kw\") pod \"node-resolver-cc7hj\" (UID: \"b58ed079-b715-420e-a11f-2023c52e1321\") " pod="openshift-dns/node-resolver-cc7hj"
Sep 16 18:40:27.109154 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.109038    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnzbm\" (UniqueName: \"kubernetes.io/projected/21e31a4b-fd39-4863-ada4-4c87f208a134-kube-api-access-cnzbm\") pod \"iptables-alerter-vktwg\" (UID: \"21e31a4b-fd39-4863-ada4-4c87f208a134\") " pod="openshift-network-operator/iptables-alerter-vktwg"
Sep 16 18:40:27.109154 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.109094    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkvjt\" (UniqueName: \"kubernetes.io/projected/9716763a-ff84-4735-b478-deb614be27c4-kube-api-access-gkvjt\") pod \"multus-additional-cni-plugins-9s7jf\" (UID: \"9716763a-ff84-4735-b478-deb614be27c4\") " pod="openshift-multus/multus-additional-cni-plugins-9s7jf"
Sep 16 18:40:27.110531 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.110507    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67cdx\" (UniqueName: \"kubernetes.io/projected/7fe5b556-37c9-40c7-a09b-5c6116b43e6f-kube-api-access-67cdx\") pod \"node-ca-6298x\" (UID: \"7fe5b556-37c9-40c7-a09b-5c6116b43e6f\") " pod="openshift-image-registry/node-ca-6298x"
Sep 16 18:40:27.110640 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.110537    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j227\" (UniqueName: \"kubernetes.io/projected/79f801e9-77ff-4d5d-8d54-ffb25be9dd3a-kube-api-access-7j227\") pod \"aws-ebs-csi-driver-node-kbvbq\" (UID: \"79f801e9-77ff-4d5d-8d54-ffb25be9dd3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbvbq"
Sep 16 18:40:27.110849 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.110830    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t4pz\" (UniqueName: \"kubernetes.io/projected/2807fdd0-b58d-446d-ac9b-84a50d9b9ac6-kube-api-access-5t4pz\") pod \"tuned-6kjpc\" (UID: \"2807fdd0-b58d-446d-ac9b-84a50d9b9ac6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kjpc"
Sep 16 18:40:27.175623 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.175585    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f76a1a1e-3387-4aae-8964-63c913eb270a-ovn-node-metrics-cert\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.175623 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.175627    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f76a1a1e-3387-4aae-8964-63c913eb270a-host-cni-netd\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.175843 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.175645    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f76a1a1e-3387-4aae-8964-63c913eb270a-run-ovn\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.175843 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.175661    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f76a1a1e-3387-4aae-8964-63c913eb270a-ovnkube-config\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.175843 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.175680    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f76a1a1e-3387-4aae-8964-63c913eb270a-ovnkube-script-lib\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.175843 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.175713    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f76a1a1e-3387-4aae-8964-63c913eb270a-run-systemd\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.175843 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.175725    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f76a1a1e-3387-4aae-8964-63c913eb270a-run-ovn\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.175843 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.175742    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f76a1a1e-3387-4aae-8964-63c913eb270a-host-cni-bin\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.175843 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.175722    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f76a1a1e-3387-4aae-8964-63c913eb270a-host-cni-netd\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.175843 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.175768    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f76a1a1e-3387-4aae-8964-63c913eb270a-run-systemd\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.175843 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.175792    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f76a1a1e-3387-4aae-8964-63c913eb270a-host-cni-bin\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.175843 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.175818    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f76a1a1e-3387-4aae-8964-63c913eb270a-host-slash\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.176332 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.175865    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f76a1a1e-3387-4aae-8964-63c913eb270a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.176332 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.175892    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f76a1a1e-3387-4aae-8964-63c913eb270a-host-slash\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.176332 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.175899    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f76a1a1e-3387-4aae-8964-63c913eb270a-env-overrides\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.176332 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.175932    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjb4h\" (UniqueName: \"kubernetes.io/projected/f76a1a1e-3387-4aae-8964-63c913eb270a-kube-api-access-tjb4h\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.176332 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.175958    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6f68d3ed-0908-4d8a-a879-5b002a923ca9-konnectivity-ca\") pod \"konnectivity-agent-7n8dt\" (UID: \"6f68d3ed-0908-4d8a-a879-5b002a923ca9\") " pod="kube-system/konnectivity-agent-7n8dt"
Sep 16 18:40:27.176332 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.175941    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f76a1a1e-3387-4aae-8964-63c913eb270a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.176332 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.175990    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f76a1a1e-3387-4aae-8964-63c913eb270a-run-openvswitch\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.176332 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.176040    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f76a1a1e-3387-4aae-8964-63c913eb270a-var-lib-openvswitch\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.176332 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.176066    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f76a1a1e-3387-4aae-8964-63c913eb270a-etc-openvswitch\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.176332 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.176086    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f76a1a1e-3387-4aae-8964-63c913eb270a-node-log\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.176332 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.176108    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f76a1a1e-3387-4aae-8964-63c913eb270a-log-socket\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.176332 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.176133    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f76a1a1e-3387-4aae-8964-63c913eb270a-host-run-netns\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.176332 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.176160    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f76a1a1e-3387-4aae-8964-63c913eb270a-systemd-units\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.176332 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.176184    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f76a1a1e-3387-4aae-8964-63c913eb270a-host-run-ovn-kubernetes\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.176332 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.176203    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6f68d3ed-0908-4d8a-a879-5b002a923ca9-agent-certs\") pod \"konnectivity-agent-7n8dt\" (UID: \"6f68d3ed-0908-4d8a-a879-5b002a923ca9\") " pod="kube-system/konnectivity-agent-7n8dt"
Sep 16 18:40:27.176332 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.176209    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f76a1a1e-3387-4aae-8964-63c913eb270a-var-lib-openvswitch\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.176332 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.176243    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f76a1a1e-3387-4aae-8964-63c913eb270a-host-kubelet\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.176895 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.176282    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f76a1a1e-3387-4aae-8964-63c913eb270a-host-kubelet\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.176895 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.176296    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f76a1a1e-3387-4aae-8964-63c913eb270a-host-run-netns\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.176895 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.176325    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f76a1a1e-3387-4aae-8964-63c913eb270a-env-overrides\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.176895 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.176347    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f76a1a1e-3387-4aae-8964-63c913eb270a-host-run-ovn-kubernetes\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.176895 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.176357    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f76a1a1e-3387-4aae-8964-63c913eb270a-etc-openvswitch\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.176895 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.176351    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f76a1a1e-3387-4aae-8964-63c913eb270a-ovnkube-config\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.176895 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.176382    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f76a1a1e-3387-4aae-8964-63c913eb270a-ovnkube-script-lib\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.176895 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.176424    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f76a1a1e-3387-4aae-8964-63c913eb270a-run-openvswitch\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.176895 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.176424    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f76a1a1e-3387-4aae-8964-63c913eb270a-systemd-units\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.176895 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.176455    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f76a1a1e-3387-4aae-8964-63c913eb270a-node-log\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.176895 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.176465    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f76a1a1e-3387-4aae-8964-63c913eb270a-log-socket\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.177647 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.177623    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6f68d3ed-0908-4d8a-a879-5b002a923ca9-konnectivity-ca\") pod \"konnectivity-agent-7n8dt\" (UID: \"6f68d3ed-0908-4d8a-a879-5b002a923ca9\") " pod="kube-system/konnectivity-agent-7n8dt"
Sep 16 18:40:27.178311 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.178292    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f76a1a1e-3387-4aae-8964-63c913eb270a-ovn-node-metrics-cert\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.178929 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.178911    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6f68d3ed-0908-4d8a-a879-5b002a923ca9-agent-certs\") pod \"konnectivity-agent-7n8dt\" (UID: \"6f68d3ed-0908-4d8a-a879-5b002a923ca9\") " pod="kube-system/konnectivity-agent-7n8dt"
Sep 16 18:40:27.188318 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.188254    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjb4h\" (UniqueName: \"kubernetes.io/projected/f76a1a1e-3387-4aae-8964-63c913eb270a-kube-api-access-tjb4h\") pod \"ovnkube-node-wxhg2\" (UID: \"f76a1a1e-3387-4aae-8964-63c913eb270a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.260714 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.260679    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6298x"
Sep 16 18:40:27.268620 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.268600    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbvbq"
Sep 16 18:40:27.278259 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.278224    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9s7jf"
Sep 16 18:40:27.284391 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.284370    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vktwg"
Sep 16 18:40:27.291295 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.291276    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6kjpc"
Sep 16 18:40:27.298014 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.297994    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cc7hj"
Sep 16 18:40:27.304663 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.304647    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wbl55"
Sep 16 18:40:27.313578 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.313558    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:27.318326 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.318308    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7n8dt"
Sep 16 18:40:27.579152 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.579071    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c42d544-c24c-4b4b-b88d-13cc74d9fa87-original-pull-secret\") pod \"global-pull-secret-syncer-8ktbc\" (UID: \"8c42d544-c24c-4b4b-b88d-13cc74d9fa87\") " pod="kube-system/global-pull-secret-syncer-8ktbc"
Sep 16 18:40:27.579152 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.579114    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e8169e6-6afe-417b-90f0-5a55bf294956-metrics-certs\") pod \"network-metrics-daemon-9vpbt\" (UID: \"1e8169e6-6afe-417b-90f0-5a55bf294956\") " pod="openshift-multus/network-metrics-daemon-9vpbt"
Sep 16 18:40:27.579379 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:27.579252    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:27.579379 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:27.579343    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c42d544-c24c-4b4b-b88d-13cc74d9fa87-original-pull-secret podName:8c42d544-c24c-4b4b-b88d-13cc74d9fa87 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:28.579323251 +0000 UTC m=+4.023184828 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c42d544-c24c-4b4b-b88d-13cc74d9fa87-original-pull-secret") pod "global-pull-secret-syncer-8ktbc" (UID: "8c42d544-c24c-4b4b-b88d-13cc74d9fa87") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:27.579379 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:27.579252    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:27.579495 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:27.579424    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e8169e6-6afe-417b-90f0-5a55bf294956-metrics-certs podName:1e8169e6-6afe-417b-90f0-5a55bf294956 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:28.579403136 +0000 UTC m=+4.023264691 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e8169e6-6afe-417b-90f0-5a55bf294956-metrics-certs") pod "network-metrics-daemon-9vpbt" (UID: "1e8169e6-6afe-417b-90f0-5a55bf294956") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:27.679936 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:27.679906    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4jzq\" (UniqueName: \"kubernetes.io/projected/cbd2c59d-9acb-4f04-8068-c0e2d13e5d41-kube-api-access-k4jzq\") pod \"network-check-target-kmqvn\" (UID: \"cbd2c59d-9acb-4f04-8068-c0e2d13e5d41\") " pod="openshift-network-diagnostics/network-check-target-kmqvn"
Sep 16 18:40:27.680109 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:27.680090    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:27.680165 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:27.680120    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:27.680165 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:27.680134    2586 projected.go:196] Error preparing data for projected volume kube-api-access-k4jzq for pod openshift-network-diagnostics/network-check-target-kmqvn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:27.680242 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:27.680198    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cbd2c59d-9acb-4f04-8068-c0e2d13e5d41-kube-api-access-k4jzq podName:cbd2c59d-9acb-4f04-8068-c0e2d13e5d41 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:28.680178115 +0000 UTC m=+4.124039684 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-k4jzq" (UniqueName: "kubernetes.io/projected/cbd2c59d-9acb-4f04-8068-c0e2d13e5d41-kube-api-access-k4jzq") pod "network-check-target-kmqvn" (UID: "cbd2c59d-9acb-4f04-8068-c0e2d13e5d41") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:27.825474 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:27.825443    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb58ed079_b715_420e_a11f_2023c52e1321.slice/crio-4d1044a4f5ce9fceb1e80f78fbbfe956608a48852724d9c0c12b5d2494db7b50 WatchSource:0}: Error finding container 4d1044a4f5ce9fceb1e80f78fbbfe956608a48852724d9c0c12b5d2494db7b50: Status 404 returned error can't find the container with id 4d1044a4f5ce9fceb1e80f78fbbfe956608a48852724d9c0c12b5d2494db7b50
Sep 16 18:40:27.826066 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:27.825911    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf76a1a1e_3387_4aae_8964_63c913eb270a.slice/crio-209a8c8ec74dff79312cbb0bf8cecce771f3ab5db06db1f472ada7acce2259f6 WatchSource:0}: Error finding container 209a8c8ec74dff79312cbb0bf8cecce771f3ab5db06db1f472ada7acce2259f6: Status 404 returned error can't find the container with id 209a8c8ec74dff79312cbb0bf8cecce771f3ab5db06db1f472ada7acce2259f6
Sep 16 18:40:27.833181 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:27.833157    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66e76bc5_f7b3_4952_b67c_0fb014f21ecb.slice/crio-3afd5bb101a82908d353cfbcab05fc86d351fae8fa60e2232ae69c82b3b2bd29 WatchSource:0}: Error finding container 3afd5bb101a82908d353cfbcab05fc86d351fae8fa60e2232ae69c82b3b2bd29: Status 404 returned error can't find the container with id 3afd5bb101a82908d353cfbcab05fc86d351fae8fa60e2232ae69c82b3b2bd29
Sep 16 18:40:27.834852 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:27.834828    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2807fdd0_b58d_446d_ac9b_84a50d9b9ac6.slice/crio-f53b29363ffda66abfe68ef97c50ea9924c759072a7bef806fe25ef3d7891087 WatchSource:0}: Error finding container f53b29363ffda66abfe68ef97c50ea9924c759072a7bef806fe25ef3d7891087: Status 404 returned error can't find the container with id f53b29363ffda66abfe68ef97c50ea9924c759072a7bef806fe25ef3d7891087
Sep 16 18:40:27.835603 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:27.835581    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f68d3ed_0908_4d8a_a879_5b002a923ca9.slice/crio-aa2d15a990b587f068046fe13ea4404055e21e225e254c5ddfbd23cb9193c6fd WatchSource:0}: Error finding container aa2d15a990b587f068046fe13ea4404055e21e225e254c5ddfbd23cb9193c6fd: Status 404 returned error can't find the container with id aa2d15a990b587f068046fe13ea4404055e21e225e254c5ddfbd23cb9193c6fd
Sep 16 18:40:27.836725 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:27.836705    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fe5b556_37c9_40c7_a09b_5c6116b43e6f.slice/crio-98148a8709dde168732864dbc57083567e15bf7a819111c43f36cd0c74a899d6 WatchSource:0}: Error finding container 98148a8709dde168732864dbc57083567e15bf7a819111c43f36cd0c74a899d6: Status 404 returned error can't find the container with id 98148a8709dde168732864dbc57083567e15bf7a819111c43f36cd0c74a899d6
Sep 16 18:40:27.837284 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:27.837245    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9716763a_ff84_4735_b478_deb614be27c4.slice/crio-aa3a76c75fcc16275e32c39cbc82e5859e99b3f7b95abc551b2d9aa752fe81ea WatchSource:0}: Error finding container aa3a76c75fcc16275e32c39cbc82e5859e99b3f7b95abc551b2d9aa752fe81ea: Status 404 returned error can't find the container with id aa3a76c75fcc16275e32c39cbc82e5859e99b3f7b95abc551b2d9aa752fe81ea
Sep 16 18:40:27.839695 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:27.839671    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79f801e9_77ff_4d5d_8d54_ffb25be9dd3a.slice/crio-09a2353ecf8f3cb8ae20219b6f2fa98ec9c35ed70a37f49e45a9a9bd7c56b0b2 WatchSource:0}: Error finding container 09a2353ecf8f3cb8ae20219b6f2fa98ec9c35ed70a37f49e45a9a9bd7c56b0b2: Status 404 returned error can't find the container with id 09a2353ecf8f3cb8ae20219b6f2fa98ec9c35ed70a37f49e45a9a9bd7c56b0b2
Sep 16 18:40:28.021872 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:28.021800    2586 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:35:27 +0000 UTC" deadline="2028-04-01 11:17:29.457830105 +0000 UTC"
Sep 16 18:40:28.021872 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:28.021866    2586 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13504h37m1.435967675s"
Sep 16 18:40:28.086208 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:28.086157    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ktbc"
Sep 16 18:40:28.086339 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:28.086279    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ktbc" podUID="8c42d544-c24c-4b4b-b88d-13cc74d9fa87"
Sep 16 18:40:28.096441 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:28.096410    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-190.ec2.internal" event={"ID":"fd4c464e8a7b0afa02a1b06875f0d90d","Type":"ContainerStarted","Data":"020d65fb4da6ee202776168ef89da67042b3bcd571454a8b15a29df44c4ea4a9"}
Sep 16 18:40:28.097610 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:28.097582    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbvbq" event={"ID":"79f801e9-77ff-4d5d-8d54-ffb25be9dd3a","Type":"ContainerStarted","Data":"09a2353ecf8f3cb8ae20219b6f2fa98ec9c35ed70a37f49e45a9a9bd7c56b0b2"}
Sep 16 18:40:28.098607 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:28.098589    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7n8dt" event={"ID":"6f68d3ed-0908-4d8a-a879-5b002a923ca9","Type":"ContainerStarted","Data":"aa2d15a990b587f068046fe13ea4404055e21e225e254c5ddfbd23cb9193c6fd"}
Sep 16 18:40:28.099854 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:28.099758    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6kjpc" event={"ID":"2807fdd0-b58d-446d-ac9b-84a50d9b9ac6","Type":"ContainerStarted","Data":"f53b29363ffda66abfe68ef97c50ea9924c759072a7bef806fe25ef3d7891087"}
Sep 16 18:40:28.101099 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:28.101073    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wbl55" event={"ID":"66e76bc5-f7b3-4952-b67c-0fb014f21ecb","Type":"ContainerStarted","Data":"3afd5bb101a82908d353cfbcab05fc86d351fae8fa60e2232ae69c82b3b2bd29"}
Sep 16 18:40:28.102281 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:28.102260    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vktwg" event={"ID":"21e31a4b-fd39-4863-ada4-4c87f208a134","Type":"ContainerStarted","Data":"20dc569995cf9aee64dfd509646b3fae9838fe94e74b0a8c7e3337adad58f872"}
Sep 16 18:40:28.103528 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:28.103496    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2" event={"ID":"f76a1a1e-3387-4aae-8964-63c913eb270a","Type":"ContainerStarted","Data":"209a8c8ec74dff79312cbb0bf8cecce771f3ab5db06db1f472ada7acce2259f6"}
Sep 16 18:40:28.104683 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:28.104664    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cc7hj" event={"ID":"b58ed079-b715-420e-a11f-2023c52e1321","Type":"ContainerStarted","Data":"4d1044a4f5ce9fceb1e80f78fbbfe956608a48852724d9c0c12b5d2494db7b50"}
Sep 16 18:40:28.106312 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:28.106253    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-190.ec2.internal" event={"ID":"aacec50d8c91383d21cd8b8f8d4aaf7a","Type":"ContainerStarted","Data":"04f7b7aa2ae68e90f7e317a9129a8fb69483de2d1f6202aaeef8e9d5a331d18d"}
Sep 16 18:40:28.110041 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:28.110016    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9s7jf" event={"ID":"9716763a-ff84-4735-b478-deb614be27c4","Type":"ContainerStarted","Data":"aa3a76c75fcc16275e32c39cbc82e5859e99b3f7b95abc551b2d9aa752fe81ea"}
Sep 16 18:40:28.111611 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:28.111593    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6298x" event={"ID":"7fe5b556-37c9-40c7-a09b-5c6116b43e6f","Type":"ContainerStarted","Data":"98148a8709dde168732864dbc57083567e15bf7a819111c43f36cd0c74a899d6"}
Sep 16 18:40:28.134402 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:28.134359    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-190.ec2.internal" podStartSLOduration=2.134347541 podStartE2EDuration="2.134347541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:28.134033801 +0000 UTC m=+3.577895377" watchObservedRunningTime="2026-09-16 18:40:28.134347541 +0000 UTC m=+3.578209114"
Sep 16 18:40:28.469541 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:28.469458    2586 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:28.585628 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:28.585587    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c42d544-c24c-4b4b-b88d-13cc74d9fa87-original-pull-secret\") pod \"global-pull-secret-syncer-8ktbc\" (UID: \"8c42d544-c24c-4b4b-b88d-13cc74d9fa87\") " pod="kube-system/global-pull-secret-syncer-8ktbc"
Sep 16 18:40:28.585820 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:28.585645    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e8169e6-6afe-417b-90f0-5a55bf294956-metrics-certs\") pod \"network-metrics-daemon-9vpbt\" (UID: \"1e8169e6-6afe-417b-90f0-5a55bf294956\") " pod="openshift-multus/network-metrics-daemon-9vpbt"
Sep 16 18:40:28.585820 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:28.585794    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:28.585939 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:28.585859    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e8169e6-6afe-417b-90f0-5a55bf294956-metrics-certs podName:1e8169e6-6afe-417b-90f0-5a55bf294956 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:30.58583974 +0000 UTC m=+6.029701298 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e8169e6-6afe-417b-90f0-5a55bf294956-metrics-certs") pod "network-metrics-daemon-9vpbt" (UID: "1e8169e6-6afe-417b-90f0-5a55bf294956") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:28.586346 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:28.586324    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:28.586434 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:28.586384    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c42d544-c24c-4b4b-b88d-13cc74d9fa87-original-pull-secret podName:8c42d544-c24c-4b4b-b88d-13cc74d9fa87 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:30.586368762 +0000 UTC m=+6.030230318 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c42d544-c24c-4b4b-b88d-13cc74d9fa87-original-pull-secret") pod "global-pull-secret-syncer-8ktbc" (UID: "8c42d544-c24c-4b4b-b88d-13cc74d9fa87") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:28.686812 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:28.686739    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4jzq\" (UniqueName: \"kubernetes.io/projected/cbd2c59d-9acb-4f04-8068-c0e2d13e5d41-kube-api-access-k4jzq\") pod \"network-check-target-kmqvn\" (UID: \"cbd2c59d-9acb-4f04-8068-c0e2d13e5d41\") " pod="openshift-network-diagnostics/network-check-target-kmqvn"
Sep 16 18:40:28.687157 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:28.686938    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:28.687157 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:28.686970    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:28.687157 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:28.686983    2586 projected.go:196] Error preparing data for projected volume kube-api-access-k4jzq for pod openshift-network-diagnostics/network-check-target-kmqvn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:28.687157 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:28.687045    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cbd2c59d-9acb-4f04-8068-c0e2d13e5d41-kube-api-access-k4jzq podName:cbd2c59d-9acb-4f04-8068-c0e2d13e5d41 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:30.687025947 +0000 UTC m=+6.130887524 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-k4jzq" (UniqueName: "kubernetes.io/projected/cbd2c59d-9acb-4f04-8068-c0e2d13e5d41-kube-api-access-k4jzq") pod "network-check-target-kmqvn" (UID: "cbd2c59d-9acb-4f04-8068-c0e2d13e5d41") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:29.022317 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:29.022222    2586 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:35:27 +0000 UTC" deadline="2028-05-22 10:10:28.380197428 +0000 UTC"
Sep 16 18:40:29.022317 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:29.022286    2586 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14727h29m59.357916036s"
Sep 16 18:40:29.089295 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:29.088568    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmqvn"
Sep 16 18:40:29.089295 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:29.088693    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kmqvn" podUID="cbd2c59d-9acb-4f04-8068-c0e2d13e5d41"
Sep 16 18:40:29.089295 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:29.089077    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vpbt"
Sep 16 18:40:29.089295 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:29.089179    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vpbt" podUID="1e8169e6-6afe-417b-90f0-5a55bf294956"
Sep 16 18:40:29.124601 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:29.123395    2586 generic.go:356] "Generic (PLEG): container finished" podID="fd4c464e8a7b0afa02a1b06875f0d90d" containerID="020d65fb4da6ee202776168ef89da67042b3bcd571454a8b15a29df44c4ea4a9" exitCode=0
Sep 16 18:40:29.124601 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:29.124401    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-190.ec2.internal" event={"ID":"fd4c464e8a7b0afa02a1b06875f0d90d","Type":"ContainerDied","Data":"020d65fb4da6ee202776168ef89da67042b3bcd571454a8b15a29df44c4ea4a9"}
Sep 16 18:40:30.086465 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:30.086376    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ktbc"
Sep 16 18:40:30.087030 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:30.086519    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ktbc" podUID="8c42d544-c24c-4b4b-b88d-13cc74d9fa87"
Sep 16 18:40:30.160329 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:30.159533    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-190.ec2.internal" event={"ID":"fd4c464e8a7b0afa02a1b06875f0d90d","Type":"ContainerStarted","Data":"e09b9bab4689259bec80a6c2b0fd11d39243dd2ec249a5907ef17d160c420e73"}
Sep 16 18:40:30.607283 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:30.607246    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c42d544-c24c-4b4b-b88d-13cc74d9fa87-original-pull-secret\") pod \"global-pull-secret-syncer-8ktbc\" (UID: \"8c42d544-c24c-4b4b-b88d-13cc74d9fa87\") " pod="kube-system/global-pull-secret-syncer-8ktbc"
Sep 16 18:40:30.607470 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:30.607303    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e8169e6-6afe-417b-90f0-5a55bf294956-metrics-certs\") pod \"network-metrics-daemon-9vpbt\" (UID: \"1e8169e6-6afe-417b-90f0-5a55bf294956\") " pod="openshift-multus/network-metrics-daemon-9vpbt"
Sep 16 18:40:30.607470 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:30.607443    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:30.607589 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:30.607507    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e8169e6-6afe-417b-90f0-5a55bf294956-metrics-certs podName:1e8169e6-6afe-417b-90f0-5a55bf294956 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:34.607488615 +0000 UTC m=+10.051350181 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e8169e6-6afe-417b-90f0-5a55bf294956-metrics-certs") pod "network-metrics-daemon-9vpbt" (UID: "1e8169e6-6afe-417b-90f0-5a55bf294956") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:30.607901 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:30.607883    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:30.607982 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:30.607935    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c42d544-c24c-4b4b-b88d-13cc74d9fa87-original-pull-secret podName:8c42d544-c24c-4b4b-b88d-13cc74d9fa87 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:34.607920195 +0000 UTC m=+10.051781762 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c42d544-c24c-4b4b-b88d-13cc74d9fa87-original-pull-secret") pod "global-pull-secret-syncer-8ktbc" (UID: "8c42d544-c24c-4b4b-b88d-13cc74d9fa87") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:30.708360 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:30.708321    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4jzq\" (UniqueName: \"kubernetes.io/projected/cbd2c59d-9acb-4f04-8068-c0e2d13e5d41-kube-api-access-k4jzq\") pod \"network-check-target-kmqvn\" (UID: \"cbd2c59d-9acb-4f04-8068-c0e2d13e5d41\") " pod="openshift-network-diagnostics/network-check-target-kmqvn"
Sep 16 18:40:30.708537 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:30.708482    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:30.708537 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:30.708505    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:30.708537 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:30.708517    2586 projected.go:196] Error preparing data for projected volume kube-api-access-k4jzq for pod openshift-network-diagnostics/network-check-target-kmqvn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:30.708691 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:30.708579    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cbd2c59d-9acb-4f04-8068-c0e2d13e5d41-kube-api-access-k4jzq podName:cbd2c59d-9acb-4f04-8068-c0e2d13e5d41 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:34.708558127 +0000 UTC m=+10.152419685 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-k4jzq" (UniqueName: "kubernetes.io/projected/cbd2c59d-9acb-4f04-8068-c0e2d13e5d41-kube-api-access-k4jzq") pod "network-check-target-kmqvn" (UID: "cbd2c59d-9acb-4f04-8068-c0e2d13e5d41") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:31.087454 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:31.087383    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmqvn"
Sep 16 18:40:31.087866 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:31.087503    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kmqvn" podUID="cbd2c59d-9acb-4f04-8068-c0e2d13e5d41"
Sep 16 18:40:31.088036 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:31.088016    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vpbt"
Sep 16 18:40:31.088150 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:31.088128    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vpbt" podUID="1e8169e6-6afe-417b-90f0-5a55bf294956"
Sep 16 18:40:32.087399 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:32.087366    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ktbc"
Sep 16 18:40:32.087572 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:32.087521    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ktbc" podUID="8c42d544-c24c-4b4b-b88d-13cc74d9fa87"
Sep 16 18:40:33.087702 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:33.087174    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vpbt"
Sep 16 18:40:33.087702 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:33.087348    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vpbt" podUID="1e8169e6-6afe-417b-90f0-5a55bf294956"
Sep 16 18:40:33.087702 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:33.087405    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmqvn"
Sep 16 18:40:33.087702 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:33.087524    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kmqvn" podUID="cbd2c59d-9acb-4f04-8068-c0e2d13e5d41"
Sep 16 18:40:34.087025 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:34.086985    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ktbc"
Sep 16 18:40:34.087166 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:34.087116    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ktbc" podUID="8c42d544-c24c-4b4b-b88d-13cc74d9fa87"
Sep 16 18:40:34.643193 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:34.643153    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c42d544-c24c-4b4b-b88d-13cc74d9fa87-original-pull-secret\") pod \"global-pull-secret-syncer-8ktbc\" (UID: \"8c42d544-c24c-4b4b-b88d-13cc74d9fa87\") " pod="kube-system/global-pull-secret-syncer-8ktbc"
Sep 16 18:40:34.643672 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:34.643208    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e8169e6-6afe-417b-90f0-5a55bf294956-metrics-certs\") pod \"network-metrics-daemon-9vpbt\" (UID: \"1e8169e6-6afe-417b-90f0-5a55bf294956\") " pod="openshift-multus/network-metrics-daemon-9vpbt"
Sep 16 18:40:34.643672 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:34.643351    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:34.643672 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:34.643384    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:34.643672 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:34.643428    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c42d544-c24c-4b4b-b88d-13cc74d9fa87-original-pull-secret podName:8c42d544-c24c-4b4b-b88d-13cc74d9fa87 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:42.643409726 +0000 UTC m=+18.087271298 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c42d544-c24c-4b4b-b88d-13cc74d9fa87-original-pull-secret") pod "global-pull-secret-syncer-8ktbc" (UID: "8c42d544-c24c-4b4b-b88d-13cc74d9fa87") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:34.643672 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:34.643451    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e8169e6-6afe-417b-90f0-5a55bf294956-metrics-certs podName:1e8169e6-6afe-417b-90f0-5a55bf294956 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:42.643440206 +0000 UTC m=+18.087301784 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e8169e6-6afe-417b-90f0-5a55bf294956-metrics-certs") pod "network-metrics-daemon-9vpbt" (UID: "1e8169e6-6afe-417b-90f0-5a55bf294956") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:34.743842 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:34.743799    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4jzq\" (UniqueName: \"kubernetes.io/projected/cbd2c59d-9acb-4f04-8068-c0e2d13e5d41-kube-api-access-k4jzq\") pod \"network-check-target-kmqvn\" (UID: \"cbd2c59d-9acb-4f04-8068-c0e2d13e5d41\") " pod="openshift-network-diagnostics/network-check-target-kmqvn"
Sep 16 18:40:34.744021 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:34.743996    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:34.744021 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:34.744020    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:34.744138 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:34.744033    2586 projected.go:196] Error preparing data for projected volume kube-api-access-k4jzq for pod openshift-network-diagnostics/network-check-target-kmqvn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:34.744138 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:34.744094    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cbd2c59d-9acb-4f04-8068-c0e2d13e5d41-kube-api-access-k4jzq podName:cbd2c59d-9acb-4f04-8068-c0e2d13e5d41 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:42.744076462 +0000 UTC m=+18.187938032 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-k4jzq" (UniqueName: "kubernetes.io/projected/cbd2c59d-9acb-4f04-8068-c0e2d13e5d41-kube-api-access-k4jzq") pod "network-check-target-kmqvn" (UID: "cbd2c59d-9acb-4f04-8068-c0e2d13e5d41") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:35.088725 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:35.088087    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmqvn"
Sep 16 18:40:35.088725 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:35.088188    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kmqvn" podUID="cbd2c59d-9acb-4f04-8068-c0e2d13e5d41"
Sep 16 18:40:35.088725 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:35.088273    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vpbt"
Sep 16 18:40:35.088725 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:35.088369    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vpbt" podUID="1e8169e6-6afe-417b-90f0-5a55bf294956"
Sep 16 18:40:36.086453 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:36.086371    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ktbc"
Sep 16 18:40:36.086875 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:36.086493    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ktbc" podUID="8c42d544-c24c-4b4b-b88d-13cc74d9fa87"
Sep 16 18:40:37.086626 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:37.086581    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vpbt"
Sep 16 18:40:37.086626 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:37.086605    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmqvn"
Sep 16 18:40:37.087208 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:37.086733    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vpbt" podUID="1e8169e6-6afe-417b-90f0-5a55bf294956"
Sep 16 18:40:37.087208 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:37.086840    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kmqvn" podUID="cbd2c59d-9acb-4f04-8068-c0e2d13e5d41"
Sep 16 18:40:38.087160 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:38.087124    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ktbc"
Sep 16 18:40:38.087552 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:38.087223    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ktbc" podUID="8c42d544-c24c-4b4b-b88d-13cc74d9fa87"
Sep 16 18:40:39.086666 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:39.086450    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vpbt"
Sep 16 18:40:39.086857 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:39.086498    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmqvn"
Sep 16 18:40:39.086857 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:39.086780    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vpbt" podUID="1e8169e6-6afe-417b-90f0-5a55bf294956"
Sep 16 18:40:39.086857 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:39.086850    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kmqvn" podUID="cbd2c59d-9acb-4f04-8068-c0e2d13e5d41"
Sep 16 18:40:40.086224 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:40.086192    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ktbc"
Sep 16 18:40:40.086678 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:40.086315    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ktbc" podUID="8c42d544-c24c-4b4b-b88d-13cc74d9fa87"
Sep 16 18:40:41.087084 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:41.087046    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vpbt"
Sep 16 18:40:41.087550 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:41.087194    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vpbt" podUID="1e8169e6-6afe-417b-90f0-5a55bf294956"
Sep 16 18:40:41.087550 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:41.087253    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmqvn"
Sep 16 18:40:41.087550 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:41.087371    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kmqvn" podUID="cbd2c59d-9acb-4f04-8068-c0e2d13e5d41"
Sep 16 18:40:42.086649 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:42.086620    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ktbc"
Sep 16 18:40:42.086881 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:42.086714    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ktbc" podUID="8c42d544-c24c-4b4b-b88d-13cc74d9fa87"
Sep 16 18:40:42.700970 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:42.700934    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c42d544-c24c-4b4b-b88d-13cc74d9fa87-original-pull-secret\") pod \"global-pull-secret-syncer-8ktbc\" (UID: \"8c42d544-c24c-4b4b-b88d-13cc74d9fa87\") " pod="kube-system/global-pull-secret-syncer-8ktbc"
Sep 16 18:40:42.701413 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:42.700978    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e8169e6-6afe-417b-90f0-5a55bf294956-metrics-certs\") pod \"network-metrics-daemon-9vpbt\" (UID: \"1e8169e6-6afe-417b-90f0-5a55bf294956\") " pod="openshift-multus/network-metrics-daemon-9vpbt"
Sep 16 18:40:42.701413 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:42.701088    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:42.701413 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:42.701095    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:42.701413 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:42.701148    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e8169e6-6afe-417b-90f0-5a55bf294956-metrics-certs podName:1e8169e6-6afe-417b-90f0-5a55bf294956 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:58.701134071 +0000 UTC m=+34.144995638 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e8169e6-6afe-417b-90f0-5a55bf294956-metrics-certs") pod "network-metrics-daemon-9vpbt" (UID: "1e8169e6-6afe-417b-90f0-5a55bf294956") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:42.701413 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:42.701162    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c42d544-c24c-4b4b-b88d-13cc74d9fa87-original-pull-secret podName:8c42d544-c24c-4b4b-b88d-13cc74d9fa87 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:58.701155655 +0000 UTC m=+34.145017210 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c42d544-c24c-4b4b-b88d-13cc74d9fa87-original-pull-secret") pod "global-pull-secret-syncer-8ktbc" (UID: "8c42d544-c24c-4b4b-b88d-13cc74d9fa87") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:42.801869 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:42.801840    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4jzq\" (UniqueName: \"kubernetes.io/projected/cbd2c59d-9acb-4f04-8068-c0e2d13e5d41-kube-api-access-k4jzq\") pod \"network-check-target-kmqvn\" (UID: \"cbd2c59d-9acb-4f04-8068-c0e2d13e5d41\") " pod="openshift-network-diagnostics/network-check-target-kmqvn"
Sep 16 18:40:42.802030 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:42.801970    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:42.802030 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:42.801987    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:42.802030 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:42.801999    2586 projected.go:196] Error preparing data for projected volume kube-api-access-k4jzq for pod openshift-network-diagnostics/network-check-target-kmqvn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:42.802202 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:42.802064    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cbd2c59d-9acb-4f04-8068-c0e2d13e5d41-kube-api-access-k4jzq podName:cbd2c59d-9acb-4f04-8068-c0e2d13e5d41 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:58.802045252 +0000 UTC m=+34.245906821 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-k4jzq" (UniqueName: "kubernetes.io/projected/cbd2c59d-9acb-4f04-8068-c0e2d13e5d41-kube-api-access-k4jzq") pod "network-check-target-kmqvn" (UID: "cbd2c59d-9acb-4f04-8068-c0e2d13e5d41") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:43.087171 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:43.087096    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmqvn"
Sep 16 18:40:43.087343 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:43.087214    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kmqvn" podUID="cbd2c59d-9acb-4f04-8068-c0e2d13e5d41"
Sep 16 18:40:43.087343 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:43.087289    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vpbt"
Sep 16 18:40:43.087450 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:43.087419    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vpbt" podUID="1e8169e6-6afe-417b-90f0-5a55bf294956"
Sep 16 18:40:44.087222 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:44.087179    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ktbc"
Sep 16 18:40:44.087659 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:44.087314    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ktbc" podUID="8c42d544-c24c-4b4b-b88d-13cc74d9fa87"
Sep 16 18:40:45.091511 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:45.091469    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmqvn"
Sep 16 18:40:45.091971 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:45.091898    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kmqvn" podUID="cbd2c59d-9acb-4f04-8068-c0e2d13e5d41"
Sep 16 18:40:45.092126 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:45.092105    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vpbt"
Sep 16 18:40:45.092414 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:45.092382    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vpbt" podUID="1e8169e6-6afe-417b-90f0-5a55bf294956"
Sep 16 18:40:46.086210 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:46.086180    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ktbc"
Sep 16 18:40:46.086343 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:46.086324    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ktbc" podUID="8c42d544-c24c-4b4b-b88d-13cc74d9fa87"
Sep 16 18:40:46.194960 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:46.194777    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbvbq" event={"ID":"79f801e9-77ff-4d5d-8d54-ffb25be9dd3a","Type":"ContainerStarted","Data":"70bffb3bd9ab2388d8be639c5e7a63ede06edb96fd062b0048b89bd4652f6c83"}
Sep 16 18:40:46.196346 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:46.196319    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7n8dt" event={"ID":"6f68d3ed-0908-4d8a-a879-5b002a923ca9","Type":"ContainerStarted","Data":"08d2ae8b75b47dbfdf9b3a70aa82ca754411aea249d71b492a52e17273568b1b"}
Sep 16 18:40:46.197788 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:46.197763    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6kjpc" event={"ID":"2807fdd0-b58d-446d-ac9b-84a50d9b9ac6","Type":"ContainerStarted","Data":"942168c189d5358b336d402bd533abcabb7bb131c208f61c10a1dc44fdf3b697"}
Sep 16 18:40:46.201125 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:46.201102    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wbl55" event={"ID":"66e76bc5-f7b3-4952-b67c-0fb014f21ecb","Type":"ContainerStarted","Data":"32a3c91ebb172e62cd28079a6de607ae55145085197700710c95e4e2f6e7f70c"}
Sep 16 18:40:46.203558 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:46.203534    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2" event={"ID":"f76a1a1e-3387-4aae-8964-63c913eb270a","Type":"ContainerStarted","Data":"a9354155716d4092976169505303a6540e84f3b4f8bd788e4069529aa771c70e"}
Sep 16 18:40:46.203654 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:46.203592    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2" event={"ID":"f76a1a1e-3387-4aae-8964-63c913eb270a","Type":"ContainerStarted","Data":"8aeecba9cbe9b26322d23ca655a8da7f00c8f45c3166f7ab27c3b8ec741ce0a6"}
Sep 16 18:40:46.203654 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:46.203644    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2" event={"ID":"f76a1a1e-3387-4aae-8964-63c913eb270a","Type":"ContainerStarted","Data":"40083215a4459cb36d992bcf1d24d03cfe6a71416a5661c438981342a7526c19"}
Sep 16 18:40:46.205368 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:46.205324    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cc7hj" event={"ID":"b58ed079-b715-420e-a11f-2023c52e1321","Type":"ContainerStarted","Data":"481f6ae924a27f20da2d95925448a8f7a14a99a94210aa2c4a2e7543c767a6a9"}
Sep 16 18:40:46.207200 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:46.207175    2586 generic.go:356] "Generic (PLEG): container finished" podID="9716763a-ff84-4735-b478-deb614be27c4" containerID="b008ea9331a8570aba11e29a0a5725a4485970f0534c23c9698e699d124647c5" exitCode=0
Sep 16 18:40:46.207470 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:46.207265    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9s7jf" event={"ID":"9716763a-ff84-4735-b478-deb614be27c4","Type":"ContainerDied","Data":"b008ea9331a8570aba11e29a0a5725a4485970f0534c23c9698e699d124647c5"}
Sep 16 18:40:46.209063 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:46.208905    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6298x" event={"ID":"7fe5b556-37c9-40c7-a09b-5c6116b43e6f","Type":"ContainerStarted","Data":"428450088b600b1f9fd79078d4fa4c3174cbdb1bc41887a68bbc5cdac242d4a2"}
Sep 16 18:40:46.237976 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:46.237930    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-190.ec2.internal" podStartSLOduration=20.237890884 podStartE2EDuration="20.237890884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:30.185888062 +0000 UTC m=+5.629749642" watchObservedRunningTime="2026-09-16 18:40:46.237890884 +0000 UTC m=+21.681752451"
Sep 16 18:40:46.238712 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:46.238672    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-7n8dt" podStartSLOduration=10.831557808 podStartE2EDuration="21.238654177s" totalImagesPullingTime="10.407096369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:46.238076899 +0000 UTC m=+21.681938477" watchObservedRunningTime="2026-09-16 18:40:46.238654177 +0000 UTC m=+21.682515754"
Sep 16 18:40:46.287968 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:46.287908    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6298x" podStartSLOduration=3.547722834 podStartE2EDuration="21.287886324s" totalImagesPullingTime="17.74016349s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:46.260086997 +0000 UTC m=+21.703948575" watchObservedRunningTime="2026-09-16 18:40:46.287886324 +0000 UTC m=+21.731747902"
Sep 16 18:40:46.324507 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:46.324448    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-6kjpc" podStartSLOduration=3.579613651 podStartE2EDuration="21.324428731s" totalImagesPullingTime="17.74481508s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:46.324189102 +0000 UTC m=+21.768050672" watchObservedRunningTime="2026-09-16 18:40:46.324428731 +0000 UTC m=+21.768290306"
Sep 16 18:40:46.345157 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:46.345112    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-cc7hj" podStartSLOduration=3.591916455 podStartE2EDuration="21.345091152s" totalImagesPullingTime="17.753174697s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:46.344426569 +0000 UTC m=+21.788288148" watchObservedRunningTime="2026-09-16 18:40:46.345091152 +0000 UTC m=+21.788952727"
Sep 16 18:40:46.732508 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:46.732482    2586 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 18:40:47.020127 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:47.019815    2586 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T18:40:46.732504Z","UUID":"1c7a799e-927f-4f00-817d-61b70a61b267","Handler":null,"Name":"","Endpoint":""}
Sep 16 18:40:47.021384 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:47.021360    2586 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 18:40:47.021518 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:47.021397    2586 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 18:40:47.086876 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:47.086846    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmqvn"
Sep 16 18:40:47.087032 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:47.086891    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vpbt"
Sep 16 18:40:47.087032 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:47.086972    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kmqvn" podUID="cbd2c59d-9acb-4f04-8068-c0e2d13e5d41"
Sep 16 18:40:47.087149 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:47.087101    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vpbt" podUID="1e8169e6-6afe-417b-90f0-5a55bf294956"
Sep 16 18:40:47.213848 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:47.213803    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbvbq" event={"ID":"79f801e9-77ff-4d5d-8d54-ffb25be9dd3a","Type":"ContainerStarted","Data":"1451470cfa14fa4e8c2310ca1d80f44dd68d33bc2bb2226716c285131093bdb3"}
Sep 16 18:40:47.215671 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:47.215629    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vktwg" event={"ID":"21e31a4b-fd39-4863-ada4-4c87f208a134","Type":"ContainerStarted","Data":"067ccc3168b9c14061543be31014a41b72ba492a2d24881d0c761f3ee4f32d4a"}
Sep 16 18:40:47.220454 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:47.220426    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2" event={"ID":"f76a1a1e-3387-4aae-8964-63c913eb270a","Type":"ContainerStarted","Data":"efd69a1f747e77f70089a11099fc2275e05b7e0150f6cda48ab4a5ec6c148dc5"}
Sep 16 18:40:47.220568 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:47.220508    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2" event={"ID":"f76a1a1e-3387-4aae-8964-63c913eb270a","Type":"ContainerStarted","Data":"0879b50245f838c8a456bf59e959c5a3cf8e9b08d6c0c13711e3236a56bda8c9"}
Sep 16 18:40:47.220568 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:47.220560    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2" event={"ID":"f76a1a1e-3387-4aae-8964-63c913eb270a","Type":"ContainerStarted","Data":"b1d3321a28069d253af30438f3cd8db40eacf5295c28195e4fe51e2c1531a8d4"}
Sep 16 18:40:47.246976 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:47.246917    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-wbl55" podStartSLOduration=4.466122477 podStartE2EDuration="22.24689538s" totalImagesPullingTime="17.780772903s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:46.369528491 +0000 UTC m=+21.813390074" watchObservedRunningTime="2026-09-16 18:40:47.24689538 +0000 UTC m=+22.690756960"
Sep 16 18:40:48.087190 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:48.087117    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ktbc"
Sep 16 18:40:48.087373 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:48.087262    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ktbc" podUID="8c42d544-c24c-4b4b-b88d-13cc74d9fa87"
Sep 16 18:40:48.225185 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:48.225148    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbvbq" event={"ID":"79f801e9-77ff-4d5d-8d54-ffb25be9dd3a","Type":"ContainerStarted","Data":"1d95fe4c3ded4f7e16224a1d0ff8d7c6aeba21c84a33dba721f74917977d92c4"}
Sep 16 18:40:48.249289 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:48.249223    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vktwg" podStartSLOduration=5.502759631 podStartE2EDuration="23.249208426s" totalImagesPullingTime="17.746448795s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:47.247292236 +0000 UTC m=+22.691153813" watchObservedRunningTime="2026-09-16 18:40:48.249208426 +0000 UTC m=+23.693070002"
Sep 16 18:40:48.249736 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:48.249702    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kbvbq" podStartSLOduration=3.829067825 podStartE2EDuration="23.249691797s" totalImagesPullingTime="19.420623972s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:48.248285204 +0000 UTC m=+23.692146779" watchObservedRunningTime="2026-09-16 18:40:48.249691797 +0000 UTC m=+23.693553374"
Sep 16 18:40:49.086549 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:49.086512    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vpbt"
Sep 16 18:40:49.086549 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:49.086552    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmqvn"
Sep 16 18:40:49.086754 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:49.086665    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vpbt" podUID="1e8169e6-6afe-417b-90f0-5a55bf294956"
Sep 16 18:40:49.086812 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:49.086786    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kmqvn" podUID="cbd2c59d-9acb-4f04-8068-c0e2d13e5d41"
Sep 16 18:40:49.231781 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:49.231746    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2" event={"ID":"f76a1a1e-3387-4aae-8964-63c913eb270a","Type":"ContainerStarted","Data":"6281539193b98b5704c2ed52cbb75671f42595641e0f31fc98ffb97aea967445"}
Sep 16 18:40:49.398491 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:49.398411    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-7n8dt"
Sep 16 18:40:49.399100 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:49.399077    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-7n8dt"
Sep 16 18:40:50.087023 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:50.086993    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ktbc"
Sep 16 18:40:50.087220 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:50.087118    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ktbc" podUID="8c42d544-c24c-4b4b-b88d-13cc74d9fa87"
Sep 16 18:40:50.233875 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:50.233846    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-7n8dt"
Sep 16 18:40:50.234368 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:50.234352    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7n8dt"
Sep 16 18:40:51.086751 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:51.086718    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmqvn"
Sep 16 18:40:51.086934 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:51.086817    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kmqvn" podUID="cbd2c59d-9acb-4f04-8068-c0e2d13e5d41"
Sep 16 18:40:51.086934 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:51.086879    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vpbt"
Sep 16 18:40:51.087021 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:51.086999    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vpbt" podUID="1e8169e6-6afe-417b-90f0-5a55bf294956"
Sep 16 18:40:51.239008 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:51.238972    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2" event={"ID":"f76a1a1e-3387-4aae-8964-63c913eb270a","Type":"ContainerStarted","Data":"a8ed81756081d041ee1dc3d2db45873ff5634b8be7601eacd9ce622639e70c2f"}
Sep 16 18:40:51.239570 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:51.239245    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:51.239570 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:51.239311    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:51.240930 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:51.240904    2586 generic.go:356] "Generic (PLEG): container finished" podID="9716763a-ff84-4735-b478-deb614be27c4" containerID="911b6d217fe32c2a1c8be315f7c5cb6f6764d143b9058fe6d9c96ea519e797a6" exitCode=0
Sep 16 18:40:51.241038 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:51.240993    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9s7jf" event={"ID":"9716763a-ff84-4735-b478-deb614be27c4","Type":"ContainerDied","Data":"911b6d217fe32c2a1c8be315f7c5cb6f6764d143b9058fe6d9c96ea519e797a6"}
Sep 16 18:40:51.253809 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:51.253790    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:51.281864 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:51.281823    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2" podStartSLOduration=8.084442653 podStartE2EDuration="26.281810423s" totalImagesPullingTime="18.19736777s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:51.281533834 +0000 UTC m=+26.725395407" watchObservedRunningTime="2026-09-16 18:40:51.281810423 +0000 UTC m=+26.725671999"
Sep 16 18:40:52.086970 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:52.086948    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ktbc"
Sep 16 18:40:52.087103 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:52.087070    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ktbc" podUID="8c42d544-c24c-4b4b-b88d-13cc74d9fa87"
Sep 16 18:40:52.246346 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:52.246312    2586 generic.go:356] "Generic (PLEG): container finished" podID="9716763a-ff84-4735-b478-deb614be27c4" containerID="93f254b20112db8a849be5ed75423c463acc6e2bc630e803173f2a5ab4663d0b" exitCode=0
Sep 16 18:40:52.247119 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:52.246393    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9s7jf" event={"ID":"9716763a-ff84-4735-b478-deb614be27c4","Type":"ContainerDied","Data":"93f254b20112db8a849be5ed75423c463acc6e2bc630e803173f2a5ab4663d0b"}
Sep 16 18:40:52.247119 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:52.246811    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:52.262045 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:52.262019    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:40:52.801572 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:52.801380    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8ktbc"]
Sep 16 18:40:52.801749 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:52.801708    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ktbc"
Sep 16 18:40:52.801840 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:52.801820    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ktbc" podUID="8c42d544-c24c-4b4b-b88d-13cc74d9fa87"
Sep 16 18:40:52.804946 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:52.804918    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kmqvn"]
Sep 16 18:40:52.805076 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:52.805023    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmqvn"
Sep 16 18:40:52.805141 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:52.805110    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kmqvn" podUID="cbd2c59d-9acb-4f04-8068-c0e2d13e5d41"
Sep 16 18:40:52.805815 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:52.805793    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9vpbt"]
Sep 16 18:40:52.805948 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:52.805896    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vpbt"
Sep 16 18:40:52.806264 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:52.805997    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vpbt" podUID="1e8169e6-6afe-417b-90f0-5a55bf294956"
Sep 16 18:40:53.251637 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:53.251540    2586 generic.go:356] "Generic (PLEG): container finished" podID="9716763a-ff84-4735-b478-deb614be27c4" containerID="08e11ef268f0c7e83f89be39a510007fb0b29b6914e559e406b9e5005045489b" exitCode=0
Sep 16 18:40:53.251637 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:53.251627    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9s7jf" event={"ID":"9716763a-ff84-4735-b478-deb614be27c4","Type":"ContainerDied","Data":"08e11ef268f0c7e83f89be39a510007fb0b29b6914e559e406b9e5005045489b"}
Sep 16 18:40:54.086388 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:54.086355    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ktbc"
Sep 16 18:40:54.086569 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:54.086355    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmqvn"
Sep 16 18:40:54.086569 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:54.086471    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ktbc" podUID="8c42d544-c24c-4b4b-b88d-13cc74d9fa87"
Sep 16 18:40:54.086569 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:54.086526    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kmqvn" podUID="cbd2c59d-9acb-4f04-8068-c0e2d13e5d41"
Sep 16 18:40:55.087134 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:55.087054    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vpbt"
Sep 16 18:40:55.087509 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:55.087183    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vpbt" podUID="1e8169e6-6afe-417b-90f0-5a55bf294956"
Sep 16 18:40:56.087152 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:56.087120    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmqvn"
Sep 16 18:40:56.087635 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:56.087256    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kmqvn" podUID="cbd2c59d-9acb-4f04-8068-c0e2d13e5d41"
Sep 16 18:40:56.087635 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:56.087327    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ktbc"
Sep 16 18:40:56.087635 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:56.087457    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ktbc" podUID="8c42d544-c24c-4b4b-b88d-13cc74d9fa87"
Sep 16 18:40:57.087157 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:57.087131    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vpbt"
Sep 16 18:40:57.087574 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:57.087282    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vpbt" podUID="1e8169e6-6afe-417b-90f0-5a55bf294956"
Sep 16 18:40:58.087204 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:58.087109    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ktbc"
Sep 16 18:40:58.087666 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:58.087219    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ktbc" podUID="8c42d544-c24c-4b4b-b88d-13cc74d9fa87"
Sep 16 18:40:58.087666 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:58.087284    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmqvn"
Sep 16 18:40:58.087666 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:58.087400    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kmqvn" podUID="cbd2c59d-9acb-4f04-8068-c0e2d13e5d41"
Sep 16 18:40:58.266730 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:58.266700    2586 generic.go:356] "Generic (PLEG): container finished" podID="9716763a-ff84-4735-b478-deb614be27c4" containerID="9db31dae307267bcf76910c225c1934885a35b644634766509ed31c6e99d6868" exitCode=0
Sep 16 18:40:58.266873 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:58.266769    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9s7jf" event={"ID":"9716763a-ff84-4735-b478-deb614be27c4","Type":"ContainerDied","Data":"9db31dae307267bcf76910c225c1934885a35b644634766509ed31c6e99d6868"}
Sep 16 18:40:58.356939 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:58.356921    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-190.ec2.internal" event="NodeReady"
Sep 16 18:40:58.357056 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:58.357046    2586 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 18:40:58.421775 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:58.421748    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6xjlp"]
Sep 16 18:40:58.430975 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:58.430956    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-cgs7w"]
Sep 16 18:40:58.431088 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:58.431072    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6xjlp"
Sep 16 18:40:58.434528 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:58.434506    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xfn4n\""
Sep 16 18:40:58.434607 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:58.434513    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 18:40:58.434811 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:58.434797    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 18:40:58.439427 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:58.439403    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cgs7w"
Sep 16 18:40:58.439888 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:58.439870    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6xjlp"]
Sep 16 18:40:58.442106 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:58.442086    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 18:40:58.442624 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:58.442433    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-mg98h\""
Sep 16 18:40:58.442624 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:58.442600    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 18:40:58.442777 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:58.442679    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 18:40:58.443809 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:58.443791    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cgs7w"]
Sep 16 18:40:58.515745 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:58.515715    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/141828f8-d629-4cbd-9ed2-875d52d4db56-metrics-tls\") pod \"dns-default-6xjlp\" (UID: \"141828f8-d629-4cbd-9ed2-875d52d4db56\") " pod="openshift-dns/dns-default-6xjlp"
Sep 16 18:40:58.515891 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:58.515755    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g75sd\" (UniqueName: \"kubernetes.io/projected/141828f8-d629-4cbd-9ed2-875d52d4db56-kube-api-access-g75sd\") pod \"dns-default-6xjlp\" (UID: \"141828f8-d629-4cbd-9ed2-875d52d4db56\") " pod="openshift-dns/dns-default-6xjlp"
Sep 16 18:40:58.515891 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:58.515780    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/141828f8-d629-4cbd-9ed2-875d52d4db56-config-volume\") pod \"dns-default-6xjlp\" (UID: \"141828f8-d629-4cbd-9ed2-875d52d4db56\") " pod="openshift-dns/dns-default-6xjlp"
Sep 16 18:40:58.515891 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:58.515824    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvdtj\" (UniqueName: \"kubernetes.io/projected/a437fd80-c809-4981-9f3e-9cb3ce96a470-kube-api-access-qvdtj\") pod \"ingress-canary-cgs7w\" (UID: \"a437fd80-c809-4981-9f3e-9cb3ce96a470\") " pod="openshift-ingress-canary/ingress-canary-cgs7w"
Sep 16 18:40:58.515989 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:58.515924    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/141828f8-d629-4cbd-9ed2-875d52d4db56-tmp-dir\") pod \"dns-default-6xjlp\" (UID: \"141828f8-d629-4cbd-9ed2-875d52d4db56\") " pod="openshift-dns/dns-default-6xjlp"
Sep 16 18:40:58.515989 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:58.515975    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a437fd80-c809-4981-9f3e-9cb3ce96a470-cert\") pod \"ingress-canary-cgs7w\" (UID: \"a437fd80-c809-4981-9f3e-9cb3ce96a470\") " pod="openshift-ingress-canary/ingress-canary-cgs7w"
Sep 16 18:40:58.617127 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:58.617055    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/141828f8-d629-4cbd-9ed2-875d52d4db56-tmp-dir\") pod \"dns-default-6xjlp\" (UID: \"141828f8-d629-4cbd-9ed2-875d52d4db56\") " pod="openshift-dns/dns-default-6xjlp"
Sep 16 18:40:58.617127 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:58.617089    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a437fd80-c809-4981-9f3e-9cb3ce96a470-cert\") pod \"ingress-canary-cgs7w\" (UID: \"a437fd80-c809-4981-9f3e-9cb3ce96a470\") " pod="openshift-ingress-canary/ingress-canary-cgs7w"
Sep 16 18:40:58.617127 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:58.617111    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/141828f8-d629-4cbd-9ed2-875d52d4db56-metrics-tls\") pod \"dns-default-6xjlp\" (UID: \"141828f8-d629-4cbd-9ed2-875d52d4db56\") " pod="openshift-dns/dns-default-6xjlp"
Sep 16 18:40:58.617400 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:58.617140    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g75sd\" (UniqueName: \"kubernetes.io/projected/141828f8-d629-4cbd-9ed2-875d52d4db56-kube-api-access-g75sd\") pod \"dns-default-6xjlp\" (UID: \"141828f8-d629-4cbd-9ed2-875d52d4db56\") " pod="openshift-dns/dns-default-6xjlp"
Sep 16 18:40:58.617400 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:58.617166    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/141828f8-d629-4cbd-9ed2-875d52d4db56-config-volume\") pod \"dns-default-6xjlp\" (UID: \"141828f8-d629-4cbd-9ed2-875d52d4db56\") " pod="openshift-dns/dns-default-6xjlp"
Sep 16 18:40:58.617400 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:58.617187    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvdtj\" (UniqueName: \"kubernetes.io/projected/a437fd80-c809-4981-9f3e-9cb3ce96a470-kube-api-access-qvdtj\") pod \"ingress-canary-cgs7w\" (UID: \"a437fd80-c809-4981-9f3e-9cb3ce96a470\") " pod="openshift-ingress-canary/ingress-canary-cgs7w"
Sep 16 18:40:58.617400 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:58.617258    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:40:58.617400 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:58.617328    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/141828f8-d629-4cbd-9ed2-875d52d4db56-metrics-tls podName:141828f8-d629-4cbd-9ed2-875d52d4db56 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:59.117306033 +0000 UTC m=+34.561167607 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/141828f8-d629-4cbd-9ed2-875d52d4db56-metrics-tls") pod "dns-default-6xjlp" (UID: "141828f8-d629-4cbd-9ed2-875d52d4db56") : secret "dns-default-metrics-tls" not found
Sep 16 18:40:58.617563 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:58.617505    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/141828f8-d629-4cbd-9ed2-875d52d4db56-tmp-dir\") pod \"dns-default-6xjlp\" (UID: \"141828f8-d629-4cbd-9ed2-875d52d4db56\") " pod="openshift-dns/dns-default-6xjlp"
Sep 16 18:40:58.617701 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:58.617685    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/141828f8-d629-4cbd-9ed2-875d52d4db56-config-volume\") pod \"dns-default-6xjlp\" (UID: \"141828f8-d629-4cbd-9ed2-875d52d4db56\") " pod="openshift-dns/dns-default-6xjlp"
Sep 16 18:40:58.621236 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:58.621206    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a437fd80-c809-4981-9f3e-9cb3ce96a470-cert\") pod \"ingress-canary-cgs7w\" (UID: \"a437fd80-c809-4981-9f3e-9cb3ce96a470\") " pod="openshift-ingress-canary/ingress-canary-cgs7w"
Sep 16 18:40:58.630807 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:58.630784    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvdtj\" (UniqueName: \"kubernetes.io/projected/a437fd80-c809-4981-9f3e-9cb3ce96a470-kube-api-access-qvdtj\") pod \"ingress-canary-cgs7w\" (UID: \"a437fd80-c809-4981-9f3e-9cb3ce96a470\") " pod="openshift-ingress-canary/ingress-canary-cgs7w"
Sep 16 18:40:58.631151 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:58.631135    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g75sd\" (UniqueName: \"kubernetes.io/projected/141828f8-d629-4cbd-9ed2-875d52d4db56-kube-api-access-g75sd\") pod \"dns-default-6xjlp\" (UID: \"141828f8-d629-4cbd-9ed2-875d52d4db56\") " pod="openshift-dns/dns-default-6xjlp"
Sep 16 18:40:58.718428 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:58.718398    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c42d544-c24c-4b4b-b88d-13cc74d9fa87-original-pull-secret\") pod \"global-pull-secret-syncer-8ktbc\" (UID: \"8c42d544-c24c-4b4b-b88d-13cc74d9fa87\") " pod="kube-system/global-pull-secret-syncer-8ktbc"
Sep 16 18:40:58.718428 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:58.718436    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e8169e6-6afe-417b-90f0-5a55bf294956-metrics-certs\") pod \"network-metrics-daemon-9vpbt\" (UID: \"1e8169e6-6afe-417b-90f0-5a55bf294956\") " pod="openshift-multus/network-metrics-daemon-9vpbt"
Sep 16 18:40:58.718653 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:58.718563    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:58.718653 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:58.718565    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:58.718653 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:58.718622    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e8169e6-6afe-417b-90f0-5a55bf294956-metrics-certs podName:1e8169e6-6afe-417b-90f0-5a55bf294956 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:30.718609788 +0000 UTC m=+66.162471342 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e8169e6-6afe-417b-90f0-5a55bf294956-metrics-certs") pod "network-metrics-daemon-9vpbt" (UID: "1e8169e6-6afe-417b-90f0-5a55bf294956") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:58.718653 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:58.718635    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c42d544-c24c-4b4b-b88d-13cc74d9fa87-original-pull-secret podName:8c42d544-c24c-4b4b-b88d-13cc74d9fa87 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:30.71862857 +0000 UTC m=+66.162490124 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c42d544-c24c-4b4b-b88d-13cc74d9fa87-original-pull-secret") pod "global-pull-secret-syncer-8ktbc" (UID: "8c42d544-c24c-4b4b-b88d-13cc74d9fa87") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:58.759098 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:58.759066    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cgs7w"
Sep 16 18:40:58.819633 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:58.819599    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4jzq\" (UniqueName: \"kubernetes.io/projected/cbd2c59d-9acb-4f04-8068-c0e2d13e5d41-kube-api-access-k4jzq\") pod \"network-check-target-kmqvn\" (UID: \"cbd2c59d-9acb-4f04-8068-c0e2d13e5d41\") " pod="openshift-network-diagnostics/network-check-target-kmqvn"
Sep 16 18:40:58.819817 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:58.819767    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:58.819817 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:58.819790    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:58.819817 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:58.819801    2586 projected.go:196] Error preparing data for projected volume kube-api-access-k4jzq for pod openshift-network-diagnostics/network-check-target-kmqvn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:58.819965 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:58.819870    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cbd2c59d-9acb-4f04-8068-c0e2d13e5d41-kube-api-access-k4jzq podName:cbd2c59d-9acb-4f04-8068-c0e2d13e5d41 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:30.819848837 +0000 UTC m=+66.263710414 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-k4jzq" (UniqueName: "kubernetes.io/projected/cbd2c59d-9acb-4f04-8068-c0e2d13e5d41-kube-api-access-k4jzq") pod "network-check-target-kmqvn" (UID: "cbd2c59d-9acb-4f04-8068-c0e2d13e5d41") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:59.074572 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:59.074499    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cgs7w"]
Sep 16 18:40:59.078727 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:40:59.078698    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda437fd80_c809_4981_9f3e_9cb3ce96a470.slice/crio-bc9fa4bf958263ff0f3c6ccdc597597333a8fdfeb1a27c90e3b6f430dea29048 WatchSource:0}: Error finding container bc9fa4bf958263ff0f3c6ccdc597597333a8fdfeb1a27c90e3b6f430dea29048: Status 404 returned error can't find the container with id bc9fa4bf958263ff0f3c6ccdc597597333a8fdfeb1a27c90e3b6f430dea29048
Sep 16 18:40:59.086867 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:59.086846    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vpbt"
Sep 16 18:40:59.093552 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:59.093531    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:40:59.094080 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:59.093552    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-564xp\""
Sep 16 18:40:59.122401 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:59.122372    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/141828f8-d629-4cbd-9ed2-875d52d4db56-metrics-tls\") pod \"dns-default-6xjlp\" (UID: \"141828f8-d629-4cbd-9ed2-875d52d4db56\") " pod="openshift-dns/dns-default-6xjlp"
Sep 16 18:40:59.122534 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:59.122481    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:40:59.122571 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:40:59.122546    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/141828f8-d629-4cbd-9ed2-875d52d4db56-metrics-tls podName:141828f8-d629-4cbd-9ed2-875d52d4db56 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:00.122525319 +0000 UTC m=+35.566386873 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/141828f8-d629-4cbd-9ed2-875d52d4db56-metrics-tls") pod "dns-default-6xjlp" (UID: "141828f8-d629-4cbd-9ed2-875d52d4db56") : secret "dns-default-metrics-tls" not found
Sep 16 18:40:59.275681 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:59.275647    2586 generic.go:356] "Generic (PLEG): container finished" podID="9716763a-ff84-4735-b478-deb614be27c4" containerID="b5e638e69feacaec652e6dded825faf773bf7f27e750f31c32f57cfd1184d7b1" exitCode=0
Sep 16 18:40:59.275836 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:59.275725    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9s7jf" event={"ID":"9716763a-ff84-4735-b478-deb614be27c4","Type":"ContainerDied","Data":"b5e638e69feacaec652e6dded825faf773bf7f27e750f31c32f57cfd1184d7b1"}
Sep 16 18:40:59.276866 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:40:59.276846    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cgs7w" event={"ID":"a437fd80-c809-4981-9f3e-9cb3ce96a470","Type":"ContainerStarted","Data":"bc9fa4bf958263ff0f3c6ccdc597597333a8fdfeb1a27c90e3b6f430dea29048"}
Sep 16 18:41:00.087190 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:41:00.087011    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ktbc"
Sep 16 18:41:00.087492 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:41:00.087057    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmqvn"
Sep 16 18:41:00.092370 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:41:00.092348    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:41:00.092557 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:41:00.092544    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:41:00.093093 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:41:00.093074    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:41:00.093166 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:41:00.093073    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-z52vg\""
Sep 16 18:41:00.132526 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:41:00.132501    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/141828f8-d629-4cbd-9ed2-875d52d4db56-metrics-tls\") pod \"dns-default-6xjlp\" (UID: \"141828f8-d629-4cbd-9ed2-875d52d4db56\") " pod="openshift-dns/dns-default-6xjlp"
Sep 16 18:41:00.132855 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:41:00.132658    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:41:00.132855 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:41:00.132728    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/141828f8-d629-4cbd-9ed2-875d52d4db56-metrics-tls podName:141828f8-d629-4cbd-9ed2-875d52d4db56 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:02.132708318 +0000 UTC m=+37.576569877 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/141828f8-d629-4cbd-9ed2-875d52d4db56-metrics-tls") pod "dns-default-6xjlp" (UID: "141828f8-d629-4cbd-9ed2-875d52d4db56") : secret "dns-default-metrics-tls" not found
Sep 16 18:41:00.284265 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:41:00.284180    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9s7jf" event={"ID":"9716763a-ff84-4735-b478-deb614be27c4","Type":"ContainerStarted","Data":"9ae7708bdf54f4b444753c7e78766351e091e5ab163fa7cfb6d0b52b73236cab"}
Sep 16 18:41:00.313764 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:41:00.313714    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9s7jf" podStartSLOduration=7.482183232 podStartE2EDuration="35.313697364s" totalImagesPullingTime="27.831514132s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:25 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:00.312101318 +0000 UTC m=+35.755962907" watchObservedRunningTime="2026-09-16 18:41:00.313697364 +0000 UTC m=+35.757558944"
Sep 16 18:41:02.149785 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:41:02.149742    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/141828f8-d629-4cbd-9ed2-875d52d4db56-metrics-tls\") pod \"dns-default-6xjlp\" (UID: \"141828f8-d629-4cbd-9ed2-875d52d4db56\") " pod="openshift-dns/dns-default-6xjlp"
Sep 16 18:41:02.150208 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:41:02.149890    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:41:02.150208 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:41:02.149958    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/141828f8-d629-4cbd-9ed2-875d52d4db56-metrics-tls podName:141828f8-d629-4cbd-9ed2-875d52d4db56 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:06.149943437 +0000 UTC m=+41.593805008 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/141828f8-d629-4cbd-9ed2-875d52d4db56-metrics-tls") pod "dns-default-6xjlp" (UID: "141828f8-d629-4cbd-9ed2-875d52d4db56") : secret "dns-default-metrics-tls" not found
Sep 16 18:41:02.290207 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:41:02.290172    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cgs7w" event={"ID":"a437fd80-c809-4981-9f3e-9cb3ce96a470","Type":"ContainerStarted","Data":"b30a66db92cafac276190d799e8d50b4c7ac9f86da86cb6fdc0d072e2122f1f1"}
Sep 16 18:41:06.176585 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:41:06.176549    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/141828f8-d629-4cbd-9ed2-875d52d4db56-metrics-tls\") pod \"dns-default-6xjlp\" (UID: \"141828f8-d629-4cbd-9ed2-875d52d4db56\") " pod="openshift-dns/dns-default-6xjlp"
Sep 16 18:41:06.176955 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:41:06.176702    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:41:06.176955 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:41:06.176778    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/141828f8-d629-4cbd-9ed2-875d52d4db56-metrics-tls podName:141828f8-d629-4cbd-9ed2-875d52d4db56 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:14.176760766 +0000 UTC m=+49.620622319 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/141828f8-d629-4cbd-9ed2-875d52d4db56-metrics-tls") pod "dns-default-6xjlp" (UID: "141828f8-d629-4cbd-9ed2-875d52d4db56") : secret "dns-default-metrics-tls" not found
Sep 16 18:41:14.235502 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:41:14.235458    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/141828f8-d629-4cbd-9ed2-875d52d4db56-metrics-tls\") pod \"dns-default-6xjlp\" (UID: \"141828f8-d629-4cbd-9ed2-875d52d4db56\") " pod="openshift-dns/dns-default-6xjlp"
Sep 16 18:41:14.235880 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:41:14.235622    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:41:14.235880 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:41:14.235691    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/141828f8-d629-4cbd-9ed2-875d52d4db56-metrics-tls podName:141828f8-d629-4cbd-9ed2-875d52d4db56 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:30.235672841 +0000 UTC m=+65.679534395 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/141828f8-d629-4cbd-9ed2-875d52d4db56-metrics-tls") pod "dns-default-6xjlp" (UID: "141828f8-d629-4cbd-9ed2-875d52d4db56") : secret "dns-default-metrics-tls" not found
Sep 16 18:41:24.264645 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:41:24.264618    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wxhg2"
Sep 16 18:41:24.344053 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:41:24.343940    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-cgs7w" podStartSLOduration=23.96069334 podStartE2EDuration="26.343899915s" totalImagesPullingTime="2.383206575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:02.31317342 +0000 UTC m=+37.757034995" watchObservedRunningTime="2026-09-16 18:41:24.343899915 +0000 UTC m=+59.787761495"
Sep 16 18:41:30.239438 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:41:30.239385    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/141828f8-d629-4cbd-9ed2-875d52d4db56-metrics-tls\") pod \"dns-default-6xjlp\" (UID: \"141828f8-d629-4cbd-9ed2-875d52d4db56\") " pod="openshift-dns/dns-default-6xjlp"
Sep 16 18:41:30.239904 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:41:30.239529    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:41:30.239904 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:41:30.239592    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/141828f8-d629-4cbd-9ed2-875d52d4db56-metrics-tls podName:141828f8-d629-4cbd-9ed2-875d52d4db56 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:02.239576271 +0000 UTC m=+97.683437826 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/141828f8-d629-4cbd-9ed2-875d52d4db56-metrics-tls") pod "dns-default-6xjlp" (UID: "141828f8-d629-4cbd-9ed2-875d52d4db56") : secret "dns-default-metrics-tls" not found
Sep 16 18:41:30.742586 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:41:30.742554    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c42d544-c24c-4b4b-b88d-13cc74d9fa87-original-pull-secret\") pod \"global-pull-secret-syncer-8ktbc\" (UID: \"8c42d544-c24c-4b4b-b88d-13cc74d9fa87\") " pod="kube-system/global-pull-secret-syncer-8ktbc"
Sep 16 18:41:30.742729 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:41:30.742592    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e8169e6-6afe-417b-90f0-5a55bf294956-metrics-certs\") pod \"network-metrics-daemon-9vpbt\" (UID: \"1e8169e6-6afe-417b-90f0-5a55bf294956\") " pod="openshift-multus/network-metrics-daemon-9vpbt"
Sep 16 18:41:30.749891 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:41:30.749872    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:41:30.751024 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:41:30.751012    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:41:30.752954 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:41:30.752935    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 18:41:30.753013 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:41:30.753004    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e8169e6-6afe-417b-90f0-5a55bf294956-metrics-certs podName:1e8169e6-6afe-417b-90f0-5a55bf294956 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:34.752988831 +0000 UTC m=+130.196850399 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1e8169e6-6afe-417b-90f0-5a55bf294956-metrics-certs") pod "network-metrics-daemon-9vpbt" (UID: "1e8169e6-6afe-417b-90f0-5a55bf294956") : secret "metrics-daemon-secret" not found
Sep 16 18:41:30.756184 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:41:30.756161    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c42d544-c24c-4b4b-b88d-13cc74d9fa87-original-pull-secret\") pod \"global-pull-secret-syncer-8ktbc\" (UID: \"8c42d544-c24c-4b4b-b88d-13cc74d9fa87\") " pod="kube-system/global-pull-secret-syncer-8ktbc"
Sep 16 18:41:30.843643 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:41:30.843600    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4jzq\" (UniqueName: \"kubernetes.io/projected/cbd2c59d-9acb-4f04-8068-c0e2d13e5d41-kube-api-access-k4jzq\") pod \"network-check-target-kmqvn\" (UID: \"cbd2c59d-9acb-4f04-8068-c0e2d13e5d41\") " pod="openshift-network-diagnostics/network-check-target-kmqvn"
Sep 16 18:41:30.846607 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:41:30.846588    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:41:30.859024 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:41:30.859006    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:41:30.867207 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:41:30.867178    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4jzq\" (UniqueName: \"kubernetes.io/projected/cbd2c59d-9acb-4f04-8068-c0e2d13e5d41-kube-api-access-k4jzq\") pod \"network-check-target-kmqvn\" (UID: \"cbd2c59d-9acb-4f04-8068-c0e2d13e5d41\") " pod="openshift-network-diagnostics/network-check-target-kmqvn"
Sep 16 18:41:30.998570 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:41:30.998471    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ktbc"
Sep 16 18:41:31.007470 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:41:31.007449    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-z52vg\""
Sep 16 18:41:31.015579 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:41:31.015562    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmqvn"
Sep 16 18:41:31.284674 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:41:31.284589    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8ktbc"]
Sep 16 18:41:31.309451 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:41:31.309425    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbd2c59d_9acb_4f04_8068_c0e2d13e5d41.slice/crio-259bb3c575bbfaabd43cd9fcaf6351b98733f37daf4e902fba0f5124ac58b738 WatchSource:0}: Error finding container 259bb3c575bbfaabd43cd9fcaf6351b98733f37daf4e902fba0f5124ac58b738: Status 404 returned error can't find the container with id 259bb3c575bbfaabd43cd9fcaf6351b98733f37daf4e902fba0f5124ac58b738
Sep 16 18:41:31.311922 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:41:31.311899    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kmqvn"]
Sep 16 18:41:31.362162 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:41:31.362131    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kmqvn" event={"ID":"cbd2c59d-9acb-4f04-8068-c0e2d13e5d41","Type":"ContainerStarted","Data":"259bb3c575bbfaabd43cd9fcaf6351b98733f37daf4e902fba0f5124ac58b738"}
Sep 16 18:41:31.363266 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:41:31.363243    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8ktbc" event={"ID":"8c42d544-c24c-4b4b-b88d-13cc74d9fa87","Type":"ContainerStarted","Data":"57a26e22bb43c728019c0fb96bde60bf7f6325933421fb42bc5aca70be936ff7"}
Sep 16 18:41:36.378922 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:41:36.378885    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kmqvn" event={"ID":"cbd2c59d-9acb-4f04-8068-c0e2d13e5d41","Type":"ContainerStarted","Data":"888d2f8ae84393b3fc76d6f7d27f62a2e003f1aa7b1d008593160b16023776bc"}
Sep 16 18:41:36.379381 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:41:36.379011    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kmqvn"
Sep 16 18:41:36.380596 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:41:36.380574    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8ktbc" event={"ID":"8c42d544-c24c-4b4b-b88d-13cc74d9fa87","Type":"ContainerStarted","Data":"1c1ea0210e3fc1983b75f3121b0cef93e5459000d87f49d104a02fdcf2863fb7"}
Sep 16 18:41:36.414904 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:41:36.414859    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-kmqvn" podStartSLOduration=67.031686889 podStartE2EDuration="1m11.414847736s" totalImagesPullingTime="4.383160847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:36.412147528 +0000 UTC m=+71.856009104" watchObservedRunningTime="2026-09-16 18:41:36.414847736 +0000 UTC m=+71.858709306"
Sep 16 18:41:36.432721 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:41:36.432684    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-8ktbc" podStartSLOduration=67.02540518 podStartE2EDuration="1m11.432672044s" totalImagesPullingTime="4.407266864s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:36.432272312 +0000 UTC m=+71.876133889" watchObservedRunningTime="2026-09-16 18:41:36.432672044 +0000 UTC m=+71.876533620"
Sep 16 18:42:02.256581 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:02.256548    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/141828f8-d629-4cbd-9ed2-875d52d4db56-metrics-tls\") pod \"dns-default-6xjlp\" (UID: \"141828f8-d629-4cbd-9ed2-875d52d4db56\") " pod="openshift-dns/dns-default-6xjlp"
Sep 16 18:42:02.257059 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:02.256695    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:42:02.257059 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:02.256748    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/141828f8-d629-4cbd-9ed2-875d52d4db56-metrics-tls podName:141828f8-d629-4cbd-9ed2-875d52d4db56 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:06.256734575 +0000 UTC m=+161.700596129 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/141828f8-d629-4cbd-9ed2-875d52d4db56-metrics-tls") pod "dns-default-6xjlp" (UID: "141828f8-d629-4cbd-9ed2-875d52d4db56") : secret "dns-default-metrics-tls" not found
Sep 16 18:42:07.385398 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:07.385364    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-kmqvn"
Sep 16 18:42:13.374740 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.374682    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-njkrg"]
Sep 16 18:42:13.377967 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.377946    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-njkrg"
Sep 16 18:42:13.380424 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.380396    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:42:13.380618 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.380593    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-ps6sd"]
Sep 16 18:42:13.381127 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.381110    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 16 18:42:13.381212 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.381137    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-j6w2l\""
Sep 16 18:42:13.381566 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.381539    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 16 18:42:13.382329 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.382311    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-986b9b8c8-9p8c8"]
Sep 16 18:42:13.382459 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.382444    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-ps6sd"
Sep 16 18:42:13.385186 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.385165    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-986b9b8c8-9p8c8"
Sep 16 18:42:13.386604 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.386583    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:42:13.386898 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.386876    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 16 18:42:13.387003 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.386973    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 16 18:42:13.387293 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.387270    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-28dmc\""
Sep 16 18:42:13.387400 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.387313    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 16 18:42:13.388180 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.388160    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-x5qmd\""
Sep 16 18:42:13.388462 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.388447    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 16 18:42:13.388536 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.388460    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 16 18:42:13.388591 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.388535    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 18:42:13.388639 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.388628    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 16 18:42:13.388684 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.388658    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 18:42:13.388786 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.388746    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 16 18:42:13.398072 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.398056    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-ps6sd"]
Sep 16 18:42:13.398853 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.398832    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-njkrg"]
Sep 16 18:42:13.405659 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.405637    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-986b9b8c8-9p8c8"]
Sep 16 18:42:13.432359 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.432333    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1596b098-d098-43ae-bd52-b750f6fd3230-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-njkrg\" (UID: \"1596b098-d098-43ae-bd52-b750f6fd3230\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-njkrg"
Sep 16 18:42:13.432446 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.432385    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hw48\" (UniqueName: \"kubernetes.io/projected/1596b098-d098-43ae-bd52-b750f6fd3230-kube-api-access-5hw48\") pod \"cluster-samples-operator-86499c89c4-njkrg\" (UID: \"1596b098-d098-43ae-bd52-b750f6fd3230\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-njkrg"
Sep 16 18:42:13.533444 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.533410    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04e761f8-c6e9-4238-bd6b-0ee2db961991-service-ca-bundle\") pod \"router-default-986b9b8c8-9p8c8\" (UID: \"04e761f8-c6e9-4238-bd6b-0ee2db961991\") " pod="openshift-ingress/router-default-986b9b8c8-9p8c8"
Sep 16 18:42:13.533444 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.533446    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hw48\" (UniqueName: \"kubernetes.io/projected/1596b098-d098-43ae-bd52-b750f6fd3230-kube-api-access-5hw48\") pod \"cluster-samples-operator-86499c89c4-njkrg\" (UID: \"1596b098-d098-43ae-bd52-b750f6fd3230\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-njkrg"
Sep 16 18:42:13.533675 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.533468    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3ab787a9-4d8d-42f1-b7cb-432be6e37285-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-ps6sd\" (UID: \"3ab787a9-4d8d-42f1-b7cb-432be6e37285\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-ps6sd"
Sep 16 18:42:13.533675 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.533485    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rks5\" (UniqueName: \"kubernetes.io/projected/3ab787a9-4d8d-42f1-b7cb-432be6e37285-kube-api-access-7rks5\") pod \"kube-storage-version-migrator-operator-6c8f84f897-ps6sd\" (UID: \"3ab787a9-4d8d-42f1-b7cb-432be6e37285\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-ps6sd"
Sep 16 18:42:13.533675 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.533547    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/04e761f8-c6e9-4238-bd6b-0ee2db961991-stats-auth\") pod \"router-default-986b9b8c8-9p8c8\" (UID: \"04e761f8-c6e9-4238-bd6b-0ee2db961991\") " pod="openshift-ingress/router-default-986b9b8c8-9p8c8"
Sep 16 18:42:13.533775 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.533695    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/04e761f8-c6e9-4238-bd6b-0ee2db961991-default-certificate\") pod \"router-default-986b9b8c8-9p8c8\" (UID: \"04e761f8-c6e9-4238-bd6b-0ee2db961991\") " pod="openshift-ingress/router-default-986b9b8c8-9p8c8"
Sep 16 18:42:13.533775 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.533743    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnc9p\" (UniqueName: \"kubernetes.io/projected/04e761f8-c6e9-4238-bd6b-0ee2db961991-kube-api-access-gnc9p\") pod \"router-default-986b9b8c8-9p8c8\" (UID: \"04e761f8-c6e9-4238-bd6b-0ee2db961991\") " pod="openshift-ingress/router-default-986b9b8c8-9p8c8"
Sep 16 18:42:13.533891 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.533783    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1596b098-d098-43ae-bd52-b750f6fd3230-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-njkrg\" (UID: \"1596b098-d098-43ae-bd52-b750f6fd3230\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-njkrg"
Sep 16 18:42:13.533891 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.533803    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04e761f8-c6e9-4238-bd6b-0ee2db961991-metrics-certs\") pod \"router-default-986b9b8c8-9p8c8\" (UID: \"04e761f8-c6e9-4238-bd6b-0ee2db961991\") " pod="openshift-ingress/router-default-986b9b8c8-9p8c8"
Sep 16 18:42:13.533891 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.533828    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3ab787a9-4d8d-42f1-b7cb-432be6e37285-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-ps6sd\" (UID: \"3ab787a9-4d8d-42f1-b7cb-432be6e37285\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-ps6sd"
Sep 16 18:42:13.533994 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:13.533912    2586 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:42:13.533994 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:13.533967    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1596b098-d098-43ae-bd52-b750f6fd3230-samples-operator-tls podName:1596b098-d098-43ae-bd52-b750f6fd3230 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:14.033947734 +0000 UTC m=+109.477809292 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1596b098-d098-43ae-bd52-b750f6fd3230-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-njkrg" (UID: "1596b098-d098-43ae-bd52-b750f6fd3230") : secret "samples-operator-tls" not found
Sep 16 18:42:13.548588 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.548561    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hw48\" (UniqueName: \"kubernetes.io/projected/1596b098-d098-43ae-bd52-b750f6fd3230-kube-api-access-5hw48\") pod \"cluster-samples-operator-86499c89c4-njkrg\" (UID: \"1596b098-d098-43ae-bd52-b750f6fd3230\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-njkrg"
Sep 16 18:42:13.578732 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.578704    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-sh5xv"]
Sep 16 18:42:13.580660 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.580646    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-sh5xv"
Sep 16 18:42:13.584243 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.584210    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 16 18:42:13.584343 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.584251    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-kb6s9"]
Sep 16 18:42:13.584343 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.584256    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:42:13.584452 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.584439    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 16 18:42:13.584944 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.584924    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-rpd65\""
Sep 16 18:42:13.585663 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.585646    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 16 18:42:13.586073 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.586058    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-mggrx"]
Sep 16 18:42:13.586181 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.586162    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-kb6s9"
Sep 16 18:42:13.587800 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.587784    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mggrx"
Sep 16 18:42:13.590424 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.590405    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 18:42:13.590424 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.590419    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 16 18:42:13.590567 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.590485    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 16 18:42:13.596500 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.596481    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-vjj4r\""
Sep 16 18:42:13.598653 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.598638    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 18:42:13.598993 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.598971    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-sw8fc\""
Sep 16 18:42:13.605044 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.605028    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 18:42:13.609441 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.609422    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 18:42:13.609677 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.609666    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-sh5xv"]
Sep 16 18:42:13.610238 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.610212    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 16 18:42:13.610386 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.610369    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-kb6s9"]
Sep 16 18:42:13.613581 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.613562    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 16 18:42:13.625032 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.624987    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-mggrx"]
Sep 16 18:42:13.634844 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.634825    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3ab787a9-4d8d-42f1-b7cb-432be6e37285-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-ps6sd\" (UID: \"3ab787a9-4d8d-42f1-b7cb-432be6e37285\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-ps6sd"
Sep 16 18:42:13.634929 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.634856    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rks5\" (UniqueName: \"kubernetes.io/projected/3ab787a9-4d8d-42f1-b7cb-432be6e37285-kube-api-access-7rks5\") pod \"kube-storage-version-migrator-operator-6c8f84f897-ps6sd\" (UID: \"3ab787a9-4d8d-42f1-b7cb-432be6e37285\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-ps6sd"
Sep 16 18:42:13.634929 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.634875    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/04e761f8-c6e9-4238-bd6b-0ee2db961991-stats-auth\") pod \"router-default-986b9b8c8-9p8c8\" (UID: \"04e761f8-c6e9-4238-bd6b-0ee2db961991\") " pod="openshift-ingress/router-default-986b9b8c8-9p8c8"
Sep 16 18:42:13.634929 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.634902    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/04e761f8-c6e9-4238-bd6b-0ee2db961991-default-certificate\") pod \"router-default-986b9b8c8-9p8c8\" (UID: \"04e761f8-c6e9-4238-bd6b-0ee2db961991\") " pod="openshift-ingress/router-default-986b9b8c8-9p8c8"
Sep 16 18:42:13.635066 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.634934    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dbvz\" (UniqueName: \"kubernetes.io/projected/3a6129d0-3d10-4f44-881a-632760e357fd-kube-api-access-6dbvz\") pod \"service-ca-operator-cf7c776b7-sh5xv\" (UID: \"3a6129d0-3d10-4f44-881a-632760e357fd\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-sh5xv"
Sep 16 18:42:13.635066 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.634953    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnc9p\" (UniqueName: \"kubernetes.io/projected/04e761f8-c6e9-4238-bd6b-0ee2db961991-kube-api-access-gnc9p\") pod \"router-default-986b9b8c8-9p8c8\" (UID: \"04e761f8-c6e9-4238-bd6b-0ee2db961991\") " pod="openshift-ingress/router-default-986b9b8c8-9p8c8"
Sep 16 18:42:13.635066 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.634969    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3a6129d0-3d10-4f44-881a-632760e357fd-serving-cert\") pod \"service-ca-operator-cf7c776b7-sh5xv\" (UID: \"3a6129d0-3d10-4f44-881a-632760e357fd\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-sh5xv"
Sep 16 18:42:13.635066 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.635001    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3a6129d0-3d10-4f44-881a-632760e357fd-config\") pod \"service-ca-operator-cf7c776b7-sh5xv\" (UID: \"3a6129d0-3d10-4f44-881a-632760e357fd\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-sh5xv"
Sep 16 18:42:13.635257 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.635197    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04e761f8-c6e9-4238-bd6b-0ee2db961991-metrics-certs\") pod \"router-default-986b9b8c8-9p8c8\" (UID: \"04e761f8-c6e9-4238-bd6b-0ee2db961991\") " pod="openshift-ingress/router-default-986b9b8c8-9p8c8"
Sep 16 18:42:13.635257 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.635251    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3ab787a9-4d8d-42f1-b7cb-432be6e37285-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-ps6sd\" (UID: \"3ab787a9-4d8d-42f1-b7cb-432be6e37285\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-ps6sd"
Sep 16 18:42:13.635376 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.635297    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04e761f8-c6e9-4238-bd6b-0ee2db961991-service-ca-bundle\") pod \"router-default-986b9b8c8-9p8c8\" (UID: \"04e761f8-c6e9-4238-bd6b-0ee2db961991\") " pod="openshift-ingress/router-default-986b9b8c8-9p8c8"
Sep 16 18:42:13.635376 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:13.635316    2586 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:42:13.635376 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:13.635373    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04e761f8-c6e9-4238-bd6b-0ee2db961991-metrics-certs podName:04e761f8-c6e9-4238-bd6b-0ee2db961991 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:14.135354602 +0000 UTC m=+109.579216175 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/04e761f8-c6e9-4238-bd6b-0ee2db961991-metrics-certs") pod "router-default-986b9b8c8-9p8c8" (UID: "04e761f8-c6e9-4238-bd6b-0ee2db961991") : secret "router-metrics-certs-default" not found
Sep 16 18:42:13.635531 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:13.635453    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/04e761f8-c6e9-4238-bd6b-0ee2db961991-service-ca-bundle podName:04e761f8-c6e9-4238-bd6b-0ee2db961991 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:14.1354348 +0000 UTC m=+109.579296373 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/04e761f8-c6e9-4238-bd6b-0ee2db961991-service-ca-bundle") pod "router-default-986b9b8c8-9p8c8" (UID: "04e761f8-c6e9-4238-bd6b-0ee2db961991") : configmap references non-existent config key: service-ca.crt
Sep 16 18:42:13.635783 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.635762    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3ab787a9-4d8d-42f1-b7cb-432be6e37285-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-ps6sd\" (UID: \"3ab787a9-4d8d-42f1-b7cb-432be6e37285\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-ps6sd"
Sep 16 18:42:13.637092 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.637068    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3ab787a9-4d8d-42f1-b7cb-432be6e37285-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-ps6sd\" (UID: \"3ab787a9-4d8d-42f1-b7cb-432be6e37285\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-ps6sd"
Sep 16 18:42:13.637364 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.637347    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/04e761f8-c6e9-4238-bd6b-0ee2db961991-stats-auth\") pod \"router-default-986b9b8c8-9p8c8\" (UID: \"04e761f8-c6e9-4238-bd6b-0ee2db961991\") " pod="openshift-ingress/router-default-986b9b8c8-9p8c8"
Sep 16 18:42:13.637431 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.637396    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/04e761f8-c6e9-4238-bd6b-0ee2db961991-default-certificate\") pod \"router-default-986b9b8c8-9p8c8\" (UID: \"04e761f8-c6e9-4238-bd6b-0ee2db961991\") " pod="openshift-ingress/router-default-986b9b8c8-9p8c8"
Sep 16 18:42:13.657961 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.657939    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 16 18:42:13.674853 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.674827    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rks5\" (UniqueName: \"kubernetes.io/projected/3ab787a9-4d8d-42f1-b7cb-432be6e37285-kube-api-access-7rks5\") pod \"kube-storage-version-migrator-operator-6c8f84f897-ps6sd\" (UID: \"3ab787a9-4d8d-42f1-b7cb-432be6e37285\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-ps6sd"
Sep 16 18:42:13.679944 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.679921    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnc9p\" (UniqueName: \"kubernetes.io/projected/04e761f8-c6e9-4238-bd6b-0ee2db961991-kube-api-access-gnc9p\") pod \"router-default-986b9b8c8-9p8c8\" (UID: \"04e761f8-c6e9-4238-bd6b-0ee2db961991\") " pod="openshift-ingress/router-default-986b9b8c8-9p8c8"
Sep 16 18:42:13.696937 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.696919    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-ps6sd"
Sep 16 18:42:13.736454 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.736426    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f801294c-a332-41af-bede-44bcd7a22e2c-serving-cert\") pod \"insights-operator-5844b856db-kb6s9\" (UID: \"f801294c-a332-41af-bede-44bcd7a22e2c\") " pod="openshift-insights/insights-operator-5844b856db-kb6s9"
Sep 16 18:42:13.736595 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.736460    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f801294c-a332-41af-bede-44bcd7a22e2c-service-ca-bundle\") pod \"insights-operator-5844b856db-kb6s9\" (UID: \"f801294c-a332-41af-bede-44bcd7a22e2c\") " pod="openshift-insights/insights-operator-5844b856db-kb6s9"
Sep 16 18:42:13.736595 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.736484    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f01001cd-d72f-4267-8e14-44359c95f66c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-mggrx\" (UID: \"f01001cd-d72f-4267-8e14-44359c95f66c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mggrx"
Sep 16 18:42:13.736595 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.736526    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7grs5\" (UniqueName: \"kubernetes.io/projected/f01001cd-d72f-4267-8e14-44359c95f66c-kube-api-access-7grs5\") pod \"cluster-monitoring-operator-6cd954b897-mggrx\" (UID: \"f01001cd-d72f-4267-8e14-44359c95f66c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mggrx"
Sep 16 18:42:13.736595 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.736578    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f801294c-a332-41af-bede-44bcd7a22e2c-snapshots\") pod \"insights-operator-5844b856db-kb6s9\" (UID: \"f801294c-a332-41af-bede-44bcd7a22e2c\") " pod="openshift-insights/insights-operator-5844b856db-kb6s9"
Sep 16 18:42:13.736741 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.736620    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f801294c-a332-41af-bede-44bcd7a22e2c-trusted-ca-bundle\") pod \"insights-operator-5844b856db-kb6s9\" (UID: \"f801294c-a332-41af-bede-44bcd7a22e2c\") " pod="openshift-insights/insights-operator-5844b856db-kb6s9"
Sep 16 18:42:13.736741 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.736638    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwntn\" (UniqueName: \"kubernetes.io/projected/f801294c-a332-41af-bede-44bcd7a22e2c-kube-api-access-xwntn\") pod \"insights-operator-5844b856db-kb6s9\" (UID: \"f801294c-a332-41af-bede-44bcd7a22e2c\") " pod="openshift-insights/insights-operator-5844b856db-kb6s9"
Sep 16 18:42:13.736741 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.736654    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f01001cd-d72f-4267-8e14-44359c95f66c-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-mggrx\" (UID: \"f01001cd-d72f-4267-8e14-44359c95f66c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mggrx"
Sep 16 18:42:13.736741 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.736697    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dbvz\" (UniqueName: \"kubernetes.io/projected/3a6129d0-3d10-4f44-881a-632760e357fd-kube-api-access-6dbvz\") pod \"service-ca-operator-cf7c776b7-sh5xv\" (UID: \"3a6129d0-3d10-4f44-881a-632760e357fd\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-sh5xv"
Sep 16 18:42:13.736741 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.736735    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3a6129d0-3d10-4f44-881a-632760e357fd-serving-cert\") pod \"service-ca-operator-cf7c776b7-sh5xv\" (UID: \"3a6129d0-3d10-4f44-881a-632760e357fd\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-sh5xv"
Sep 16 18:42:13.736942 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.736749    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3a6129d0-3d10-4f44-881a-632760e357fd-config\") pod \"service-ca-operator-cf7c776b7-sh5xv\" (UID: \"3a6129d0-3d10-4f44-881a-632760e357fd\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-sh5xv"
Sep 16 18:42:13.736942 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.736765    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f801294c-a332-41af-bede-44bcd7a22e2c-tmp\") pod \"insights-operator-5844b856db-kb6s9\" (UID: \"f801294c-a332-41af-bede-44bcd7a22e2c\") " pod="openshift-insights/insights-operator-5844b856db-kb6s9"
Sep 16 18:42:13.737262 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.737219    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3a6129d0-3d10-4f44-881a-632760e357fd-config\") pod \"service-ca-operator-cf7c776b7-sh5xv\" (UID: \"3a6129d0-3d10-4f44-881a-632760e357fd\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-sh5xv"
Sep 16 18:42:13.739038 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.739008    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3a6129d0-3d10-4f44-881a-632760e357fd-serving-cert\") pod \"service-ca-operator-cf7c776b7-sh5xv\" (UID: \"3a6129d0-3d10-4f44-881a-632760e357fd\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-sh5xv"
Sep 16 18:42:13.755820 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.755772    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dbvz\" (UniqueName: \"kubernetes.io/projected/3a6129d0-3d10-4f44-881a-632760e357fd-kube-api-access-6dbvz\") pod \"service-ca-operator-cf7c776b7-sh5xv\" (UID: \"3a6129d0-3d10-4f44-881a-632760e357fd\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-sh5xv"
Sep 16 18:42:13.837077 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.837040    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwntn\" (UniqueName: \"kubernetes.io/projected/f801294c-a332-41af-bede-44bcd7a22e2c-kube-api-access-xwntn\") pod \"insights-operator-5844b856db-kb6s9\" (UID: \"f801294c-a332-41af-bede-44bcd7a22e2c\") " pod="openshift-insights/insights-operator-5844b856db-kb6s9"
Sep 16 18:42:13.837215 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.837082    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f01001cd-d72f-4267-8e14-44359c95f66c-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-mggrx\" (UID: \"f01001cd-d72f-4267-8e14-44359c95f66c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mggrx"
Sep 16 18:42:13.837309 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.837263    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f801294c-a332-41af-bede-44bcd7a22e2c-tmp\") pod \"insights-operator-5844b856db-kb6s9\" (UID: \"f801294c-a332-41af-bede-44bcd7a22e2c\") " pod="openshift-insights/insights-operator-5844b856db-kb6s9"
Sep 16 18:42:13.837368 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.837317    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f801294c-a332-41af-bede-44bcd7a22e2c-serving-cert\") pod \"insights-operator-5844b856db-kb6s9\" (UID: \"f801294c-a332-41af-bede-44bcd7a22e2c\") " pod="openshift-insights/insights-operator-5844b856db-kb6s9"
Sep 16 18:42:13.837368 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.837346    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f801294c-a332-41af-bede-44bcd7a22e2c-service-ca-bundle\") pod \"insights-operator-5844b856db-kb6s9\" (UID: \"f801294c-a332-41af-bede-44bcd7a22e2c\") " pod="openshift-insights/insights-operator-5844b856db-kb6s9"
Sep 16 18:42:13.837480 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.837373    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f01001cd-d72f-4267-8e14-44359c95f66c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-mggrx\" (UID: \"f01001cd-d72f-4267-8e14-44359c95f66c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mggrx"
Sep 16 18:42:13.837480 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.837402    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7grs5\" (UniqueName: \"kubernetes.io/projected/f01001cd-d72f-4267-8e14-44359c95f66c-kube-api-access-7grs5\") pod \"cluster-monitoring-operator-6cd954b897-mggrx\" (UID: \"f01001cd-d72f-4267-8e14-44359c95f66c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mggrx"
Sep 16 18:42:13.837573 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:13.837504    2586 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:42:13.837573 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.837547    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f801294c-a332-41af-bede-44bcd7a22e2c-snapshots\") pod \"insights-operator-5844b856db-kb6s9\" (UID: \"f801294c-a332-41af-bede-44bcd7a22e2c\") " pod="openshift-insights/insights-operator-5844b856db-kb6s9"
Sep 16 18:42:13.837573 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:13.837568    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f01001cd-d72f-4267-8e14-44359c95f66c-cluster-monitoring-operator-tls podName:f01001cd-d72f-4267-8e14-44359c95f66c nodeName:}" failed. No retries permitted until 2026-09-16 18:42:14.337549734 +0000 UTC m=+109.781411310 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f01001cd-d72f-4267-8e14-44359c95f66c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-mggrx" (UID: "f01001cd-d72f-4267-8e14-44359c95f66c") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:42:13.837740 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.837644    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f801294c-a332-41af-bede-44bcd7a22e2c-trusted-ca-bundle\") pod \"insights-operator-5844b856db-kb6s9\" (UID: \"f801294c-a332-41af-bede-44bcd7a22e2c\") " pod="openshift-insights/insights-operator-5844b856db-kb6s9"
Sep 16 18:42:13.837740 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.837698    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f801294c-a332-41af-bede-44bcd7a22e2c-tmp\") pod \"insights-operator-5844b856db-kb6s9\" (UID: \"f801294c-a332-41af-bede-44bcd7a22e2c\") " pod="openshift-insights/insights-operator-5844b856db-kb6s9"
Sep 16 18:42:13.838303 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.838272    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f01001cd-d72f-4267-8e14-44359c95f66c-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-mggrx\" (UID: \"f01001cd-d72f-4267-8e14-44359c95f66c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mggrx"
Sep 16 18:42:13.839106 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.838455    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f801294c-a332-41af-bede-44bcd7a22e2c-snapshots\") pod \"insights-operator-5844b856db-kb6s9\" (UID: \"f801294c-a332-41af-bede-44bcd7a22e2c\") " pod="openshift-insights/insights-operator-5844b856db-kb6s9"
Sep 16 18:42:13.839106 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.838633    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f801294c-a332-41af-bede-44bcd7a22e2c-service-ca-bundle\") pod \"insights-operator-5844b856db-kb6s9\" (UID: \"f801294c-a332-41af-bede-44bcd7a22e2c\") " pod="openshift-insights/insights-operator-5844b856db-kb6s9"
Sep 16 18:42:13.839657 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.839534    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f801294c-a332-41af-bede-44bcd7a22e2c-trusted-ca-bundle\") pod \"insights-operator-5844b856db-kb6s9\" (UID: \"f801294c-a332-41af-bede-44bcd7a22e2c\") " pod="openshift-insights/insights-operator-5844b856db-kb6s9"
Sep 16 18:42:13.842097 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.839908    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f801294c-a332-41af-bede-44bcd7a22e2c-serving-cert\") pod \"insights-operator-5844b856db-kb6s9\" (UID: \"f801294c-a332-41af-bede-44bcd7a22e2c\") " pod="openshift-insights/insights-operator-5844b856db-kb6s9"
Sep 16 18:42:13.854538 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.854515    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwntn\" (UniqueName: \"kubernetes.io/projected/f801294c-a332-41af-bede-44bcd7a22e2c-kube-api-access-xwntn\") pod \"insights-operator-5844b856db-kb6s9\" (UID: \"f801294c-a332-41af-bede-44bcd7a22e2c\") " pod="openshift-insights/insights-operator-5844b856db-kb6s9"
Sep 16 18:42:13.855185 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.855164    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7grs5\" (UniqueName: \"kubernetes.io/projected/f01001cd-d72f-4267-8e14-44359c95f66c-kube-api-access-7grs5\") pod \"cluster-monitoring-operator-6cd954b897-mggrx\" (UID: \"f01001cd-d72f-4267-8e14-44359c95f66c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mggrx"
Sep 16 18:42:13.891171 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.891122    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-sh5xv"
Sep 16 18:42:13.896849 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.896831    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-kb6s9"
Sep 16 18:42:13.976491 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:13.976449    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-ps6sd"]
Sep 16 18:42:13.978901 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:42:13.978880    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ab787a9_4d8d_42f1_b7cb_432be6e37285.slice/crio-e6f91a9e723ef1ff40f3fa572551cbe79c110895873b18af7eec9d5a2d678506 WatchSource:0}: Error finding container e6f91a9e723ef1ff40f3fa572551cbe79c110895873b18af7eec9d5a2d678506: Status 404 returned error can't find the container with id e6f91a9e723ef1ff40f3fa572551cbe79c110895873b18af7eec9d5a2d678506
Sep 16 18:42:14.041391 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:14.041366    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1596b098-d098-43ae-bd52-b750f6fd3230-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-njkrg\" (UID: \"1596b098-d098-43ae-bd52-b750f6fd3230\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-njkrg"
Sep 16 18:42:14.041541 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:14.041520    2586 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:42:14.041632 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:14.041622    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1596b098-d098-43ae-bd52-b750f6fd3230-samples-operator-tls podName:1596b098-d098-43ae-bd52-b750f6fd3230 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:15.041606774 +0000 UTC m=+110.485468327 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1596b098-d098-43ae-bd52-b750f6fd3230-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-njkrg" (UID: "1596b098-d098-43ae-bd52-b750f6fd3230") : secret "samples-operator-tls" not found
Sep 16 18:42:14.142338 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:14.142265    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04e761f8-c6e9-4238-bd6b-0ee2db961991-metrics-certs\") pod \"router-default-986b9b8c8-9p8c8\" (UID: \"04e761f8-c6e9-4238-bd6b-0ee2db961991\") " pod="openshift-ingress/router-default-986b9b8c8-9p8c8"
Sep 16 18:42:14.142338 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:14.142306    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04e761f8-c6e9-4238-bd6b-0ee2db961991-service-ca-bundle\") pod \"router-default-986b9b8c8-9p8c8\" (UID: \"04e761f8-c6e9-4238-bd6b-0ee2db961991\") " pod="openshift-ingress/router-default-986b9b8c8-9p8c8"
Sep 16 18:42:14.142493 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:14.142438    2586 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:42:14.142493 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:14.142453    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/04e761f8-c6e9-4238-bd6b-0ee2db961991-service-ca-bundle podName:04e761f8-c6e9-4238-bd6b-0ee2db961991 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:15.142440572 +0000 UTC m=+110.586302126 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/04e761f8-c6e9-4238-bd6b-0ee2db961991-service-ca-bundle") pod "router-default-986b9b8c8-9p8c8" (UID: "04e761f8-c6e9-4238-bd6b-0ee2db961991") : configmap references non-existent config key: service-ca.crt
Sep 16 18:42:14.142596 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:14.142501    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04e761f8-c6e9-4238-bd6b-0ee2db961991-metrics-certs podName:04e761f8-c6e9-4238-bd6b-0ee2db961991 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:15.142483025 +0000 UTC m=+110.586344586 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/04e761f8-c6e9-4238-bd6b-0ee2db961991-metrics-certs") pod "router-default-986b9b8c8-9p8c8" (UID: "04e761f8-c6e9-4238-bd6b-0ee2db961991") : secret "router-metrics-certs-default" not found
Sep 16 18:42:14.162550 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:14.162530    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-sh5xv"]
Sep 16 18:42:14.164860 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:42:14.164835    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a6129d0_3d10_4f44_881a_632760e357fd.slice/crio-3ac2a04195f658c7419253cc6b6d608b975d43ea5427f100ae6a1c7ca986d1ca WatchSource:0}: Error finding container 3ac2a04195f658c7419253cc6b6d608b975d43ea5427f100ae6a1c7ca986d1ca: Status 404 returned error can't find the container with id 3ac2a04195f658c7419253cc6b6d608b975d43ea5427f100ae6a1c7ca986d1ca
Sep 16 18:42:14.166401 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:14.166382    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-kb6s9"]
Sep 16 18:42:14.169606 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:42:14.169579    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf801294c_a332_41af_bede_44bcd7a22e2c.slice/crio-f4a702350f5dab1b15fa157e221bfb84f59a22a45f603dc8c23429b8bfd79f94 WatchSource:0}: Error finding container f4a702350f5dab1b15fa157e221bfb84f59a22a45f603dc8c23429b8bfd79f94: Status 404 returned error can't find the container with id f4a702350f5dab1b15fa157e221bfb84f59a22a45f603dc8c23429b8bfd79f94
Sep 16 18:42:14.343758 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:14.343721    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f01001cd-d72f-4267-8e14-44359c95f66c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-mggrx\" (UID: \"f01001cd-d72f-4267-8e14-44359c95f66c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mggrx"
Sep 16 18:42:14.343912 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:14.343863    2586 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:42:14.343951 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:14.343926    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f01001cd-d72f-4267-8e14-44359c95f66c-cluster-monitoring-operator-tls podName:f01001cd-d72f-4267-8e14-44359c95f66c nodeName:}" failed. No retries permitted until 2026-09-16 18:42:15.343909496 +0000 UTC m=+110.787771057 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f01001cd-d72f-4267-8e14-44359c95f66c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-mggrx" (UID: "f01001cd-d72f-4267-8e14-44359c95f66c") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:42:14.473365 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:14.473272    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-kb6s9" event={"ID":"f801294c-a332-41af-bede-44bcd7a22e2c","Type":"ContainerStarted","Data":"f4a702350f5dab1b15fa157e221bfb84f59a22a45f603dc8c23429b8bfd79f94"}
Sep 16 18:42:14.474626 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:14.474593    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-ps6sd" event={"ID":"3ab787a9-4d8d-42f1-b7cb-432be6e37285","Type":"ContainerStarted","Data":"e6f91a9e723ef1ff40f3fa572551cbe79c110895873b18af7eec9d5a2d678506"}
Sep 16 18:42:14.475953 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:14.475931    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-sh5xv" event={"ID":"3a6129d0-3d10-4f44-881a-632760e357fd","Type":"ContainerStarted","Data":"3ac2a04195f658c7419253cc6b6d608b975d43ea5427f100ae6a1c7ca986d1ca"}
Sep 16 18:42:15.050012 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:15.049978    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1596b098-d098-43ae-bd52-b750f6fd3230-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-njkrg\" (UID: \"1596b098-d098-43ae-bd52-b750f6fd3230\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-njkrg"
Sep 16 18:42:15.050199 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:15.050172    2586 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:42:15.050284 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:15.050266    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1596b098-d098-43ae-bd52-b750f6fd3230-samples-operator-tls podName:1596b098-d098-43ae-bd52-b750f6fd3230 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:17.050245675 +0000 UTC m=+112.494107252 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1596b098-d098-43ae-bd52-b750f6fd3230-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-njkrg" (UID: "1596b098-d098-43ae-bd52-b750f6fd3230") : secret "samples-operator-tls" not found
Sep 16 18:42:15.151169 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:15.151135    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04e761f8-c6e9-4238-bd6b-0ee2db961991-metrics-certs\") pod \"router-default-986b9b8c8-9p8c8\" (UID: \"04e761f8-c6e9-4238-bd6b-0ee2db961991\") " pod="openshift-ingress/router-default-986b9b8c8-9p8c8"
Sep 16 18:42:15.151411 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:15.151191    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04e761f8-c6e9-4238-bd6b-0ee2db961991-service-ca-bundle\") pod \"router-default-986b9b8c8-9p8c8\" (UID: \"04e761f8-c6e9-4238-bd6b-0ee2db961991\") " pod="openshift-ingress/router-default-986b9b8c8-9p8c8"
Sep 16 18:42:15.151411 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:15.151360    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/04e761f8-c6e9-4238-bd6b-0ee2db961991-service-ca-bundle podName:04e761f8-c6e9-4238-bd6b-0ee2db961991 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:17.15133696 +0000 UTC m=+112.595198534 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/04e761f8-c6e9-4238-bd6b-0ee2db961991-service-ca-bundle") pod "router-default-986b9b8c8-9p8c8" (UID: "04e761f8-c6e9-4238-bd6b-0ee2db961991") : configmap references non-existent config key: service-ca.crt
Sep 16 18:42:15.152204 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:15.152181    2586 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:42:15.152341 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:15.152257    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04e761f8-c6e9-4238-bd6b-0ee2db961991-metrics-certs podName:04e761f8-c6e9-4238-bd6b-0ee2db961991 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:17.152222276 +0000 UTC m=+112.596083830 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/04e761f8-c6e9-4238-bd6b-0ee2db961991-metrics-certs") pod "router-default-986b9b8c8-9p8c8" (UID: "04e761f8-c6e9-4238-bd6b-0ee2db961991") : secret "router-metrics-certs-default" not found
Sep 16 18:42:15.353864 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:15.353824    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f01001cd-d72f-4267-8e14-44359c95f66c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-mggrx\" (UID: \"f01001cd-d72f-4267-8e14-44359c95f66c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mggrx"
Sep 16 18:42:15.354082 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:15.354065    2586 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:42:15.354181 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:15.354154    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f01001cd-d72f-4267-8e14-44359c95f66c-cluster-monitoring-operator-tls podName:f01001cd-d72f-4267-8e14-44359c95f66c nodeName:}" failed. No retries permitted until 2026-09-16 18:42:17.354138587 +0000 UTC m=+112.798000153 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f01001cd-d72f-4267-8e14-44359c95f66c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-mggrx" (UID: "f01001cd-d72f-4267-8e14-44359c95f66c") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:42:17.069636 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:17.069541    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1596b098-d098-43ae-bd52-b750f6fd3230-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-njkrg\" (UID: \"1596b098-d098-43ae-bd52-b750f6fd3230\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-njkrg"
Sep 16 18:42:17.070057 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:17.069705    2586 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:42:17.070057 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:17.069816    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1596b098-d098-43ae-bd52-b750f6fd3230-samples-operator-tls podName:1596b098-d098-43ae-bd52-b750f6fd3230 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:21.069795608 +0000 UTC m=+116.513657167 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1596b098-d098-43ae-bd52-b750f6fd3230-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-njkrg" (UID: "1596b098-d098-43ae-bd52-b750f6fd3230") : secret "samples-operator-tls" not found
Sep 16 18:42:17.175761 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:17.170887    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04e761f8-c6e9-4238-bd6b-0ee2db961991-metrics-certs\") pod \"router-default-986b9b8c8-9p8c8\" (UID: \"04e761f8-c6e9-4238-bd6b-0ee2db961991\") " pod="openshift-ingress/router-default-986b9b8c8-9p8c8"
Sep 16 18:42:17.175761 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:17.170959    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04e761f8-c6e9-4238-bd6b-0ee2db961991-service-ca-bundle\") pod \"router-default-986b9b8c8-9p8c8\" (UID: \"04e761f8-c6e9-4238-bd6b-0ee2db961991\") " pod="openshift-ingress/router-default-986b9b8c8-9p8c8"
Sep 16 18:42:17.175761 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:17.171445    2586 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:42:17.175761 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:17.171521    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04e761f8-c6e9-4238-bd6b-0ee2db961991-metrics-certs podName:04e761f8-c6e9-4238-bd6b-0ee2db961991 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:21.171497658 +0000 UTC m=+116.615359213 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/04e761f8-c6e9-4238-bd6b-0ee2db961991-metrics-certs") pod "router-default-986b9b8c8-9p8c8" (UID: "04e761f8-c6e9-4238-bd6b-0ee2db961991") : secret "router-metrics-certs-default" not found
Sep 16 18:42:17.175761 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:17.171723    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/04e761f8-c6e9-4238-bd6b-0ee2db961991-service-ca-bundle podName:04e761f8-c6e9-4238-bd6b-0ee2db961991 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:21.171708801 +0000 UTC m=+116.615570369 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/04e761f8-c6e9-4238-bd6b-0ee2db961991-service-ca-bundle") pod "router-default-986b9b8c8-9p8c8" (UID: "04e761f8-c6e9-4238-bd6b-0ee2db961991") : configmap references non-existent config key: service-ca.crt
Sep 16 18:42:17.372988 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:17.372952    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f01001cd-d72f-4267-8e14-44359c95f66c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-mggrx\" (UID: \"f01001cd-d72f-4267-8e14-44359c95f66c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mggrx"
Sep 16 18:42:17.373153 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:17.373133    2586 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:42:17.373242 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:17.373215    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f01001cd-d72f-4267-8e14-44359c95f66c-cluster-monitoring-operator-tls podName:f01001cd-d72f-4267-8e14-44359c95f66c nodeName:}" failed. No retries permitted until 2026-09-16 18:42:21.373194554 +0000 UTC m=+116.817056115 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f01001cd-d72f-4267-8e14-44359c95f66c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-mggrx" (UID: "f01001cd-d72f-4267-8e14-44359c95f66c") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:42:17.487112 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:17.487071    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-kb6s9" event={"ID":"f801294c-a332-41af-bede-44bcd7a22e2c","Type":"ContainerStarted","Data":"54d1ff7a498c35ede8927a6eb7f4d08ff2723d2e58a3fb9944902a2793d2b334"}
Sep 16 18:42:17.488829 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:17.488793    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-ps6sd" event={"ID":"3ab787a9-4d8d-42f1-b7cb-432be6e37285","Type":"ContainerStarted","Data":"686f76a96709dfc13833c83ae1a5c2451a061377af95265780d0f5c50b11af92"}
Sep 16 18:42:17.490388 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:17.490362    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-sh5xv" event={"ID":"3a6129d0-3d10-4f44-881a-632760e357fd","Type":"ContainerStarted","Data":"f2b0ddbdd93432bc1d1fb750ababbcebe53e20289771e58e5983a4d0ba945531"}
Sep 16 18:42:17.527619 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:17.527578    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-kb6s9" podStartSLOduration=1.897021685 podStartE2EDuration="4.527565237s" totalImagesPullingTime="2.630543552s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:17.525894067 +0000 UTC m=+112.969755648" watchObservedRunningTime="2026-09-16 18:42:17.527565237 +0000 UTC m=+112.971426812"
Sep 16 18:42:17.556215 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:17.556165    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-sh5xv" podStartSLOduration=1.921065219 podStartE2EDuration="4.556146294s" totalImagesPullingTime="2.635081075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:17.555692096 +0000 UTC m=+112.999553672" watchObservedRunningTime="2026-09-16 18:42:17.556146294 +0000 UTC m=+113.000007873"
Sep 16 18:42:17.587052 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:17.587001    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-ps6sd" podStartSLOduration=1.770640666 podStartE2EDuration="4.586983898s" totalImagesPullingTime="2.816343232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:17.58645538 +0000 UTC m=+113.030316954" watchObservedRunningTime="2026-09-16 18:42:17.586983898 +0000 UTC m=+113.030845476"
Sep 16 18:42:18.264964 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:18.264931    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ccfm2"]
Sep 16 18:42:18.267550 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:18.267535    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ccfm2"
Sep 16 18:42:18.270824 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:18.270794    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tvh6w\""
Sep 16 18:42:18.271164 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:18.271142    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 18:42:18.271164 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:18.271158    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 18:42:18.306660 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:18.306635    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ccfm2"]
Sep 16 18:42:18.382636 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:18.382602    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cfb2ecb2-e9bb-4c47-bd68-3a835f63365e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ccfm2\" (UID: \"cfb2ecb2-e9bb-4c47-bd68-3a835f63365e\") " pod="openshift-insights/insights-runtime-extractor-ccfm2"
Sep 16 18:42:18.382773 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:18.382726    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cfb2ecb2-e9bb-4c47-bd68-3a835f63365e-data-volume\") pod \"insights-runtime-extractor-ccfm2\" (UID: \"cfb2ecb2-e9bb-4c47-bd68-3a835f63365e\") " pod="openshift-insights/insights-runtime-extractor-ccfm2"
Sep 16 18:42:18.382773 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:18.382761    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m22ct\" (UniqueName: \"kubernetes.io/projected/cfb2ecb2-e9bb-4c47-bd68-3a835f63365e-kube-api-access-m22ct\") pod \"insights-runtime-extractor-ccfm2\" (UID: \"cfb2ecb2-e9bb-4c47-bd68-3a835f63365e\") " pod="openshift-insights/insights-runtime-extractor-ccfm2"
Sep 16 18:42:18.382867 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:18.382798    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cfb2ecb2-e9bb-4c47-bd68-3a835f63365e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ccfm2\" (UID: \"cfb2ecb2-e9bb-4c47-bd68-3a835f63365e\") " pod="openshift-insights/insights-runtime-extractor-ccfm2"
Sep 16 18:42:18.382867 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:18.382818    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cfb2ecb2-e9bb-4c47-bd68-3a835f63365e-crio-socket\") pod \"insights-runtime-extractor-ccfm2\" (UID: \"cfb2ecb2-e9bb-4c47-bd68-3a835f63365e\") " pod="openshift-insights/insights-runtime-extractor-ccfm2"
Sep 16 18:42:18.483464 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:18.483428    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cfb2ecb2-e9bb-4c47-bd68-3a835f63365e-data-volume\") pod \"insights-runtime-extractor-ccfm2\" (UID: \"cfb2ecb2-e9bb-4c47-bd68-3a835f63365e\") " pod="openshift-insights/insights-runtime-extractor-ccfm2"
Sep 16 18:42:18.483633 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:18.483475    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m22ct\" (UniqueName: \"kubernetes.io/projected/cfb2ecb2-e9bb-4c47-bd68-3a835f63365e-kube-api-access-m22ct\") pod \"insights-runtime-extractor-ccfm2\" (UID: \"cfb2ecb2-e9bb-4c47-bd68-3a835f63365e\") " pod="openshift-insights/insights-runtime-extractor-ccfm2"
Sep 16 18:42:18.483633 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:18.483535    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cfb2ecb2-e9bb-4c47-bd68-3a835f63365e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ccfm2\" (UID: \"cfb2ecb2-e9bb-4c47-bd68-3a835f63365e\") " pod="openshift-insights/insights-runtime-extractor-ccfm2"
Sep 16 18:42:18.483633 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:18.483570    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cfb2ecb2-e9bb-4c47-bd68-3a835f63365e-crio-socket\") pod \"insights-runtime-extractor-ccfm2\" (UID: \"cfb2ecb2-e9bb-4c47-bd68-3a835f63365e\") " pod="openshift-insights/insights-runtime-extractor-ccfm2"
Sep 16 18:42:18.483799 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:18.483658    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cfb2ecb2-e9bb-4c47-bd68-3a835f63365e-crio-socket\") pod \"insights-runtime-extractor-ccfm2\" (UID: \"cfb2ecb2-e9bb-4c47-bd68-3a835f63365e\") " pod="openshift-insights/insights-runtime-extractor-ccfm2"
Sep 16 18:42:18.483799 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:18.483666    2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:42:18.483799 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:18.483732    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cfb2ecb2-e9bb-4c47-bd68-3a835f63365e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ccfm2\" (UID: \"cfb2ecb2-e9bb-4c47-bd68-3a835f63365e\") " pod="openshift-insights/insights-runtime-extractor-ccfm2"
Sep 16 18:42:18.483930 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:18.483814    2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:42:18.483930 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:18.483847    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cfb2ecb2-e9bb-4c47-bd68-3a835f63365e-kube-rbac-proxy-cm podName:cfb2ecb2-e9bb-4c47-bd68-3a835f63365e nodeName:}" failed. No retries permitted until 2026-09-16 18:42:18.983821787 +0000 UTC m=+114.427683350 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cfb2ecb2-e9bb-4c47-bd68-3a835f63365e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ccfm2" (UID: "cfb2ecb2-e9bb-4c47-bd68-3a835f63365e") : configmap "kube-rbac-proxy" not found
Sep 16 18:42:18.483930 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:18.483876    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfb2ecb2-e9bb-4c47-bd68-3a835f63365e-insights-runtime-extractor-tls podName:cfb2ecb2-e9bb-4c47-bd68-3a835f63365e nodeName:}" failed. No retries permitted until 2026-09-16 18:42:18.983859865 +0000 UTC m=+114.427721442 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cfb2ecb2-e9bb-4c47-bd68-3a835f63365e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ccfm2" (UID: "cfb2ecb2-e9bb-4c47-bd68-3a835f63365e") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:42:18.484033 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:18.483960    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cfb2ecb2-e9bb-4c47-bd68-3a835f63365e-data-volume\") pod \"insights-runtime-extractor-ccfm2\" (UID: \"cfb2ecb2-e9bb-4c47-bd68-3a835f63365e\") " pod="openshift-insights/insights-runtime-extractor-ccfm2"
Sep 16 18:42:18.504479 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:18.504454    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m22ct\" (UniqueName: \"kubernetes.io/projected/cfb2ecb2-e9bb-4c47-bd68-3a835f63365e-kube-api-access-m22ct\") pod \"insights-runtime-extractor-ccfm2\" (UID: \"cfb2ecb2-e9bb-4c47-bd68-3a835f63365e\") " pod="openshift-insights/insights-runtime-extractor-ccfm2"
Sep 16 18:42:18.989025 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:18.988984    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cfb2ecb2-e9bb-4c47-bd68-3a835f63365e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ccfm2\" (UID: \"cfb2ecb2-e9bb-4c47-bd68-3a835f63365e\") " pod="openshift-insights/insights-runtime-extractor-ccfm2"
Sep 16 18:42:18.989279 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:18.989054    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cfb2ecb2-e9bb-4c47-bd68-3a835f63365e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ccfm2\" (UID: \"cfb2ecb2-e9bb-4c47-bd68-3a835f63365e\") " pod="openshift-insights/insights-runtime-extractor-ccfm2"
Sep 16 18:42:18.989279 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:18.989104    2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:42:18.989279 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:18.989171    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cfb2ecb2-e9bb-4c47-bd68-3a835f63365e-kube-rbac-proxy-cm podName:cfb2ecb2-e9bb-4c47-bd68-3a835f63365e nodeName:}" failed. No retries permitted until 2026-09-16 18:42:19.989156476 +0000 UTC m=+115.433018030 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cfb2ecb2-e9bb-4c47-bd68-3a835f63365e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ccfm2" (UID: "cfb2ecb2-e9bb-4c47-bd68-3a835f63365e") : configmap "kube-rbac-proxy" not found
Sep 16 18:42:18.989279 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:18.989199    2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:42:18.989419 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:18.989282    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfb2ecb2-e9bb-4c47-bd68-3a835f63365e-insights-runtime-extractor-tls podName:cfb2ecb2-e9bb-4c47-bd68-3a835f63365e nodeName:}" failed. No retries permitted until 2026-09-16 18:42:19.989267843 +0000 UTC m=+115.433129413 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cfb2ecb2-e9bb-4c47-bd68-3a835f63365e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ccfm2" (UID: "cfb2ecb2-e9bb-4c47-bd68-3a835f63365e") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:42:19.945442 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:19.945411    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cc7hj_b58ed079-b715-420e-a11f-2023c52e1321/dns-node-resolver/0.log"
Sep 16 18:42:19.996885 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:19.996847    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cfb2ecb2-e9bb-4c47-bd68-3a835f63365e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ccfm2\" (UID: \"cfb2ecb2-e9bb-4c47-bd68-3a835f63365e\") " pod="openshift-insights/insights-runtime-extractor-ccfm2"
Sep 16 18:42:19.997031 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:19.996950    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cfb2ecb2-e9bb-4c47-bd68-3a835f63365e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ccfm2\" (UID: \"cfb2ecb2-e9bb-4c47-bd68-3a835f63365e\") " pod="openshift-insights/insights-runtime-extractor-ccfm2"
Sep 16 18:42:19.997031 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:19.997003    2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:42:19.997100 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:19.997032    2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:42:19.997100 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:19.997066    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfb2ecb2-e9bb-4c47-bd68-3a835f63365e-insights-runtime-extractor-tls podName:cfb2ecb2-e9bb-4c47-bd68-3a835f63365e nodeName:}" failed. No retries permitted until 2026-09-16 18:42:21.997051792 +0000 UTC m=+117.440913346 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cfb2ecb2-e9bb-4c47-bd68-3a835f63365e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ccfm2" (UID: "cfb2ecb2-e9bb-4c47-bd68-3a835f63365e") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:42:19.997100 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:19.997080    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cfb2ecb2-e9bb-4c47-bd68-3a835f63365e-kube-rbac-proxy-cm podName:cfb2ecb2-e9bb-4c47-bd68-3a835f63365e nodeName:}" failed. No retries permitted until 2026-09-16 18:42:21.997073978 +0000 UTC m=+117.440935531 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cfb2ecb2-e9bb-4c47-bd68-3a835f63365e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ccfm2" (UID: "cfb2ecb2-e9bb-4c47-bd68-3a835f63365e") : configmap "kube-rbac-proxy" not found
Sep 16 18:42:20.860875 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:20.860843    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-lrr4h"]
Sep 16 18:42:20.863195 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:20.863175    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-lrr4h"
Sep 16 18:42:20.867788 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:20.867770    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-bv5xx\""
Sep 16 18:42:20.868369 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:20.868353    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 16 18:42:20.868689 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:20.868659    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 16 18:42:20.868784 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:20.868711    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 16 18:42:20.869311 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:20.869293    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 16 18:42:20.877607 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:20.877588    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-lrr4h"]
Sep 16 18:42:20.954112 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:20.954089    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6298x_7fe5b556-37c9-40c7-a09b-5c6116b43e6f/node-ca/0.log"
Sep 16 18:42:21.004706 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:21.004674    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/992570f4-886b-4233-b4e0-1ed8cd51c461-signing-key\") pod \"service-ca-568869fdc5-lrr4h\" (UID: \"992570f4-886b-4233-b4e0-1ed8cd51c461\") " pod="openshift-service-ca/service-ca-568869fdc5-lrr4h"
Sep 16 18:42:21.004840 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:21.004712    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl5nb\" (UniqueName: \"kubernetes.io/projected/992570f4-886b-4233-b4e0-1ed8cd51c461-kube-api-access-bl5nb\") pod \"service-ca-568869fdc5-lrr4h\" (UID: \"992570f4-886b-4233-b4e0-1ed8cd51c461\") " pod="openshift-service-ca/service-ca-568869fdc5-lrr4h"
Sep 16 18:42:21.004840 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:21.004731    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/992570f4-886b-4233-b4e0-1ed8cd51c461-signing-cabundle\") pod \"service-ca-568869fdc5-lrr4h\" (UID: \"992570f4-886b-4233-b4e0-1ed8cd51c461\") " pod="openshift-service-ca/service-ca-568869fdc5-lrr4h"
Sep 16 18:42:21.105838 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:21.105806    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/992570f4-886b-4233-b4e0-1ed8cd51c461-signing-key\") pod \"service-ca-568869fdc5-lrr4h\" (UID: \"992570f4-886b-4233-b4e0-1ed8cd51c461\") " pod="openshift-service-ca/service-ca-568869fdc5-lrr4h"
Sep 16 18:42:21.105988 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:21.105844    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl5nb\" (UniqueName: \"kubernetes.io/projected/992570f4-886b-4233-b4e0-1ed8cd51c461-kube-api-access-bl5nb\") pod \"service-ca-568869fdc5-lrr4h\" (UID: \"992570f4-886b-4233-b4e0-1ed8cd51c461\") " pod="openshift-service-ca/service-ca-568869fdc5-lrr4h"
Sep 16 18:42:21.105988 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:21.105863    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/992570f4-886b-4233-b4e0-1ed8cd51c461-signing-cabundle\") pod \"service-ca-568869fdc5-lrr4h\" (UID: \"992570f4-886b-4233-b4e0-1ed8cd51c461\") " pod="openshift-service-ca/service-ca-568869fdc5-lrr4h"
Sep 16 18:42:21.106080 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:21.105990    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1596b098-d098-43ae-bd52-b750f6fd3230-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-njkrg\" (UID: \"1596b098-d098-43ae-bd52-b750f6fd3230\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-njkrg"
Sep 16 18:42:21.106123 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:21.106102    2586 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 18:42:21.106191 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:21.106180    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1596b098-d098-43ae-bd52-b750f6fd3230-samples-operator-tls podName:1596b098-d098-43ae-bd52-b750f6fd3230 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:29.106161695 +0000 UTC m=+124.550023252 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1596b098-d098-43ae-bd52-b750f6fd3230-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-njkrg" (UID: "1596b098-d098-43ae-bd52-b750f6fd3230") : secret "samples-operator-tls" not found
Sep 16 18:42:21.106613 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:21.106586    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/992570f4-886b-4233-b4e0-1ed8cd51c461-signing-cabundle\") pod \"service-ca-568869fdc5-lrr4h\" (UID: \"992570f4-886b-4233-b4e0-1ed8cd51c461\") " pod="openshift-service-ca/service-ca-568869fdc5-lrr4h"
Sep 16 18:42:21.108322 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:21.108292    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/992570f4-886b-4233-b4e0-1ed8cd51c461-signing-key\") pod \"service-ca-568869fdc5-lrr4h\" (UID: \"992570f4-886b-4233-b4e0-1ed8cd51c461\") " pod="openshift-service-ca/service-ca-568869fdc5-lrr4h"
Sep 16 18:42:21.117079 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:21.117024    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl5nb\" (UniqueName: \"kubernetes.io/projected/992570f4-886b-4233-b4e0-1ed8cd51c461-kube-api-access-bl5nb\") pod \"service-ca-568869fdc5-lrr4h\" (UID: \"992570f4-886b-4233-b4e0-1ed8cd51c461\") " pod="openshift-service-ca/service-ca-568869fdc5-lrr4h"
Sep 16 18:42:21.173026 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:21.172993    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-lrr4h"
Sep 16 18:42:21.206627 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:21.206588    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04e761f8-c6e9-4238-bd6b-0ee2db961991-metrics-certs\") pod \"router-default-986b9b8c8-9p8c8\" (UID: \"04e761f8-c6e9-4238-bd6b-0ee2db961991\") " pod="openshift-ingress/router-default-986b9b8c8-9p8c8"
Sep 16 18:42:21.206767 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:21.206652    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04e761f8-c6e9-4238-bd6b-0ee2db961991-service-ca-bundle\") pod \"router-default-986b9b8c8-9p8c8\" (UID: \"04e761f8-c6e9-4238-bd6b-0ee2db961991\") " pod="openshift-ingress/router-default-986b9b8c8-9p8c8"
Sep 16 18:42:21.206767 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:21.206742    2586 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:42:21.206842 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:21.206815    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04e761f8-c6e9-4238-bd6b-0ee2db961991-metrics-certs podName:04e761f8-c6e9-4238-bd6b-0ee2db961991 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:29.206794432 +0000 UTC m=+124.650656004 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/04e761f8-c6e9-4238-bd6b-0ee2db961991-metrics-certs") pod "router-default-986b9b8c8-9p8c8" (UID: "04e761f8-c6e9-4238-bd6b-0ee2db961991") : secret "router-metrics-certs-default" not found
Sep 16 18:42:21.206842 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:21.206832    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/04e761f8-c6e9-4238-bd6b-0ee2db961991-service-ca-bundle podName:04e761f8-c6e9-4238-bd6b-0ee2db961991 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:29.206824319 +0000 UTC m=+124.650685873 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/04e761f8-c6e9-4238-bd6b-0ee2db961991-service-ca-bundle") pod "router-default-986b9b8c8-9p8c8" (UID: "04e761f8-c6e9-4238-bd6b-0ee2db961991") : configmap references non-existent config key: service-ca.crt
Sep 16 18:42:21.408632 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:21.408558    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f01001cd-d72f-4267-8e14-44359c95f66c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-mggrx\" (UID: \"f01001cd-d72f-4267-8e14-44359c95f66c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mggrx"
Sep 16 18:42:21.408773 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:21.408708    2586 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:42:21.408815 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:21.408777    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f01001cd-d72f-4267-8e14-44359c95f66c-cluster-monitoring-operator-tls podName:f01001cd-d72f-4267-8e14-44359c95f66c nodeName:}" failed. No retries permitted until 2026-09-16 18:42:29.40876118 +0000 UTC m=+124.852622734 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f01001cd-d72f-4267-8e14-44359c95f66c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-mggrx" (UID: "f01001cd-d72f-4267-8e14-44359c95f66c") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:42:21.446357 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:21.446335    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-lrr4h"]
Sep 16 18:42:21.448426 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:42:21.448404    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod992570f4_886b_4233_b4e0_1ed8cd51c461.slice/crio-004ce5fadd796f1ff7d02e90b1f7c9fe1a852af31d183642bfe645164f8d0f3d WatchSource:0}: Error finding container 004ce5fadd796f1ff7d02e90b1f7c9fe1a852af31d183642bfe645164f8d0f3d: Status 404 returned error can't find the container with id 004ce5fadd796f1ff7d02e90b1f7c9fe1a852af31d183642bfe645164f8d0f3d
Sep 16 18:42:21.505357 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:21.505332    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-lrr4h" event={"ID":"992570f4-886b-4233-b4e0-1ed8cd51c461","Type":"ContainerStarted","Data":"004ce5fadd796f1ff7d02e90b1f7c9fe1a852af31d183642bfe645164f8d0f3d"}
Sep 16 18:42:21.749514 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:21.749444    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cgs7w_a437fd80-c809-4981-9f3e-9cb3ce96a470/serve-healthcheck-canary/0.log"
Sep 16 18:42:22.014819 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:22.014731    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cfb2ecb2-e9bb-4c47-bd68-3a835f63365e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ccfm2\" (UID: \"cfb2ecb2-e9bb-4c47-bd68-3a835f63365e\") " pod="openshift-insights/insights-runtime-extractor-ccfm2"
Sep 16 18:42:22.015166 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:22.014825    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cfb2ecb2-e9bb-4c47-bd68-3a835f63365e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ccfm2\" (UID: \"cfb2ecb2-e9bb-4c47-bd68-3a835f63365e\") " pod="openshift-insights/insights-runtime-extractor-ccfm2"
Sep 16 18:42:22.015166 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:22.014857    2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:42:22.015166 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:22.014924    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cfb2ecb2-e9bb-4c47-bd68-3a835f63365e-kube-rbac-proxy-cm podName:cfb2ecb2-e9bb-4c47-bd68-3a835f63365e nodeName:}" failed. No retries permitted until 2026-09-16 18:42:26.014910585 +0000 UTC m=+121.458772143 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cfb2ecb2-e9bb-4c47-bd68-3a835f63365e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ccfm2" (UID: "cfb2ecb2-e9bb-4c47-bd68-3a835f63365e") : configmap "kube-rbac-proxy" not found
Sep 16 18:42:22.015166 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:22.014987    2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:42:22.015166 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:22.015049    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfb2ecb2-e9bb-4c47-bd68-3a835f63365e-insights-runtime-extractor-tls podName:cfb2ecb2-e9bb-4c47-bd68-3a835f63365e nodeName:}" failed. No retries permitted until 2026-09-16 18:42:26.015033792 +0000 UTC m=+121.458895346 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cfb2ecb2-e9bb-4c47-bd68-3a835f63365e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ccfm2" (UID: "cfb2ecb2-e9bb-4c47-bd68-3a835f63365e") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:42:22.510331 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:22.510296    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-lrr4h" event={"ID":"992570f4-886b-4233-b4e0-1ed8cd51c461","Type":"ContainerStarted","Data":"c252d5d98747d9648c7b21683cf378ef25f80884ca80da4f0f75cc74f0d554c7"}
Sep 16 18:42:22.553991 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:22.553945    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-lrr4h" podStartSLOduration=2.553930769 podStartE2EDuration="2.553930769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:22.553573947 +0000 UTC m=+117.997435524" watchObservedRunningTime="2026-09-16 18:42:22.553930769 +0000 UTC m=+117.997792345"
Sep 16 18:42:26.048491 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:26.048450    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cfb2ecb2-e9bb-4c47-bd68-3a835f63365e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ccfm2\" (UID: \"cfb2ecb2-e9bb-4c47-bd68-3a835f63365e\") " pod="openshift-insights/insights-runtime-extractor-ccfm2"
Sep 16 18:42:26.048944 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:26.048576    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cfb2ecb2-e9bb-4c47-bd68-3a835f63365e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ccfm2\" (UID: \"cfb2ecb2-e9bb-4c47-bd68-3a835f63365e\") " pod="openshift-insights/insights-runtime-extractor-ccfm2"
Sep 16 18:42:26.048944 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:26.048603    2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:42:26.048944 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:26.048676    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfb2ecb2-e9bb-4c47-bd68-3a835f63365e-insights-runtime-extractor-tls podName:cfb2ecb2-e9bb-4c47-bd68-3a835f63365e nodeName:}" failed. No retries permitted until 2026-09-16 18:42:34.048660028 +0000 UTC m=+129.492521791 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cfb2ecb2-e9bb-4c47-bd68-3a835f63365e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ccfm2" (UID: "cfb2ecb2-e9bb-4c47-bd68-3a835f63365e") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:42:26.048944 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:26.048685    2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:42:26.048944 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:26.048740    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cfb2ecb2-e9bb-4c47-bd68-3a835f63365e-kube-rbac-proxy-cm podName:cfb2ecb2-e9bb-4c47-bd68-3a835f63365e nodeName:}" failed. No retries permitted until 2026-09-16 18:42:34.048722873 +0000 UTC m=+129.492584450 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cfb2ecb2-e9bb-4c47-bd68-3a835f63365e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ccfm2" (UID: "cfb2ecb2-e9bb-4c47-bd68-3a835f63365e") : configmap "kube-rbac-proxy" not found
Sep 16 18:42:29.176103 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:29.176067    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1596b098-d098-43ae-bd52-b750f6fd3230-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-njkrg\" (UID: \"1596b098-d098-43ae-bd52-b750f6fd3230\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-njkrg"
Sep 16 18:42:29.178428 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:29.178404    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1596b098-d098-43ae-bd52-b750f6fd3230-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-njkrg\" (UID: \"1596b098-d098-43ae-bd52-b750f6fd3230\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-njkrg"
Sep 16 18:42:29.277437 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:29.277401    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04e761f8-c6e9-4238-bd6b-0ee2db961991-metrics-certs\") pod \"router-default-986b9b8c8-9p8c8\" (UID: \"04e761f8-c6e9-4238-bd6b-0ee2db961991\") " pod="openshift-ingress/router-default-986b9b8c8-9p8c8"
Sep 16 18:42:29.277588 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:29.277449    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04e761f8-c6e9-4238-bd6b-0ee2db961991-service-ca-bundle\") pod \"router-default-986b9b8c8-9p8c8\" (UID: \"04e761f8-c6e9-4238-bd6b-0ee2db961991\") " pod="openshift-ingress/router-default-986b9b8c8-9p8c8"
Sep 16 18:42:29.277662 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:29.277644    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/04e761f8-c6e9-4238-bd6b-0ee2db961991-service-ca-bundle podName:04e761f8-c6e9-4238-bd6b-0ee2db961991 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:45.277625865 +0000 UTC m=+140.721487419 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/04e761f8-c6e9-4238-bd6b-0ee2db961991-service-ca-bundle") pod "router-default-986b9b8c8-9p8c8" (UID: "04e761f8-c6e9-4238-bd6b-0ee2db961991") : configmap references non-existent config key: service-ca.crt
Sep 16 18:42:29.279803 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:29.279774    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04e761f8-c6e9-4238-bd6b-0ee2db961991-metrics-certs\") pod \"router-default-986b9b8c8-9p8c8\" (UID: \"04e761f8-c6e9-4238-bd6b-0ee2db961991\") " pod="openshift-ingress/router-default-986b9b8c8-9p8c8"
Sep 16 18:42:29.293771 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:29.293749    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-j6w2l\""
Sep 16 18:42:29.301448 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:29.301431    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-njkrg"
Sep 16 18:42:29.479651 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:29.479567    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f01001cd-d72f-4267-8e14-44359c95f66c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-mggrx\" (UID: \"f01001cd-d72f-4267-8e14-44359c95f66c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mggrx"
Sep 16 18:42:29.479804 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:29.479749    2586 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:42:29.479874 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:29.479834    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f01001cd-d72f-4267-8e14-44359c95f66c-cluster-monitoring-operator-tls podName:f01001cd-d72f-4267-8e14-44359c95f66c nodeName:}" failed. No retries permitted until 2026-09-16 18:42:45.479810188 +0000 UTC m=+140.923671765 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f01001cd-d72f-4267-8e14-44359c95f66c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-mggrx" (UID: "f01001cd-d72f-4267-8e14-44359c95f66c") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:42:29.575196 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:29.575164    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-njkrg"]
Sep 16 18:42:30.545266 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:30.545211    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-njkrg" event={"ID":"1596b098-d098-43ae-bd52-b750f6fd3230","Type":"ContainerStarted","Data":"df63fde815e5f5e95c40f18ebef7c6752824444b0a4bff15b68523febe283939"}
Sep 16 18:42:31.550428 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:31.550346    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-njkrg" event={"ID":"1596b098-d098-43ae-bd52-b750f6fd3230","Type":"ContainerStarted","Data":"7e2272ab40cc216216c5b928a3d2d2561694e9bd8a659b2b8427ce3ec8b6d40e"}
Sep 16 18:42:31.550428 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:31.550403    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-njkrg" event={"ID":"1596b098-d098-43ae-bd52-b750f6fd3230","Type":"ContainerStarted","Data":"33f3b0b1de96b0743ca9207b8844620f03a7c4364e9a12a75f79719d85ca4979"}
Sep 16 18:42:31.577442 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:31.577384    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-njkrg" podStartSLOduration=17.080978588 podStartE2EDuration="18.577364538s" totalImagesPullingTime="1.49638595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:31.577100248 +0000 UTC m=+127.020961828" watchObservedRunningTime="2026-09-16 18:42:31.577364538 +0000 UTC m=+127.021226115"
Sep 16 18:42:34.119150 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:34.119112    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cfb2ecb2-e9bb-4c47-bd68-3a835f63365e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ccfm2\" (UID: \"cfb2ecb2-e9bb-4c47-bd68-3a835f63365e\") " pod="openshift-insights/insights-runtime-extractor-ccfm2"
Sep 16 18:42:34.119568 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:34.119206    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cfb2ecb2-e9bb-4c47-bd68-3a835f63365e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ccfm2\" (UID: \"cfb2ecb2-e9bb-4c47-bd68-3a835f63365e\") " pod="openshift-insights/insights-runtime-extractor-ccfm2"
Sep 16 18:42:34.119857 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:34.119834    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cfb2ecb2-e9bb-4c47-bd68-3a835f63365e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ccfm2\" (UID: \"cfb2ecb2-e9bb-4c47-bd68-3a835f63365e\") " pod="openshift-insights/insights-runtime-extractor-ccfm2"
Sep 16 18:42:34.121562 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:34.121540    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cfb2ecb2-e9bb-4c47-bd68-3a835f63365e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ccfm2\" (UID: \"cfb2ecb2-e9bb-4c47-bd68-3a835f63365e\") " pod="openshift-insights/insights-runtime-extractor-ccfm2"
Sep 16 18:42:34.179280 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:34.179250    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tvh6w\""
Sep 16 18:42:34.187188 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:34.187171    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ccfm2"
Sep 16 18:42:34.456556 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:34.456482    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ccfm2"]
Sep 16 18:42:34.459276 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:42:34.459224    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfb2ecb2_e9bb_4c47_bd68_3a835f63365e.slice/crio-d15fc5301fe5a1a8c7a9cf0034a6867cca618efd3ba0bf93b61293517dea007d WatchSource:0}: Error finding container d15fc5301fe5a1a8c7a9cf0034a6867cca618efd3ba0bf93b61293517dea007d: Status 404 returned error can't find the container with id d15fc5301fe5a1a8c7a9cf0034a6867cca618efd3ba0bf93b61293517dea007d
Sep 16 18:42:34.561025 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:34.560990    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ccfm2" event={"ID":"cfb2ecb2-e9bb-4c47-bd68-3a835f63365e","Type":"ContainerStarted","Data":"e57aec77d388cc40e47f2bc3253db91fc0f9f7765b5480ec5405c133298eb9ad"}
Sep 16 18:42:34.561146 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:34.561078    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ccfm2" event={"ID":"cfb2ecb2-e9bb-4c47-bd68-3a835f63365e","Type":"ContainerStarted","Data":"d15fc5301fe5a1a8c7a9cf0034a6867cca618efd3ba0bf93b61293517dea007d"}
Sep 16 18:42:34.824921 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:34.824833    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e8169e6-6afe-417b-90f0-5a55bf294956-metrics-certs\") pod \"network-metrics-daemon-9vpbt\" (UID: \"1e8169e6-6afe-417b-90f0-5a55bf294956\") " pod="openshift-multus/network-metrics-daemon-9vpbt"
Sep 16 18:42:34.827101 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:34.827076    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e8169e6-6afe-417b-90f0-5a55bf294956-metrics-certs\") pod \"network-metrics-daemon-9vpbt\" (UID: \"1e8169e6-6afe-417b-90f0-5a55bf294956\") " pod="openshift-multus/network-metrics-daemon-9vpbt"
Sep 16 18:42:35.098004 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:35.097978    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-564xp\""
Sep 16 18:42:35.106054 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:35.106035    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vpbt"
Sep 16 18:42:35.372716 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:35.372693    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9vpbt"]
Sep 16 18:42:35.377696 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:42:35.377661    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e8169e6_6afe_417b_90f0_5a55bf294956.slice/crio-3398646c57ace936bc8b6e5ec7afcdad6c6a2deb26b4e6af8ab8fa31d29e8a6e WatchSource:0}: Error finding container 3398646c57ace936bc8b6e5ec7afcdad6c6a2deb26b4e6af8ab8fa31d29e8a6e: Status 404 returned error can't find the container with id 3398646c57ace936bc8b6e5ec7afcdad6c6a2deb26b4e6af8ab8fa31d29e8a6e
Sep 16 18:42:35.565157 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:35.565111    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9vpbt" event={"ID":"1e8169e6-6afe-417b-90f0-5a55bf294956","Type":"ContainerStarted","Data":"3398646c57ace936bc8b6e5ec7afcdad6c6a2deb26b4e6af8ab8fa31d29e8a6e"}
Sep 16 18:42:35.566931 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:35.566904    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ccfm2" event={"ID":"cfb2ecb2-e9bb-4c47-bd68-3a835f63365e","Type":"ContainerStarted","Data":"1aef7b2ea66902059d113a00e215a05d09a350f836c9bde07c914e9eb1049bbf"}
Sep 16 18:42:37.575402 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:37.575367    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ccfm2" event={"ID":"cfb2ecb2-e9bb-4c47-bd68-3a835f63365e","Type":"ContainerStarted","Data":"361422f242bbbbc4898c1a15e6de8191f71c5526caf6f71b8067a5a47eacd209"}
Sep 16 18:42:37.577015 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:37.576990    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9vpbt" event={"ID":"1e8169e6-6afe-417b-90f0-5a55bf294956","Type":"ContainerStarted","Data":"62e4c7cf5ee308bc813033b53a04be6c62b2d70583df1b633e68c78e08bd8f90"}
Sep 16 18:42:37.577131 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:37.577036    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9vpbt" event={"ID":"1e8169e6-6afe-417b-90f0-5a55bf294956","Type":"ContainerStarted","Data":"888fd4db24b754f4578434cd40ecb34e2e4de6bc63a4cb58c5fda1f98703b42d"}
Sep 16 18:42:37.603100 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:37.603037    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-ccfm2" podStartSLOduration=17.455910768 podStartE2EDuration="19.603017312s" totalImagesPullingTime="2.147106544s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:42:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:37.601492921 +0000 UTC m=+133.045354508" watchObservedRunningTime="2026-09-16 18:42:37.603017312 +0000 UTC m=+133.046878888"
Sep 16 18:42:37.627345 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:37.627298    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9vpbt" podStartSLOduration=131.29570777 podStartE2EDuration="2m12.627283971s" totalImagesPullingTime="1.331576201s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:37.626142297 +0000 UTC m=+133.070003867" watchObservedRunningTime="2026-09-16 18:42:37.627283971 +0000 UTC m=+133.071145547"
Sep 16 18:42:41.027840 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:41.027804    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-cjdzd"]
Sep 16 18:42:41.030910 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:41.030893    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-cjdzd"
Sep 16 18:42:41.037845 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:41.037825    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-gdxv4\""
Sep 16 18:42:41.047713 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:41.047685    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-cjdzd"]
Sep 16 18:42:41.069087 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:41.069056    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqvz2\" (UniqueName: \"kubernetes.io/projected/1251b6fc-227d-4f5d-be85-61be0d79fbbf-kube-api-access-fqvz2\") pod \"network-check-source-6f8b768455-cjdzd\" (UID: \"1251b6fc-227d-4f5d-be85-61be0d79fbbf\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-cjdzd"
Sep 16 18:42:41.136780 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:41.136747    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-wd4p9"]
Sep 16 18:42:41.139954 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:41.139931    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-wd4p9"
Sep 16 18:42:41.143972 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:41.143949    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 18:42:41.144316 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:41.144300    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-6psm9\""
Sep 16 18:42:41.145956 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:41.145938    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 18:42:41.163320 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:41.163295    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-wd4p9"]
Sep 16 18:42:41.170478 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:41.170451    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjnbq\" (UniqueName: \"kubernetes.io/projected/0fb1d7e8-fe62-4665-b5a8-8bcbd0eff71e-kube-api-access-fjnbq\") pod \"downloads-76f99584f4-wd4p9\" (UID: \"0fb1d7e8-fe62-4665-b5a8-8bcbd0eff71e\") " pod="openshift-console/downloads-76f99584f4-wd4p9"
Sep 16 18:42:41.170598 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:41.170542    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqvz2\" (UniqueName: \"kubernetes.io/projected/1251b6fc-227d-4f5d-be85-61be0d79fbbf-kube-api-access-fqvz2\") pod \"network-check-source-6f8b768455-cjdzd\" (UID: \"1251b6fc-227d-4f5d-be85-61be0d79fbbf\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-cjdzd"
Sep 16 18:42:41.186701 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:41.186667    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqvz2\" (UniqueName: \"kubernetes.io/projected/1251b6fc-227d-4f5d-be85-61be0d79fbbf-kube-api-access-fqvz2\") pod \"network-check-source-6f8b768455-cjdzd\" (UID: \"1251b6fc-227d-4f5d-be85-61be0d79fbbf\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-cjdzd"
Sep 16 18:42:41.223119 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:41.223091    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-64ccc449b-b924q"]
Sep 16 18:42:41.226352 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:41.226334    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64ccc449b-b924q"
Sep 16 18:42:41.229322 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:41.229298    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 18:42:41.229509 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:41.229494    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-gq4nt\""
Sep 16 18:42:41.229935 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:41.229918    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 18:42:41.230355 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:41.230339    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 18:42:41.239176 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:41.239152    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 18:42:41.250152 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:41.250132    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64ccc449b-b924q"]
Sep 16 18:42:41.271262 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:41.271217    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4c90494c-a26f-4e94-9ec4-61e7453a659b-registry-certificates\") pod \"image-registry-64ccc449b-b924q\" (UID: \"4c90494c-a26f-4e94-9ec4-61e7453a659b\") " pod="openshift-image-registry/image-registry-64ccc449b-b924q"
Sep 16 18:42:41.271382 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:41.271273    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4c90494c-a26f-4e94-9ec4-61e7453a659b-trusted-ca\") pod \"image-registry-64ccc449b-b924q\" (UID: \"4c90494c-a26f-4e94-9ec4-61e7453a659b\") " pod="openshift-image-registry/image-registry-64ccc449b-b924q"
Sep 16 18:42:41.271382 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:41.271357    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4c90494c-a26f-4e94-9ec4-61e7453a659b-registry-tls\") pod \"image-registry-64ccc449b-b924q\" (UID: \"4c90494c-a26f-4e94-9ec4-61e7453a659b\") " pod="openshift-image-registry/image-registry-64ccc449b-b924q"
Sep 16 18:42:41.271463 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:41.271412    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4c90494c-a26f-4e94-9ec4-61e7453a659b-image-registry-private-configuration\") pod \"image-registry-64ccc449b-b924q\" (UID: \"4c90494c-a26f-4e94-9ec4-61e7453a659b\") " pod="openshift-image-registry/image-registry-64ccc449b-b924q"
Sep 16 18:42:41.271463 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:41.271444    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjnbq\" (UniqueName: \"kubernetes.io/projected/0fb1d7e8-fe62-4665-b5a8-8bcbd0eff71e-kube-api-access-fjnbq\") pod \"downloads-76f99584f4-wd4p9\" (UID: \"0fb1d7e8-fe62-4665-b5a8-8bcbd0eff71e\") " pod="openshift-console/downloads-76f99584f4-wd4p9"
Sep 16 18:42:41.271540 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:41.271460    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4c90494c-a26f-4e94-9ec4-61e7453a659b-installation-pull-secrets\") pod \"image-registry-64ccc449b-b924q\" (UID: \"4c90494c-a26f-4e94-9ec4-61e7453a659b\") " pod="openshift-image-registry/image-registry-64ccc449b-b924q"
Sep 16 18:42:41.271540 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:41.271489    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkr74\" (UniqueName: \"kubernetes.io/projected/4c90494c-a26f-4e94-9ec4-61e7453a659b-kube-api-access-bkr74\") pod \"image-registry-64ccc449b-b924q\" (UID: \"4c90494c-a26f-4e94-9ec4-61e7453a659b\") " pod="openshift-image-registry/image-registry-64ccc449b-b924q"
Sep 16 18:42:41.271629 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:41.271616    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4c90494c-a26f-4e94-9ec4-61e7453a659b-ca-trust-extracted\") pod \"image-registry-64ccc449b-b924q\" (UID: \"4c90494c-a26f-4e94-9ec4-61e7453a659b\") " pod="openshift-image-registry/image-registry-64ccc449b-b924q"
Sep 16 18:42:41.271666 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:41.271648    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4c90494c-a26f-4e94-9ec4-61e7453a659b-bound-sa-token\") pod \"image-registry-64ccc449b-b924q\" (UID: \"4c90494c-a26f-4e94-9ec4-61e7453a659b\") " pod="openshift-image-registry/image-registry-64ccc449b-b924q"
Sep 16 18:42:41.288688 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:41.288638    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjnbq\" (UniqueName: \"kubernetes.io/projected/0fb1d7e8-fe62-4665-b5a8-8bcbd0eff71e-kube-api-access-fjnbq\") pod \"downloads-76f99584f4-wd4p9\" (UID: \"0fb1d7e8-fe62-4665-b5a8-8bcbd0eff71e\") " pod="openshift-console/downloads-76f99584f4-wd4p9"
Sep 16 18:42:41.339391 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:41.339360    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-cjdzd"
Sep 16 18:42:41.372369 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:41.372341    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4c90494c-a26f-4e94-9ec4-61e7453a659b-image-registry-private-configuration\") pod \"image-registry-64ccc449b-b924q\" (UID: \"4c90494c-a26f-4e94-9ec4-61e7453a659b\") " pod="openshift-image-registry/image-registry-64ccc449b-b924q"
Sep 16 18:42:41.372520 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:41.372375    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4c90494c-a26f-4e94-9ec4-61e7453a659b-installation-pull-secrets\") pod \"image-registry-64ccc449b-b924q\" (UID: \"4c90494c-a26f-4e94-9ec4-61e7453a659b\") " pod="openshift-image-registry/image-registry-64ccc449b-b924q"
Sep 16 18:42:41.372520 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:41.372403    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkr74\" (UniqueName: \"kubernetes.io/projected/4c90494c-a26f-4e94-9ec4-61e7453a659b-kube-api-access-bkr74\") pod \"image-registry-64ccc449b-b924q\" (UID: \"4c90494c-a26f-4e94-9ec4-61e7453a659b\") " pod="openshift-image-registry/image-registry-64ccc449b-b924q"
Sep 16 18:42:41.372520 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:41.372444    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4c90494c-a26f-4e94-9ec4-61e7453a659b-ca-trust-extracted\") pod \"image-registry-64ccc449b-b924q\" (UID: \"4c90494c-a26f-4e94-9ec4-61e7453a659b\") " pod="openshift-image-registry/image-registry-64ccc449b-b924q"
Sep 16 18:42:41.372520 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:41.372481    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4c90494c-a26f-4e94-9ec4-61e7453a659b-bound-sa-token\") pod \"image-registry-64ccc449b-b924q\" (UID: \"4c90494c-a26f-4e94-9ec4-61e7453a659b\") " pod="openshift-image-registry/image-registry-64ccc449b-b924q"
Sep 16 18:42:41.372748 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:41.372520    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4c90494c-a26f-4e94-9ec4-61e7453a659b-registry-certificates\") pod \"image-registry-64ccc449b-b924q\" (UID: \"4c90494c-a26f-4e94-9ec4-61e7453a659b\") " pod="openshift-image-registry/image-registry-64ccc449b-b924q"
Sep 16 18:42:41.372748 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:41.372558    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4c90494c-a26f-4e94-9ec4-61e7453a659b-trusted-ca\") pod \"image-registry-64ccc449b-b924q\" (UID: \"4c90494c-a26f-4e94-9ec4-61e7453a659b\") " pod="openshift-image-registry/image-registry-64ccc449b-b924q"
Sep 16 18:42:41.372748 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:41.372605    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4c90494c-a26f-4e94-9ec4-61e7453a659b-registry-tls\") pod \"image-registry-64ccc449b-b924q\" (UID: \"4c90494c-a26f-4e94-9ec4-61e7453a659b\") " pod="openshift-image-registry/image-registry-64ccc449b-b924q"
Sep 16 18:42:41.372949 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:41.372911    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4c90494c-a26f-4e94-9ec4-61e7453a659b-ca-trust-extracted\") pod \"image-registry-64ccc449b-b924q\" (UID: \"4c90494c-a26f-4e94-9ec4-61e7453a659b\") " pod="openshift-image-registry/image-registry-64ccc449b-b924q"
Sep 16 18:42:41.373614 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:41.373589    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4c90494c-a26f-4e94-9ec4-61e7453a659b-registry-certificates\") pod \"image-registry-64ccc449b-b924q\" (UID: \"4c90494c-a26f-4e94-9ec4-61e7453a659b\") " pod="openshift-image-registry/image-registry-64ccc449b-b924q"
Sep 16 18:42:41.373785 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:41.373769    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4c90494c-a26f-4e94-9ec4-61e7453a659b-trusted-ca\") pod \"image-registry-64ccc449b-b924q\" (UID: \"4c90494c-a26f-4e94-9ec4-61e7453a659b\") " pod="openshift-image-registry/image-registry-64ccc449b-b924q"
Sep 16 18:42:41.375066 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:41.375036    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4c90494c-a26f-4e94-9ec4-61e7453a659b-image-registry-private-configuration\") pod \"image-registry-64ccc449b-b924q\" (UID: \"4c90494c-a26f-4e94-9ec4-61e7453a659b\") " pod="openshift-image-registry/image-registry-64ccc449b-b924q"
Sep 16 18:42:41.375279 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:41.375238    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4c90494c-a26f-4e94-9ec4-61e7453a659b-registry-tls\") pod \"image-registry-64ccc449b-b924q\" (UID: \"4c90494c-a26f-4e94-9ec4-61e7453a659b\") " pod="openshift-image-registry/image-registry-64ccc449b-b924q"
Sep 16 18:42:41.375364 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:41.375317    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4c90494c-a26f-4e94-9ec4-61e7453a659b-installation-pull-secrets\") pod \"image-registry-64ccc449b-b924q\" (UID: \"4c90494c-a26f-4e94-9ec4-61e7453a659b\") " pod="openshift-image-registry/image-registry-64ccc449b-b924q"
Sep 16 18:42:41.385200 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:41.385141    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4c90494c-a26f-4e94-9ec4-61e7453a659b-bound-sa-token\") pod \"image-registry-64ccc449b-b924q\" (UID: \"4c90494c-a26f-4e94-9ec4-61e7453a659b\") " pod="openshift-image-registry/image-registry-64ccc449b-b924q"
Sep 16 18:42:41.385657 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:41.385605    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkr74\" (UniqueName: \"kubernetes.io/projected/4c90494c-a26f-4e94-9ec4-61e7453a659b-kube-api-access-bkr74\") pod \"image-registry-64ccc449b-b924q\" (UID: \"4c90494c-a26f-4e94-9ec4-61e7453a659b\") " pod="openshift-image-registry/image-registry-64ccc449b-b924q"
Sep 16 18:42:41.448864 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:41.448831    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-wd4p9"
Sep 16 18:42:41.535408 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:41.535381    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64ccc449b-b924q"
Sep 16 18:42:41.614915 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:41.614885    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-cjdzd"]
Sep 16 18:42:41.617881 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:42:41.617854    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1251b6fc_227d_4f5d_be85_61be0d79fbbf.slice/crio-03555b11dd2d849d4b8905d9857af635d74b638c3d552c0a44eb0ff957ae9603 WatchSource:0}: Error finding container 03555b11dd2d849d4b8905d9857af635d74b638c3d552c0a44eb0ff957ae9603: Status 404 returned error can't find the container with id 03555b11dd2d849d4b8905d9857af635d74b638c3d552c0a44eb0ff957ae9603
Sep 16 18:42:41.723174 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:41.723141    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-wd4p9"]
Sep 16 18:42:41.727591 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:42:41.727570    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fb1d7e8_fe62_4665_b5a8_8bcbd0eff71e.slice/crio-bde4ca06e2b75569b4c654d585d2bebf7c3c27883052c54335716fd50f9884e5 WatchSource:0}: Error finding container bde4ca06e2b75569b4c654d585d2bebf7c3c27883052c54335716fd50f9884e5: Status 404 returned error can't find the container with id bde4ca06e2b75569b4c654d585d2bebf7c3c27883052c54335716fd50f9884e5
Sep 16 18:42:41.811142 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:41.811062    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64ccc449b-b924q"]
Sep 16 18:42:41.815603 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:42:41.815577    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c90494c_a26f_4e94_9ec4_61e7453a659b.slice/crio-044796a6e5b277fc9774f4c659157654856091ecb25484e5693217dfa30bd86a WatchSource:0}: Error finding container 044796a6e5b277fc9774f4c659157654856091ecb25484e5693217dfa30bd86a: Status 404 returned error can't find the container with id 044796a6e5b277fc9774f4c659157654856091ecb25484e5693217dfa30bd86a
Sep 16 18:42:42.595738 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:42.595687    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-wd4p9" event={"ID":"0fb1d7e8-fe62-4665-b5a8-8bcbd0eff71e","Type":"ContainerStarted","Data":"bde4ca06e2b75569b4c654d585d2bebf7c3c27883052c54335716fd50f9884e5"}
Sep 16 18:42:42.597504 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:42.597470    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64ccc449b-b924q" event={"ID":"4c90494c-a26f-4e94-9ec4-61e7453a659b","Type":"ContainerStarted","Data":"fbde7d7b08d57fdae07f43cbffc52a89c09a0981357fe70646e626872520f5e7"}
Sep 16 18:42:42.597627 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:42.597513    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64ccc449b-b924q" event={"ID":"4c90494c-a26f-4e94-9ec4-61e7453a659b","Type":"ContainerStarted","Data":"044796a6e5b277fc9774f4c659157654856091ecb25484e5693217dfa30bd86a"}
Sep 16 18:42:42.597627 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:42.597585    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-64ccc449b-b924q"
Sep 16 18:42:42.599514 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:42.599489    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-cjdzd" event={"ID":"1251b6fc-227d-4f5d-be85-61be0d79fbbf","Type":"ContainerStarted","Data":"2cf5b79bb062345a2c9f97623266b73dcf9f0360f093ccc498870b1cdeba7fd5"}
Sep 16 18:42:42.599628 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:42.599535    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-cjdzd" event={"ID":"1251b6fc-227d-4f5d-be85-61be0d79fbbf","Type":"ContainerStarted","Data":"03555b11dd2d849d4b8905d9857af635d74b638c3d552c0a44eb0ff957ae9603"}
Sep 16 18:42:42.624381 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:42.624332    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-64ccc449b-b924q" podStartSLOduration=1.624313693 podStartE2EDuration="1.624313693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:42.624069893 +0000 UTC m=+138.067931468" watchObservedRunningTime="2026-09-16 18:42:42.624313693 +0000 UTC m=+138.068175261"
Sep 16 18:42:42.649287 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:42.649210    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-cjdzd" podStartSLOduration=1.6491915449999999 podStartE2EDuration="1.649191545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:42.647777049 +0000 UTC m=+138.091638626" watchObservedRunningTime="2026-09-16 18:42:42.649191545 +0000 UTC m=+138.093053125"
Sep 16 18:42:45.307141 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:45.307105    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04e761f8-c6e9-4238-bd6b-0ee2db961991-service-ca-bundle\") pod \"router-default-986b9b8c8-9p8c8\" (UID: \"04e761f8-c6e9-4238-bd6b-0ee2db961991\") " pod="openshift-ingress/router-default-986b9b8c8-9p8c8"
Sep 16 18:42:45.307854 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:45.307829    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04e761f8-c6e9-4238-bd6b-0ee2db961991-service-ca-bundle\") pod \"router-default-986b9b8c8-9p8c8\" (UID: \"04e761f8-c6e9-4238-bd6b-0ee2db961991\") " pod="openshift-ingress/router-default-986b9b8c8-9p8c8"
Sep 16 18:42:45.505917 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:45.505876    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-x5qmd\""
Sep 16 18:42:45.508252 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:45.508177    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f01001cd-d72f-4267-8e14-44359c95f66c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-mggrx\" (UID: \"f01001cd-d72f-4267-8e14-44359c95f66c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mggrx"
Sep 16 18:42:45.511028 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:45.511001    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f01001cd-d72f-4267-8e14-44359c95f66c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-mggrx\" (UID: \"f01001cd-d72f-4267-8e14-44359c95f66c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mggrx"
Sep 16 18:42:45.513726 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:45.513704    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-986b9b8c8-9p8c8"
Sep 16 18:42:45.704223 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:45.704192    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-sw8fc\""
Sep 16 18:42:45.712437 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:45.712412    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mggrx"
Sep 16 18:42:45.802221 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:45.802188    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-986b9b8c8-9p8c8"]
Sep 16 18:42:45.807490 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:42:45.807458    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04e761f8_c6e9_4238_bd6b_0ee2db961991.slice/crio-24f3d781a834c3097f106067f1aeb758c42eb2add4f11a41ad5c253c7654c05a WatchSource:0}: Error finding container 24f3d781a834c3097f106067f1aeb758c42eb2add4f11a41ad5c253c7654c05a: Status 404 returned error can't find the container with id 24f3d781a834c3097f106067f1aeb758c42eb2add4f11a41ad5c253c7654c05a
Sep 16 18:42:46.006862 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:46.006824    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-mggrx"]
Sep 16 18:42:46.009765 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:42:46.009741    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf01001cd_d72f_4267_8e14_44359c95f66c.slice/crio-b9ab531ce4f4a00e29993bb9b6f728ff243eb5ea1dda276d43a56d3881370af8 WatchSource:0}: Error finding container b9ab531ce4f4a00e29993bb9b6f728ff243eb5ea1dda276d43a56d3881370af8: Status 404 returned error can't find the container with id b9ab531ce4f4a00e29993bb9b6f728ff243eb5ea1dda276d43a56d3881370af8
Sep 16 18:42:46.617932 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:46.617887    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mggrx" event={"ID":"f01001cd-d72f-4267-8e14-44359c95f66c","Type":"ContainerStarted","Data":"b9ab531ce4f4a00e29993bb9b6f728ff243eb5ea1dda276d43a56d3881370af8"}
Sep 16 18:42:46.620089 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:46.620054    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-986b9b8c8-9p8c8" event={"ID":"04e761f8-c6e9-4238-bd6b-0ee2db961991","Type":"ContainerStarted","Data":"c2076fd545482ba66d441e3eff612c181e84683d1d069418a8bf9d25193e9b8c"}
Sep 16 18:42:46.620252 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:46.620104    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-986b9b8c8-9p8c8" event={"ID":"04e761f8-c6e9-4238-bd6b-0ee2db961991","Type":"ContainerStarted","Data":"24f3d781a834c3097f106067f1aeb758c42eb2add4f11a41ad5c253c7654c05a"}
Sep 16 18:42:46.645409 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:46.645241    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-986b9b8c8-9p8c8" podStartSLOduration=33.645207201 podStartE2EDuration="33.645207201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:46.644514394 +0000 UTC m=+142.088375972" watchObservedRunningTime="2026-09-16 18:42:46.645207201 +0000 UTC m=+142.089068778"
Sep 16 18:42:47.513919 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:47.513879    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-986b9b8c8-9p8c8"
Sep 16 18:42:47.517036 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:47.517000    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-986b9b8c8-9p8c8"
Sep 16 18:42:47.623750 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:47.623679    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-986b9b8c8-9p8c8"
Sep 16 18:42:47.625078 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:47.625053    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-986b9b8c8-9p8c8"
Sep 16 18:42:48.632410 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:48.632374    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mggrx" event={"ID":"f01001cd-d72f-4267-8e14-44359c95f66c","Type":"ContainerStarted","Data":"4bf43d2020910a6d3ff542b1d4e8f01e08ce75fa8d2623deaeb43f191f89a5b7"}
Sep 16 18:42:48.652829 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:48.652766    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-mggrx" podStartSLOduration=34.019732106 podStartE2EDuration="35.652745217s" totalImagesPullingTime="1.633013111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:48.652113451 +0000 UTC m=+144.095975028" watchObservedRunningTime="2026-09-16 18:42:48.652745217 +0000 UTC m=+144.096606793"
Sep 16 18:42:55.870627 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:55.870590    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-5ppvl"]
Sep 16 18:42:55.874122 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:55.874099    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-5ppvl"
Sep 16 18:42:55.876481 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:55.876450    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-mf27k\""
Sep 16 18:42:55.876821 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:55.876636    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 16 18:42:55.877684 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:55.877664    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 16 18:42:55.877804 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:55.877696    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 18:42:55.891953 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:55.891924    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-5ppvl"]
Sep 16 18:42:55.896992 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:55.896968    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rdxf6"]
Sep 16 18:42:55.899673 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:55.899650    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rdxf6"
Sep 16 18:42:55.903909 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:55.903604    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 18:42:55.904018 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:55.903976    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6dgwx\""
Sep 16 18:42:55.908784 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:55.904187    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 18:42:55.908784 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:55.904258    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 18:42:55.993508 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:55.993299    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ac2e187-f76e-4497-82dd-c8b902153ace-metrics-client-ca\") pod \"node-exporter-rdxf6\" (UID: \"0ac2e187-f76e-4497-82dd-c8b902153ace\") " pod="openshift-monitoring/node-exporter-rdxf6"
Sep 16 18:42:55.993508 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:55.993409    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whnqt\" (UniqueName: \"kubernetes.io/projected/0ac2e187-f76e-4497-82dd-c8b902153ace-kube-api-access-whnqt\") pod \"node-exporter-rdxf6\" (UID: \"0ac2e187-f76e-4497-82dd-c8b902153ace\") " pod="openshift-monitoring/node-exporter-rdxf6"
Sep 16 18:42:55.993508 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:55.993455    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0ac2e187-f76e-4497-82dd-c8b902153ace-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rdxf6\" (UID: \"0ac2e187-f76e-4497-82dd-c8b902153ace\") " pod="openshift-monitoring/node-exporter-rdxf6"
Sep 16 18:42:55.993803 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:55.993513    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0ac2e187-f76e-4497-82dd-c8b902153ace-node-exporter-wtmp\") pod \"node-exporter-rdxf6\" (UID: \"0ac2e187-f76e-4497-82dd-c8b902153ace\") " pod="openshift-monitoring/node-exporter-rdxf6"
Sep 16 18:42:55.993803 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:55.993599    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4hwm\" (UniqueName: \"kubernetes.io/projected/73f60141-a105-4a43-a0a3-a4e7b8263963-kube-api-access-c4hwm\") pod \"openshift-state-metrics-67dcf8c979-5ppvl\" (UID: \"73f60141-a105-4a43-a0a3-a4e7b8263963\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-5ppvl"
Sep 16 18:42:55.993803 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:55.993671    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0ac2e187-f76e-4497-82dd-c8b902153ace-sys\") pod \"node-exporter-rdxf6\" (UID: \"0ac2e187-f76e-4497-82dd-c8b902153ace\") " pod="openshift-monitoring/node-exporter-rdxf6"
Sep 16 18:42:55.993803 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:55.993702    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/73f60141-a105-4a43-a0a3-a4e7b8263963-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-5ppvl\" (UID: \"73f60141-a105-4a43-a0a3-a4e7b8263963\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-5ppvl"
Sep 16 18:42:55.993803 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:55.993733    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/73f60141-a105-4a43-a0a3-a4e7b8263963-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-5ppvl\" (UID: \"73f60141-a105-4a43-a0a3-a4e7b8263963\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-5ppvl"
Sep 16 18:42:55.993803 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:55.993764    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0ac2e187-f76e-4497-82dd-c8b902153ace-root\") pod \"node-exporter-rdxf6\" (UID: \"0ac2e187-f76e-4497-82dd-c8b902153ace\") " pod="openshift-monitoring/node-exporter-rdxf6"
Sep 16 18:42:55.993803 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:55.993789    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73f60141-a105-4a43-a0a3-a4e7b8263963-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-5ppvl\" (UID: \"73f60141-a105-4a43-a0a3-a4e7b8263963\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-5ppvl"
Sep 16 18:42:55.994209 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:55.993816    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0ac2e187-f76e-4497-82dd-c8b902153ace-node-exporter-tls\") pod \"node-exporter-rdxf6\" (UID: \"0ac2e187-f76e-4497-82dd-c8b902153ace\") " pod="openshift-monitoring/node-exporter-rdxf6"
Sep 16 18:42:55.994209 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:55.993845    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0ac2e187-f76e-4497-82dd-c8b902153ace-node-exporter-textfile\") pod \"node-exporter-rdxf6\" (UID: \"0ac2e187-f76e-4497-82dd-c8b902153ace\") " pod="openshift-monitoring/node-exporter-rdxf6"
Sep 16 18:42:55.994209 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:55.993869    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0ac2e187-f76e-4497-82dd-c8b902153ace-node-exporter-accelerators-collector-config\") pod \"node-exporter-rdxf6\" (UID: \"0ac2e187-f76e-4497-82dd-c8b902153ace\") " pod="openshift-monitoring/node-exporter-rdxf6"
Sep 16 18:42:56.097256 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:56.094699    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ac2e187-f76e-4497-82dd-c8b902153ace-metrics-client-ca\") pod \"node-exporter-rdxf6\" (UID: \"0ac2e187-f76e-4497-82dd-c8b902153ace\") " pod="openshift-monitoring/node-exporter-rdxf6"
Sep 16 18:42:56.097256 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:56.094772    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whnqt\" (UniqueName: \"kubernetes.io/projected/0ac2e187-f76e-4497-82dd-c8b902153ace-kube-api-access-whnqt\") pod \"node-exporter-rdxf6\" (UID: \"0ac2e187-f76e-4497-82dd-c8b902153ace\") " pod="openshift-monitoring/node-exporter-rdxf6"
Sep 16 18:42:56.097256 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:56.094808    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0ac2e187-f76e-4497-82dd-c8b902153ace-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rdxf6\" (UID: \"0ac2e187-f76e-4497-82dd-c8b902153ace\") " pod="openshift-monitoring/node-exporter-rdxf6"
Sep 16 18:42:56.097256 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:56.094844    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0ac2e187-f76e-4497-82dd-c8b902153ace-node-exporter-wtmp\") pod \"node-exporter-rdxf6\" (UID: \"0ac2e187-f76e-4497-82dd-c8b902153ace\") " pod="openshift-monitoring/node-exporter-rdxf6"
Sep 16 18:42:56.097256 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:56.094870    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4hwm\" (UniqueName: \"kubernetes.io/projected/73f60141-a105-4a43-a0a3-a4e7b8263963-kube-api-access-c4hwm\") pod \"openshift-state-metrics-67dcf8c979-5ppvl\" (UID: \"73f60141-a105-4a43-a0a3-a4e7b8263963\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-5ppvl"
Sep 16 18:42:56.097256 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:56.094901    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0ac2e187-f76e-4497-82dd-c8b902153ace-sys\") pod \"node-exporter-rdxf6\" (UID: \"0ac2e187-f76e-4497-82dd-c8b902153ace\") " pod="openshift-monitoring/node-exporter-rdxf6"
Sep 16 18:42:56.097256 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:56.094927    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/73f60141-a105-4a43-a0a3-a4e7b8263963-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-5ppvl\" (UID: \"73f60141-a105-4a43-a0a3-a4e7b8263963\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-5ppvl"
Sep 16 18:42:56.097256 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:56.094956    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/73f60141-a105-4a43-a0a3-a4e7b8263963-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-5ppvl\" (UID: \"73f60141-a105-4a43-a0a3-a4e7b8263963\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-5ppvl"
Sep 16 18:42:56.097256 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:56.094982    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0ac2e187-f76e-4497-82dd-c8b902153ace-root\") pod \"node-exporter-rdxf6\" (UID: \"0ac2e187-f76e-4497-82dd-c8b902153ace\") " pod="openshift-monitoring/node-exporter-rdxf6"
Sep 16 18:42:56.097256 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:56.095009    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73f60141-a105-4a43-a0a3-a4e7b8263963-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-5ppvl\" (UID: \"73f60141-a105-4a43-a0a3-a4e7b8263963\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-5ppvl"
Sep 16 18:42:56.097256 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:56.095037    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0ac2e187-f76e-4497-82dd-c8b902153ace-node-exporter-tls\") pod \"node-exporter-rdxf6\" (UID: \"0ac2e187-f76e-4497-82dd-c8b902153ace\") " pod="openshift-monitoring/node-exporter-rdxf6"
Sep 16 18:42:56.097256 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:56.095064    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0ac2e187-f76e-4497-82dd-c8b902153ace-node-exporter-textfile\") pod \"node-exporter-rdxf6\" (UID: \"0ac2e187-f76e-4497-82dd-c8b902153ace\") " pod="openshift-monitoring/node-exporter-rdxf6"
Sep 16 18:42:56.097256 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:56.095089    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0ac2e187-f76e-4497-82dd-c8b902153ace-node-exporter-accelerators-collector-config\") pod \"node-exporter-rdxf6\" (UID: \"0ac2e187-f76e-4497-82dd-c8b902153ace\") " pod="openshift-monitoring/node-exporter-rdxf6"
Sep 16 18:42:56.097256 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:56.095832    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0ac2e187-f76e-4497-82dd-c8b902153ace-node-exporter-accelerators-collector-config\") pod \"node-exporter-rdxf6\" (UID: \"0ac2e187-f76e-4497-82dd-c8b902153ace\") " pod="openshift-monitoring/node-exporter-rdxf6"
Sep 16 18:42:56.097256 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:56.096291    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ac2e187-f76e-4497-82dd-c8b902153ace-metrics-client-ca\") pod \"node-exporter-rdxf6\" (UID: \"0ac2e187-f76e-4497-82dd-c8b902153ace\") " pod="openshift-monitoring/node-exporter-rdxf6"
Sep 16 18:42:56.097256 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:56.097036    2586 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 16 18:42:56.098167 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:56.097102    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73f60141-a105-4a43-a0a3-a4e7b8263963-openshift-state-metrics-tls podName:73f60141-a105-4a43-a0a3-a4e7b8263963 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:56.597084169 +0000 UTC m=+152.040945741 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/73f60141-a105-4a43-a0a3-a4e7b8263963-openshift-state-metrics-tls") pod "openshift-state-metrics-67dcf8c979-5ppvl" (UID: "73f60141-a105-4a43-a0a3-a4e7b8263963") : secret "openshift-state-metrics-tls" not found
Sep 16 18:42:56.099643 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:56.098317    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0ac2e187-f76e-4497-82dd-c8b902153ace-sys\") pod \"node-exporter-rdxf6\" (UID: \"0ac2e187-f76e-4497-82dd-c8b902153ace\") " pod="openshift-monitoring/node-exporter-rdxf6"
Sep 16 18:42:56.099643 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:56.098413    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0ac2e187-f76e-4497-82dd-c8b902153ace-node-exporter-wtmp\") pod \"node-exporter-rdxf6\" (UID: \"0ac2e187-f76e-4497-82dd-c8b902153ace\") " pod="openshift-monitoring/node-exporter-rdxf6"
Sep 16 18:42:56.099643 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:56.099086    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73f60141-a105-4a43-a0a3-a4e7b8263963-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-5ppvl\" (UID: \"73f60141-a105-4a43-a0a3-a4e7b8263963\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-5ppvl"
Sep 16 18:42:56.099643 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:56.099151    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0ac2e187-f76e-4497-82dd-c8b902153ace-root\") pod \"node-exporter-rdxf6\" (UID: \"0ac2e187-f76e-4497-82dd-c8b902153ace\") " pod="openshift-monitoring/node-exporter-rdxf6"
Sep 16 18:42:56.099643 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:56.099245    2586 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 16 18:42:56.099643 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:42:56.099292    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ac2e187-f76e-4497-82dd-c8b902153ace-node-exporter-tls podName:0ac2e187-f76e-4497-82dd-c8b902153ace nodeName:}" failed. No retries permitted until 2026-09-16 18:42:56.599276216 +0000 UTC m=+152.043137784 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/0ac2e187-f76e-4497-82dd-c8b902153ace-node-exporter-tls") pod "node-exporter-rdxf6" (UID: "0ac2e187-f76e-4497-82dd-c8b902153ace") : secret "node-exporter-tls" not found
Sep 16 18:42:56.099643 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:56.099518    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0ac2e187-f76e-4497-82dd-c8b902153ace-node-exporter-textfile\") pod \"node-exporter-rdxf6\" (UID: \"0ac2e187-f76e-4497-82dd-c8b902153ace\") " pod="openshift-monitoring/node-exporter-rdxf6"
Sep 16 18:42:56.100632 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:56.100609    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0ac2e187-f76e-4497-82dd-c8b902153ace-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rdxf6\" (UID: \"0ac2e187-f76e-4497-82dd-c8b902153ace\") " pod="openshift-monitoring/node-exporter-rdxf6"
Sep 16 18:42:56.102834 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:56.102807    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/73f60141-a105-4a43-a0a3-a4e7b8263963-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-5ppvl\" (UID: \"73f60141-a105-4a43-a0a3-a4e7b8263963\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-5ppvl"
Sep 16 18:42:56.108914 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:56.108891    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whnqt\" (UniqueName: \"kubernetes.io/projected/0ac2e187-f76e-4497-82dd-c8b902153ace-kube-api-access-whnqt\") pod \"node-exporter-rdxf6\" (UID: \"0ac2e187-f76e-4497-82dd-c8b902153ace\") " pod="openshift-monitoring/node-exporter-rdxf6"
Sep 16 18:42:56.112812 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:56.112768    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4hwm\" (UniqueName: \"kubernetes.io/projected/73f60141-a105-4a43-a0a3-a4e7b8263963-kube-api-access-c4hwm\") pod \"openshift-state-metrics-67dcf8c979-5ppvl\" (UID: \"73f60141-a105-4a43-a0a3-a4e7b8263963\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-5ppvl"
Sep 16 18:42:56.599888 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:56.599855    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0ac2e187-f76e-4497-82dd-c8b902153ace-node-exporter-tls\") pod \"node-exporter-rdxf6\" (UID: \"0ac2e187-f76e-4497-82dd-c8b902153ace\") " pod="openshift-monitoring/node-exporter-rdxf6"
Sep 16 18:42:56.600079 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:56.599983    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/73f60141-a105-4a43-a0a3-a4e7b8263963-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-5ppvl\" (UID: \"73f60141-a105-4a43-a0a3-a4e7b8263963\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-5ppvl"
Sep 16 18:42:56.602584 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:56.602544    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0ac2e187-f76e-4497-82dd-c8b902153ace-node-exporter-tls\") pod \"node-exporter-rdxf6\" (UID: \"0ac2e187-f76e-4497-82dd-c8b902153ace\") " pod="openshift-monitoring/node-exporter-rdxf6"
Sep 16 18:42:56.602813 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:56.602783    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/73f60141-a105-4a43-a0a3-a4e7b8263963-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-5ppvl\" (UID: \"73f60141-a105-4a43-a0a3-a4e7b8263963\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-5ppvl"
Sep 16 18:42:56.785741 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:56.785709    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-5ppvl"
Sep 16 18:42:56.819208 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:56.819179    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rdxf6"
Sep 16 18:42:58.307404 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:42:58.307197    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ac2e187_f76e_4497_82dd_c8b902153ace.slice/crio-ba86fbbde73ae02bc2b63f157ea1d6b280fe9e958c92724ac6b028b9237fc642 WatchSource:0}: Error finding container ba86fbbde73ae02bc2b63f157ea1d6b280fe9e958c92724ac6b028b9237fc642: Status 404 returned error can't find the container with id ba86fbbde73ae02bc2b63f157ea1d6b280fe9e958c92724ac6b028b9237fc642
Sep 16 18:42:58.596177 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:58.596101    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-5ppvl"]
Sep 16 18:42:58.600018 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:42:58.599986    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73f60141_a105_4a43_a0a3_a4e7b8263963.slice/crio-177fa8a4c7036e7b4fddfb9d712da854ef9af53ec3ff457f5fde2851bdeee461 WatchSource:0}: Error finding container 177fa8a4c7036e7b4fddfb9d712da854ef9af53ec3ff457f5fde2851bdeee461: Status 404 returned error can't find the container with id 177fa8a4c7036e7b4fddfb9d712da854ef9af53ec3ff457f5fde2851bdeee461
Sep 16 18:42:58.673822 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:58.673578    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-wd4p9" event={"ID":"0fb1d7e8-fe62-4665-b5a8-8bcbd0eff71e","Type":"ContainerStarted","Data":"8466b44cd7ba147b016e4ffe4e9a5816bc598fd9a34269e73dc11c9c2a762b90"}
Sep 16 18:42:58.674628 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:58.674076    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-wd4p9"
Sep 16 18:42:58.675883 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:58.675857    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-5ppvl" event={"ID":"73f60141-a105-4a43-a0a3-a4e7b8263963","Type":"ContainerStarted","Data":"177fa8a4c7036e7b4fddfb9d712da854ef9af53ec3ff457f5fde2851bdeee461"}
Sep 16 18:42:58.677473 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:58.677442    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rdxf6" event={"ID":"0ac2e187-f76e-4497-82dd-c8b902153ace","Type":"ContainerStarted","Data":"ba86fbbde73ae02bc2b63f157ea1d6b280fe9e958c92724ac6b028b9237fc642"}
Sep 16 18:42:58.684051 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:58.684029    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-wd4p9"
Sep 16 18:42:58.718419 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:58.718362    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-wd4p9" podStartSLOduration=1.060833528 podStartE2EDuration="17.718343442s" totalImagesPullingTime="16.657509914s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:58.717171174 +0000 UTC m=+154.161032749" watchObservedRunningTime="2026-09-16 18:42:58.718343442 +0000 UTC m=+154.162205018"
Sep 16 18:42:59.686005 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:59.685897    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-5ppvl" event={"ID":"73f60141-a105-4a43-a0a3-a4e7b8263963","Type":"ContainerStarted","Data":"f9ca30b5ab1384dced70ec1523931ae8090c7e80e3d92d8e0b7e37d0e8b22480"}
Sep 16 18:42:59.686005 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:42:59.685959    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-5ppvl" event={"ID":"73f60141-a105-4a43-a0a3-a4e7b8263963","Type":"ContainerStarted","Data":"118a5ad356fc472937abb3fd0bad1399b00dd155ccfae98a813193ea6c740b30"}
Sep 16 18:43:00.691960 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:00.691759    2586 generic.go:356] "Generic (PLEG): container finished" podID="0ac2e187-f76e-4497-82dd-c8b902153ace" containerID="ca16b92c92a866096ffc1545bbdf01002a0a5e86c623e01d067bdd65e7151a90" exitCode=0
Sep 16 18:43:00.691960 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:00.691850    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rdxf6" event={"ID":"0ac2e187-f76e-4497-82dd-c8b902153ace","Type":"ContainerDied","Data":"ca16b92c92a866096ffc1545bbdf01002a0a5e86c623e01d067bdd65e7151a90"}
Sep 16 18:43:00.695245 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:00.695195    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-5ppvl" event={"ID":"73f60141-a105-4a43-a0a3-a4e7b8263963","Type":"ContainerStarted","Data":"0ac002f9c3633145f7c0225ed712ca1058cfe94b4403087adfdd432993aff431"}
Sep 16 18:43:00.758156 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:00.758107    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-5ppvl" podStartSLOduration=4.287064221 podStartE2EDuration="5.758090548s" totalImagesPullingTime="1.471026327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:00.756612825 +0000 UTC m=+156.200474402" watchObservedRunningTime="2026-09-16 18:43:00.758090548 +0000 UTC m=+156.201952124"
Sep 16 18:43:01.454794 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:43:01.454745    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-6xjlp" podUID="141828f8-d629-4cbd-9ed2-875d52d4db56"
Sep 16 18:43:01.702782 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:01.702754    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6xjlp"
Sep 16 18:43:01.702782 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:01.702769    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rdxf6" event={"ID":"0ac2e187-f76e-4497-82dd-c8b902153ace","Type":"ContainerStarted","Data":"14e9798563ae2436a295052476224cb0f7debf3baa0ff86353b6d19eef3ebc74"}
Sep 16 18:43:01.703324 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:01.702823    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rdxf6" event={"ID":"0ac2e187-f76e-4497-82dd-c8b902153ace","Type":"ContainerStarted","Data":"798b7e751eddb97e284fe6ec3dbc3ca879eafa6509344ccdd920b32fd3876b1a"}
Sep 16 18:43:01.736331 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:01.736217    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rdxf6" podStartSLOduration=4.761781729 podStartE2EDuration="6.736200525s" totalImagesPullingTime="1.884966834s" totalInitContainerRuntime="89.451962ms" isStatefulPod=true podCreationTimestamp="2026-09-16 18:42:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:01.735002215 +0000 UTC m=+157.178863790" watchObservedRunningTime="2026-09-16 18:43:01.736200525 +0000 UTC m=+157.180062105"
Sep 16 18:43:02.482950 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:02.482912    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7d4ddd4649-jmfhq"]
Sep 16 18:43:02.516281 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:02.516245    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d4ddd4649-jmfhq"]
Sep 16 18:43:02.516446 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:02.516346    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d4ddd4649-jmfhq"
Sep 16 18:43:02.519268 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:02.519218    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 16 18:43:02.520578 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:02.519949    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 16 18:43:02.520578 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:02.520207    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 16 18:43:02.520578 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:02.520293    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 16 18:43:02.520578 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:02.520293    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 16 18:43:02.520578 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:02.520211    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-ph8qj\""
Sep 16 18:43:02.528335 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:02.527811    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 16 18:43:02.666007 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:02.665973    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/605e74c3-34fd-41c6-b1d5-cd6656d45243-service-ca\") pod \"console-7d4ddd4649-jmfhq\" (UID: \"605e74c3-34fd-41c6-b1d5-cd6656d45243\") " pod="openshift-console/console-7d4ddd4649-jmfhq"
Sep 16 18:43:02.666007 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:02.666015    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/605e74c3-34fd-41c6-b1d5-cd6656d45243-console-oauth-config\") pod \"console-7d4ddd4649-jmfhq\" (UID: \"605e74c3-34fd-41c6-b1d5-cd6656d45243\") " pod="openshift-console/console-7d4ddd4649-jmfhq"
Sep 16 18:43:02.666303 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:02.666057    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/605e74c3-34fd-41c6-b1d5-cd6656d45243-console-config\") pod \"console-7d4ddd4649-jmfhq\" (UID: \"605e74c3-34fd-41c6-b1d5-cd6656d45243\") " pod="openshift-console/console-7d4ddd4649-jmfhq"
Sep 16 18:43:02.666303 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:02.666118    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/605e74c3-34fd-41c6-b1d5-cd6656d45243-oauth-serving-cert\") pod \"console-7d4ddd4649-jmfhq\" (UID: \"605e74c3-34fd-41c6-b1d5-cd6656d45243\") " pod="openshift-console/console-7d4ddd4649-jmfhq"
Sep 16 18:43:02.666303 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:02.666251    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/605e74c3-34fd-41c6-b1d5-cd6656d45243-console-serving-cert\") pod \"console-7d4ddd4649-jmfhq\" (UID: \"605e74c3-34fd-41c6-b1d5-cd6656d45243\") " pod="openshift-console/console-7d4ddd4649-jmfhq"
Sep 16 18:43:02.666303 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:02.666289    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/605e74c3-34fd-41c6-b1d5-cd6656d45243-trusted-ca-bundle\") pod \"console-7d4ddd4649-jmfhq\" (UID: \"605e74c3-34fd-41c6-b1d5-cd6656d45243\") " pod="openshift-console/console-7d4ddd4649-jmfhq"
Sep 16 18:43:02.666486 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:02.666324    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d225t\" (UniqueName: \"kubernetes.io/projected/605e74c3-34fd-41c6-b1d5-cd6656d45243-kube-api-access-d225t\") pod \"console-7d4ddd4649-jmfhq\" (UID: \"605e74c3-34fd-41c6-b1d5-cd6656d45243\") " pod="openshift-console/console-7d4ddd4649-jmfhq"
Sep 16 18:43:02.766940 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:02.766904    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/605e74c3-34fd-41c6-b1d5-cd6656d45243-console-serving-cert\") pod \"console-7d4ddd4649-jmfhq\" (UID: \"605e74c3-34fd-41c6-b1d5-cd6656d45243\") " pod="openshift-console/console-7d4ddd4649-jmfhq"
Sep 16 18:43:02.767409 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:02.766958    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/605e74c3-34fd-41c6-b1d5-cd6656d45243-trusted-ca-bundle\") pod \"console-7d4ddd4649-jmfhq\" (UID: \"605e74c3-34fd-41c6-b1d5-cd6656d45243\") " pod="openshift-console/console-7d4ddd4649-jmfhq"
Sep 16 18:43:02.767409 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:02.766986    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d225t\" (UniqueName: \"kubernetes.io/projected/605e74c3-34fd-41c6-b1d5-cd6656d45243-kube-api-access-d225t\") pod \"console-7d4ddd4649-jmfhq\" (UID: \"605e74c3-34fd-41c6-b1d5-cd6656d45243\") " pod="openshift-console/console-7d4ddd4649-jmfhq"
Sep 16 18:43:02.767409 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:02.767018    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/605e74c3-34fd-41c6-b1d5-cd6656d45243-service-ca\") pod \"console-7d4ddd4649-jmfhq\" (UID: \"605e74c3-34fd-41c6-b1d5-cd6656d45243\") " pod="openshift-console/console-7d4ddd4649-jmfhq"
Sep 16 18:43:02.767409 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:02.767149    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/605e74c3-34fd-41c6-b1d5-cd6656d45243-console-oauth-config\") pod \"console-7d4ddd4649-jmfhq\" (UID: \"605e74c3-34fd-41c6-b1d5-cd6656d45243\") " pod="openshift-console/console-7d4ddd4649-jmfhq"
Sep 16 18:43:02.767409 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:02.767218    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/605e74c3-34fd-41c6-b1d5-cd6656d45243-console-config\") pod \"console-7d4ddd4649-jmfhq\" (UID: \"605e74c3-34fd-41c6-b1d5-cd6656d45243\") " pod="openshift-console/console-7d4ddd4649-jmfhq"
Sep 16 18:43:02.767409 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:02.767259    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/605e74c3-34fd-41c6-b1d5-cd6656d45243-oauth-serving-cert\") pod \"console-7d4ddd4649-jmfhq\" (UID: \"605e74c3-34fd-41c6-b1d5-cd6656d45243\") " pod="openshift-console/console-7d4ddd4649-jmfhq"
Sep 16 18:43:02.768094 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:02.768064    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/605e74c3-34fd-41c6-b1d5-cd6656d45243-trusted-ca-bundle\") pod \"console-7d4ddd4649-jmfhq\" (UID: \"605e74c3-34fd-41c6-b1d5-cd6656d45243\") " pod="openshift-console/console-7d4ddd4649-jmfhq"
Sep 16 18:43:02.768547 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:02.768521    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/605e74c3-34fd-41c6-b1d5-cd6656d45243-service-ca\") pod \"console-7d4ddd4649-jmfhq\" (UID: \"605e74c3-34fd-41c6-b1d5-cd6656d45243\") " pod="openshift-console/console-7d4ddd4649-jmfhq"
Sep 16 18:43:02.769816 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:02.769794    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/605e74c3-34fd-41c6-b1d5-cd6656d45243-console-serving-cert\") pod \"console-7d4ddd4649-jmfhq\" (UID: \"605e74c3-34fd-41c6-b1d5-cd6656d45243\") " pod="openshift-console/console-7d4ddd4649-jmfhq"
Sep 16 18:43:02.770146 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:02.770128    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/605e74c3-34fd-41c6-b1d5-cd6656d45243-console-oauth-config\") pod \"console-7d4ddd4649-jmfhq\" (UID: \"605e74c3-34fd-41c6-b1d5-cd6656d45243\") " pod="openshift-console/console-7d4ddd4649-jmfhq"
Sep 16 18:43:02.776944 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:02.776895    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d225t\" (UniqueName: \"kubernetes.io/projected/605e74c3-34fd-41c6-b1d5-cd6656d45243-kube-api-access-d225t\") pod \"console-7d4ddd4649-jmfhq\" (UID: \"605e74c3-34fd-41c6-b1d5-cd6656d45243\") " pod="openshift-console/console-7d4ddd4649-jmfhq"
Sep 16 18:43:02.778132 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:02.778109    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/605e74c3-34fd-41c6-b1d5-cd6656d45243-console-config\") pod \"console-7d4ddd4649-jmfhq\" (UID: \"605e74c3-34fd-41c6-b1d5-cd6656d45243\") " pod="openshift-console/console-7d4ddd4649-jmfhq"
Sep 16 18:43:02.778285 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:02.778222    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/605e74c3-34fd-41c6-b1d5-cd6656d45243-oauth-serving-cert\") pod \"console-7d4ddd4649-jmfhq\" (UID: \"605e74c3-34fd-41c6-b1d5-cd6656d45243\") " pod="openshift-console/console-7d4ddd4649-jmfhq"
Sep 16 18:43:02.831519 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:02.831476    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d4ddd4649-jmfhq"
Sep 16 18:43:03.156279 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:03.156240    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d4ddd4649-jmfhq"]
Sep 16 18:43:03.159547 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:43:03.159518    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod605e74c3_34fd_41c6_b1d5_cd6656d45243.slice/crio-f75e250fdf436ec73d22f9543caef3482b02466ab7c85c4a461dfe891836bc7f WatchSource:0}: Error finding container f75e250fdf436ec73d22f9543caef3482b02466ab7c85c4a461dfe891836bc7f: Status 404 returned error can't find the container with id f75e250fdf436ec73d22f9543caef3482b02466ab7c85c4a461dfe891836bc7f
Sep 16 18:43:03.608179 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:03.608149    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-64ccc449b-b924q"
Sep 16 18:43:03.712953 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:03.712907    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d4ddd4649-jmfhq" event={"ID":"605e74c3-34fd-41c6-b1d5-cd6656d45243","Type":"ContainerStarted","Data":"f75e250fdf436ec73d22f9543caef3482b02466ab7c85c4a461dfe891836bc7f"}
Sep 16 18:43:06.302484 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:06.302443    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/141828f8-d629-4cbd-9ed2-875d52d4db56-metrics-tls\") pod \"dns-default-6xjlp\" (UID: \"141828f8-d629-4cbd-9ed2-875d52d4db56\") " pod="openshift-dns/dns-default-6xjlp"
Sep 16 18:43:06.305574 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:06.305515    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/141828f8-d629-4cbd-9ed2-875d52d4db56-metrics-tls\") pod \"dns-default-6xjlp\" (UID: \"141828f8-d629-4cbd-9ed2-875d52d4db56\") " pod="openshift-dns/dns-default-6xjlp"
Sep 16 18:43:06.507144 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:06.507107    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xfn4n\""
Sep 16 18:43:06.514407 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:06.514378    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6xjlp"
Sep 16 18:43:06.827207 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:06.827173    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6xjlp"]
Sep 16 18:43:06.929819 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:43:06.929767    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod141828f8_d629_4cbd_9ed2_875d52d4db56.slice/crio-63d9bd7db39b184594665c7176bf33eece777f18bcb86ee53f4ef976ec343ba6 WatchSource:0}: Error finding container 63d9bd7db39b184594665c7176bf33eece777f18bcb86ee53f4ef976ec343ba6: Status 404 returned error can't find the container with id 63d9bd7db39b184594665c7176bf33eece777f18bcb86ee53f4ef976ec343ba6
Sep 16 18:43:07.740485 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:07.740398    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d4ddd4649-jmfhq" event={"ID":"605e74c3-34fd-41c6-b1d5-cd6656d45243","Type":"ContainerStarted","Data":"11f2a3438a3692242d03c14c792f7d3ec10da509d851bf587a3ae7aea43ea60c"}
Sep 16 18:43:07.741919 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:07.741892    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6xjlp" event={"ID":"141828f8-d629-4cbd-9ed2-875d52d4db56","Type":"ContainerStarted","Data":"63d9bd7db39b184594665c7176bf33eece777f18bcb86ee53f4ef976ec343ba6"}
Sep 16 18:43:07.780446 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:07.780388    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7d4ddd4649-jmfhq" podStartSLOduration=1.9774294590000001 podStartE2EDuration="5.78036711s" totalImagesPullingTime="3.802937651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:43:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:07.779099605 +0000 UTC m=+163.222961182" watchObservedRunningTime="2026-09-16 18:43:07.78036711 +0000 UTC m=+163.224228687"
Sep 16 18:43:09.750873 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:09.750837    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6xjlp" event={"ID":"141828f8-d629-4cbd-9ed2-875d52d4db56","Type":"ContainerStarted","Data":"7a23f110a76ce0e55ac2ef9c158edce09116b898153ab7f8e836c999975777a2"}
Sep 16 18:43:09.751331 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:09.750889    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6xjlp" event={"ID":"141828f8-d629-4cbd-9ed2-875d52d4db56","Type":"ContainerStarted","Data":"a99c1352c18a40adf602d47c871497b71e98819eb123810efaaba7bca57f8df8"}
Sep 16 18:43:09.751331 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:09.750954    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6xjlp"
Sep 16 18:43:09.773886 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:09.773841    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-6xjlp" podStartSLOduration=129.77711905500001 podStartE2EDuration="2m11.773820502s" totalImagesPullingTime="1.996701447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:09.773115746 +0000 UTC m=+165.216977322" watchObservedRunningTime="2026-09-16 18:43:09.773820502 +0000 UTC m=+165.217682080"
Sep 16 18:43:12.832066 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:12.832026    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7d4ddd4649-jmfhq"
Sep 16 18:43:12.832542 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:12.832090    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7d4ddd4649-jmfhq"
Sep 16 18:43:12.837057 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:12.837032    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7d4ddd4649-jmfhq"
Sep 16 18:43:13.768717 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:13.768684    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7d4ddd4649-jmfhq"
Sep 16 18:43:19.757323 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:19.757252    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6xjlp"
Sep 16 18:43:22.801667 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:22.801576    2586 generic.go:356] "Generic (PLEG): container finished" podID="3a6129d0-3d10-4f44-881a-632760e357fd" containerID="f2b0ddbdd93432bc1d1fb750ababbcebe53e20289771e58e5983a4d0ba945531" exitCode=0
Sep 16 18:43:22.801667 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:22.801621    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-sh5xv" event={"ID":"3a6129d0-3d10-4f44-881a-632760e357fd","Type":"ContainerDied","Data":"f2b0ddbdd93432bc1d1fb750ababbcebe53e20289771e58e5983a4d0ba945531"}
Sep 16 18:43:22.802131 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:22.801930    2586 scope.go:117] "RemoveContainer" containerID="f2b0ddbdd93432bc1d1fb750ababbcebe53e20289771e58e5983a4d0ba945531"
Sep 16 18:43:23.807097 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:23.807059    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-sh5xv" event={"ID":"3a6129d0-3d10-4f44-881a-632760e357fd","Type":"ContainerStarted","Data":"8b3646d6d242eb9f7aac6e454d3c9fe449d7e63e60401de9b6e9c4e20f783e71"}
Sep 16 18:43:32.840757 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:32.840722    2586 generic.go:356] "Generic (PLEG): container finished" podID="f801294c-a332-41af-bede-44bcd7a22e2c" containerID="54d1ff7a498c35ede8927a6eb7f4d08ff2723d2e58a3fb9944902a2793d2b334" exitCode=0
Sep 16 18:43:32.841218 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:32.840799    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-kb6s9" event={"ID":"f801294c-a332-41af-bede-44bcd7a22e2c","Type":"ContainerDied","Data":"54d1ff7a498c35ede8927a6eb7f4d08ff2723d2e58a3fb9944902a2793d2b334"}
Sep 16 18:43:32.841218 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:32.841163    2586 scope.go:117] "RemoveContainer" containerID="54d1ff7a498c35ede8927a6eb7f4d08ff2723d2e58a3fb9944902a2793d2b334"
Sep 16 18:43:32.842382 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:32.842358    2586 generic.go:356] "Generic (PLEG): container finished" podID="3ab787a9-4d8d-42f1-b7cb-432be6e37285" containerID="686f76a96709dfc13833c83ae1a5c2451a061377af95265780d0f5c50b11af92" exitCode=0
Sep 16 18:43:32.842470 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:32.842431    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-ps6sd" event={"ID":"3ab787a9-4d8d-42f1-b7cb-432be6e37285","Type":"ContainerDied","Data":"686f76a96709dfc13833c83ae1a5c2451a061377af95265780d0f5c50b11af92"}
Sep 16 18:43:32.842699 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:32.842686    2586 scope.go:117] "RemoveContainer" containerID="686f76a96709dfc13833c83ae1a5c2451a061377af95265780d0f5c50b11af92"
Sep 16 18:43:33.848425 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:33.848390    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-kb6s9" event={"ID":"f801294c-a332-41af-bede-44bcd7a22e2c","Type":"ContainerStarted","Data":"436bd7428f467d4f135ff3d8c6363e977d8a0468cce08b2f6ea80db8fe6588e8"}
Sep 16 18:43:33.850207 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:43:33.850181    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-ps6sd" event={"ID":"3ab787a9-4d8d-42f1-b7cb-432be6e37285","Type":"ContainerStarted","Data":"2daa1134a413c1a66f6171113f454290548952f53f5c698bc29453917f69d9ed"}
Sep 16 18:44:23.539480 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:44:23.539450    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-59bb745b6f-gh5tc"]
Sep 16 18:44:23.541872 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:44:23.541852    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59bb745b6f-gh5tc"
Sep 16 18:44:23.555814 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:44:23.555787    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59bb745b6f-gh5tc"]
Sep 16 18:44:23.697200 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:44:23.697161    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtmlw\" (UniqueName: \"kubernetes.io/projected/b18aa2f3-1cc3-4e45-97b7-122a66d979d2-kube-api-access-dtmlw\") pod \"console-59bb745b6f-gh5tc\" (UID: \"b18aa2f3-1cc3-4e45-97b7-122a66d979d2\") " pod="openshift-console/console-59bb745b6f-gh5tc"
Sep 16 18:44:23.697200 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:44:23.697204    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b18aa2f3-1cc3-4e45-97b7-122a66d979d2-oauth-serving-cert\") pod \"console-59bb745b6f-gh5tc\" (UID: \"b18aa2f3-1cc3-4e45-97b7-122a66d979d2\") " pod="openshift-console/console-59bb745b6f-gh5tc"
Sep 16 18:44:23.697442 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:44:23.697268    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b18aa2f3-1cc3-4e45-97b7-122a66d979d2-console-oauth-config\") pod \"console-59bb745b6f-gh5tc\" (UID: \"b18aa2f3-1cc3-4e45-97b7-122a66d979d2\") " pod="openshift-console/console-59bb745b6f-gh5tc"
Sep 16 18:44:23.697442 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:44:23.697295    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b18aa2f3-1cc3-4e45-97b7-122a66d979d2-console-config\") pod \"console-59bb745b6f-gh5tc\" (UID: \"b18aa2f3-1cc3-4e45-97b7-122a66d979d2\") " pod="openshift-console/console-59bb745b6f-gh5tc"
Sep 16 18:44:23.697442 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:44:23.697320    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b18aa2f3-1cc3-4e45-97b7-122a66d979d2-trusted-ca-bundle\") pod \"console-59bb745b6f-gh5tc\" (UID: \"b18aa2f3-1cc3-4e45-97b7-122a66d979d2\") " pod="openshift-console/console-59bb745b6f-gh5tc"
Sep 16 18:44:23.697442 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:44:23.697373    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b18aa2f3-1cc3-4e45-97b7-122a66d979d2-console-serving-cert\") pod \"console-59bb745b6f-gh5tc\" (UID: \"b18aa2f3-1cc3-4e45-97b7-122a66d979d2\") " pod="openshift-console/console-59bb745b6f-gh5tc"
Sep 16 18:44:23.697442 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:44:23.697405    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b18aa2f3-1cc3-4e45-97b7-122a66d979d2-service-ca\") pod \"console-59bb745b6f-gh5tc\" (UID: \"b18aa2f3-1cc3-4e45-97b7-122a66d979d2\") " pod="openshift-console/console-59bb745b6f-gh5tc"
Sep 16 18:44:23.798203 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:44:23.798127    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b18aa2f3-1cc3-4e45-97b7-122a66d979d2-console-oauth-config\") pod \"console-59bb745b6f-gh5tc\" (UID: \"b18aa2f3-1cc3-4e45-97b7-122a66d979d2\") " pod="openshift-console/console-59bb745b6f-gh5tc"
Sep 16 18:44:23.798203 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:44:23.798159    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b18aa2f3-1cc3-4e45-97b7-122a66d979d2-console-config\") pod \"console-59bb745b6f-gh5tc\" (UID: \"b18aa2f3-1cc3-4e45-97b7-122a66d979d2\") " pod="openshift-console/console-59bb745b6f-gh5tc"
Sep 16 18:44:23.798203 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:44:23.798178    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b18aa2f3-1cc3-4e45-97b7-122a66d979d2-trusted-ca-bundle\") pod \"console-59bb745b6f-gh5tc\" (UID: \"b18aa2f3-1cc3-4e45-97b7-122a66d979d2\") " pod="openshift-console/console-59bb745b6f-gh5tc"
Sep 16 18:44:23.798487 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:44:23.798214    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b18aa2f3-1cc3-4e45-97b7-122a66d979d2-console-serving-cert\") pod \"console-59bb745b6f-gh5tc\" (UID: \"b18aa2f3-1cc3-4e45-97b7-122a66d979d2\") " pod="openshift-console/console-59bb745b6f-gh5tc"
Sep 16 18:44:23.798487 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:44:23.798283    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b18aa2f3-1cc3-4e45-97b7-122a66d979d2-service-ca\") pod \"console-59bb745b6f-gh5tc\" (UID: \"b18aa2f3-1cc3-4e45-97b7-122a66d979d2\") " pod="openshift-console/console-59bb745b6f-gh5tc"
Sep 16 18:44:23.798487 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:44:23.798358    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtmlw\" (UniqueName: \"kubernetes.io/projected/b18aa2f3-1cc3-4e45-97b7-122a66d979d2-kube-api-access-dtmlw\") pod \"console-59bb745b6f-gh5tc\" (UID: \"b18aa2f3-1cc3-4e45-97b7-122a66d979d2\") " pod="openshift-console/console-59bb745b6f-gh5tc"
Sep 16 18:44:23.798487 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:44:23.798392    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b18aa2f3-1cc3-4e45-97b7-122a66d979d2-oauth-serving-cert\") pod \"console-59bb745b6f-gh5tc\" (UID: \"b18aa2f3-1cc3-4e45-97b7-122a66d979d2\") " pod="openshift-console/console-59bb745b6f-gh5tc"
Sep 16 18:44:23.799040 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:44:23.799009    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b18aa2f3-1cc3-4e45-97b7-122a66d979d2-service-ca\") pod \"console-59bb745b6f-gh5tc\" (UID: \"b18aa2f3-1cc3-4e45-97b7-122a66d979d2\") " pod="openshift-console/console-59bb745b6f-gh5tc"
Sep 16 18:44:23.799177 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:44:23.799016    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b18aa2f3-1cc3-4e45-97b7-122a66d979d2-console-config\") pod \"console-59bb745b6f-gh5tc\" (UID: \"b18aa2f3-1cc3-4e45-97b7-122a66d979d2\") " pod="openshift-console/console-59bb745b6f-gh5tc"
Sep 16 18:44:23.799177 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:44:23.799104    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b18aa2f3-1cc3-4e45-97b7-122a66d979d2-oauth-serving-cert\") pod \"console-59bb745b6f-gh5tc\" (UID: \"b18aa2f3-1cc3-4e45-97b7-122a66d979d2\") " pod="openshift-console/console-59bb745b6f-gh5tc"
Sep 16 18:44:23.799351 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:44:23.799332    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b18aa2f3-1cc3-4e45-97b7-122a66d979d2-trusted-ca-bundle\") pod \"console-59bb745b6f-gh5tc\" (UID: \"b18aa2f3-1cc3-4e45-97b7-122a66d979d2\") " pod="openshift-console/console-59bb745b6f-gh5tc"
Sep 16 18:44:23.800638 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:44:23.800619    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b18aa2f3-1cc3-4e45-97b7-122a66d979d2-console-oauth-config\") pod \"console-59bb745b6f-gh5tc\" (UID: \"b18aa2f3-1cc3-4e45-97b7-122a66d979d2\") " pod="openshift-console/console-59bb745b6f-gh5tc"
Sep 16 18:44:23.800746 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:44:23.800730    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b18aa2f3-1cc3-4e45-97b7-122a66d979d2-console-serving-cert\") pod \"console-59bb745b6f-gh5tc\" (UID: \"b18aa2f3-1cc3-4e45-97b7-122a66d979d2\") " pod="openshift-console/console-59bb745b6f-gh5tc"
Sep 16 18:44:23.807595 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:44:23.807577    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtmlw\" (UniqueName: \"kubernetes.io/projected/b18aa2f3-1cc3-4e45-97b7-122a66d979d2-kube-api-access-dtmlw\") pod \"console-59bb745b6f-gh5tc\" (UID: \"b18aa2f3-1cc3-4e45-97b7-122a66d979d2\") " pod="openshift-console/console-59bb745b6f-gh5tc"
Sep 16 18:44:23.851593 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:44:23.851574    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59bb745b6f-gh5tc"
Sep 16 18:44:24.131739 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:44:24.131717    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59bb745b6f-gh5tc"]
Sep 16 18:44:24.134266 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:44:24.134217    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb18aa2f3_1cc3_4e45_97b7_122a66d979d2.slice/crio-8a65c8d76f2f43a62a85ad1c30b99a3379f0b68a4ad6236bb6314d36f768cd05 WatchSource:0}: Error finding container 8a65c8d76f2f43a62a85ad1c30b99a3379f0b68a4ad6236bb6314d36f768cd05: Status 404 returned error can't find the container with id 8a65c8d76f2f43a62a85ad1c30b99a3379f0b68a4ad6236bb6314d36f768cd05
Sep 16 18:44:25.034721 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:44:25.034685    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59bb745b6f-gh5tc" event={"ID":"b18aa2f3-1cc3-4e45-97b7-122a66d979d2","Type":"ContainerStarted","Data":"53ffd4fbc8f628d7ccb2690f490bb4a89f060f0f50c9312706e4e723f0b715f6"}
Sep 16 18:44:25.035061 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:44:25.034739    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59bb745b6f-gh5tc" event={"ID":"b18aa2f3-1cc3-4e45-97b7-122a66d979d2","Type":"ContainerStarted","Data":"8a65c8d76f2f43a62a85ad1c30b99a3379f0b68a4ad6236bb6314d36f768cd05"}
Sep 16 18:44:25.058856 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:44:25.058806    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-59bb745b6f-gh5tc" podStartSLOduration=2.058787827 podStartE2EDuration="2.058787827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:25.058039875 +0000 UTC m=+240.501901474" watchObservedRunningTime="2026-09-16 18:44:25.058787827 +0000 UTC m=+240.502649429"
Sep 16 18:44:33.852109 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:44:33.852064    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-59bb745b6f-gh5tc"
Sep 16 18:44:33.852629 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:44:33.852142    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-59bb745b6f-gh5tc"
Sep 16 18:44:33.856733 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:44:33.856707    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-59bb745b6f-gh5tc"
Sep 16 18:44:34.070634 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:44:34.070601    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-59bb745b6f-gh5tc"
Sep 16 18:44:34.139073 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:44:34.138985    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d4ddd4649-jmfhq"]
Sep 16 18:45:00.088601 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:00.088496    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7d4ddd4649-jmfhq" podUID="605e74c3-34fd-41c6-b1d5-cd6656d45243" containerName="console" containerID="cri-o://11f2a3438a3692242d03c14c792f7d3ec10da509d851bf587a3ae7aea43ea60c" gracePeriod=15
Sep 16 18:45:00.328155 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:00.328134    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d4ddd4649-jmfhq"
Sep 16 18:45:00.385893 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:00.385864    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/605e74c3-34fd-41c6-b1d5-cd6656d45243-service-ca\") pod \"605e74c3-34fd-41c6-b1d5-cd6656d45243\" (UID: \"605e74c3-34fd-41c6-b1d5-cd6656d45243\") "
Sep 16 18:45:00.386042 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:00.385902    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/605e74c3-34fd-41c6-b1d5-cd6656d45243-console-oauth-config\") pod \"605e74c3-34fd-41c6-b1d5-cd6656d45243\" (UID: \"605e74c3-34fd-41c6-b1d5-cd6656d45243\") "
Sep 16 18:45:00.386042 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:00.385934    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/605e74c3-34fd-41c6-b1d5-cd6656d45243-console-serving-cert\") pod \"605e74c3-34fd-41c6-b1d5-cd6656d45243\" (UID: \"605e74c3-34fd-41c6-b1d5-cd6656d45243\") "
Sep 16 18:45:00.386042 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:00.385982    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d225t\" (UniqueName: \"kubernetes.io/projected/605e74c3-34fd-41c6-b1d5-cd6656d45243-kube-api-access-d225t\") pod \"605e74c3-34fd-41c6-b1d5-cd6656d45243\" (UID: \"605e74c3-34fd-41c6-b1d5-cd6656d45243\") "
Sep 16 18:45:00.386199 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:00.386062    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/605e74c3-34fd-41c6-b1d5-cd6656d45243-console-config\") pod \"605e74c3-34fd-41c6-b1d5-cd6656d45243\" (UID: \"605e74c3-34fd-41c6-b1d5-cd6656d45243\") "
Sep 16 18:45:00.386199 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:00.386099    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/605e74c3-34fd-41c6-b1d5-cd6656d45243-oauth-serving-cert\") pod \"605e74c3-34fd-41c6-b1d5-cd6656d45243\" (UID: \"605e74c3-34fd-41c6-b1d5-cd6656d45243\") "
Sep 16 18:45:00.386199 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:00.386126    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/605e74c3-34fd-41c6-b1d5-cd6656d45243-trusted-ca-bundle\") pod \"605e74c3-34fd-41c6-b1d5-cd6656d45243\" (UID: \"605e74c3-34fd-41c6-b1d5-cd6656d45243\") "
Sep 16 18:45:00.386373 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:00.386270    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/605e74c3-34fd-41c6-b1d5-cd6656d45243-service-ca" (OuterVolumeSpecName: "service-ca") pod "605e74c3-34fd-41c6-b1d5-cd6656d45243" (UID: "605e74c3-34fd-41c6-b1d5-cd6656d45243"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:45:00.386567 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:00.386537    2586 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/605e74c3-34fd-41c6-b1d5-cd6656d45243-service-ca\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 18:45:00.386664 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:00.386569    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/605e74c3-34fd-41c6-b1d5-cd6656d45243-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "605e74c3-34fd-41c6-b1d5-cd6656d45243" (UID: "605e74c3-34fd-41c6-b1d5-cd6656d45243"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:45:00.386664 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:00.386597    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/605e74c3-34fd-41c6-b1d5-cd6656d45243-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "605e74c3-34fd-41c6-b1d5-cd6656d45243" (UID: "605e74c3-34fd-41c6-b1d5-cd6656d45243"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:45:00.386664 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:00.386640    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/605e74c3-34fd-41c6-b1d5-cd6656d45243-console-config" (OuterVolumeSpecName: "console-config") pod "605e74c3-34fd-41c6-b1d5-cd6656d45243" (UID: "605e74c3-34fd-41c6-b1d5-cd6656d45243"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:45:00.388258 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:00.388204    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/605e74c3-34fd-41c6-b1d5-cd6656d45243-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "605e74c3-34fd-41c6-b1d5-cd6656d45243" (UID: "605e74c3-34fd-41c6-b1d5-cd6656d45243"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:45:00.388258 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:00.388205    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/605e74c3-34fd-41c6-b1d5-cd6656d45243-kube-api-access-d225t" (OuterVolumeSpecName: "kube-api-access-d225t") pod "605e74c3-34fd-41c6-b1d5-cd6656d45243" (UID: "605e74c3-34fd-41c6-b1d5-cd6656d45243"). InnerVolumeSpecName "kube-api-access-d225t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:45:00.388411 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:00.388393    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/605e74c3-34fd-41c6-b1d5-cd6656d45243-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "605e74c3-34fd-41c6-b1d5-cd6656d45243" (UID: "605e74c3-34fd-41c6-b1d5-cd6656d45243"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:45:00.487072 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:00.487034    2586 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/605e74c3-34fd-41c6-b1d5-cd6656d45243-console-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 18:45:00.487072 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:00.487067    2586 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/605e74c3-34fd-41c6-b1d5-cd6656d45243-oauth-serving-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 18:45:00.487072 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:00.487079    2586 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/605e74c3-34fd-41c6-b1d5-cd6656d45243-trusted-ca-bundle\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 18:45:00.487335 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:00.487088    2586 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/605e74c3-34fd-41c6-b1d5-cd6656d45243-console-oauth-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 18:45:00.487335 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:00.487096    2586 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/605e74c3-34fd-41c6-b1d5-cd6656d45243-console-serving-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 18:45:00.487335 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:00.487105    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d225t\" (UniqueName: \"kubernetes.io/projected/605e74c3-34fd-41c6-b1d5-cd6656d45243-kube-api-access-d225t\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 18:45:01.164469 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:01.164435    2586 generic.go:356] "Generic (PLEG): container finished" podID="605e74c3-34fd-41c6-b1d5-cd6656d45243" containerID="11f2a3438a3692242d03c14c792f7d3ec10da509d851bf587a3ae7aea43ea60c" exitCode=0
Sep 16 18:45:01.164884 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:01.164484    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d4ddd4649-jmfhq"
Sep 16 18:45:01.164884 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:01.164510    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d4ddd4649-jmfhq" event={"ID":"605e74c3-34fd-41c6-b1d5-cd6656d45243","Type":"ContainerDied","Data":"11f2a3438a3692242d03c14c792f7d3ec10da509d851bf587a3ae7aea43ea60c"}
Sep 16 18:45:01.164884 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:01.164550    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d4ddd4649-jmfhq" event={"ID":"605e74c3-34fd-41c6-b1d5-cd6656d45243","Type":"ContainerDied","Data":"f75e250fdf436ec73d22f9543caef3482b02466ab7c85c4a461dfe891836bc7f"}
Sep 16 18:45:01.164884 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:01.164580    2586 scope.go:117] "RemoveContainer" containerID="11f2a3438a3692242d03c14c792f7d3ec10da509d851bf587a3ae7aea43ea60c"
Sep 16 18:45:01.172632 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:01.172612    2586 scope.go:117] "RemoveContainer" containerID="11f2a3438a3692242d03c14c792f7d3ec10da509d851bf587a3ae7aea43ea60c"
Sep 16 18:45:01.172909 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:45:01.172883    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11f2a3438a3692242d03c14c792f7d3ec10da509d851bf587a3ae7aea43ea60c\": container with ID starting with 11f2a3438a3692242d03c14c792f7d3ec10da509d851bf587a3ae7aea43ea60c not found: ID does not exist" containerID="11f2a3438a3692242d03c14c792f7d3ec10da509d851bf587a3ae7aea43ea60c"
Sep 16 18:45:01.172952 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:01.172921    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11f2a3438a3692242d03c14c792f7d3ec10da509d851bf587a3ae7aea43ea60c"} err="failed to get container status \"11f2a3438a3692242d03c14c792f7d3ec10da509d851bf587a3ae7aea43ea60c\": rpc error: code = NotFound desc = could not find container \"11f2a3438a3692242d03c14c792f7d3ec10da509d851bf587a3ae7aea43ea60c\": container with ID starting with 11f2a3438a3692242d03c14c792f7d3ec10da509d851bf587a3ae7aea43ea60c not found: ID does not exist"
Sep 16 18:45:01.188129 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:01.188096    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d4ddd4649-jmfhq"]
Sep 16 18:45:01.190631 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:01.190609    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7d4ddd4649-jmfhq"]
Sep 16 18:45:03.091486 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:03.091450    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="605e74c3-34fd-41c6-b1d5-cd6656d45243" path="/var/lib/kubelet/pods/605e74c3-34fd-41c6-b1d5-cd6656d45243/volumes"
Sep 16 18:45:24.989502 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:24.989479    2586 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 18:45:40.190518 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:40.190487    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-79d487d845-9s8sg"]
Sep 16 18:45:40.191980 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:40.190774    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="605e74c3-34fd-41c6-b1d5-cd6656d45243" containerName="console"
Sep 16 18:45:40.191980 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:40.190784    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="605e74c3-34fd-41c6-b1d5-cd6656d45243" containerName="console"
Sep 16 18:45:40.191980 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:40.190835    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="605e74c3-34fd-41c6-b1d5-cd6656d45243" containerName="console"
Sep 16 18:45:40.192604 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:40.192587    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79d487d845-9s8sg"
Sep 16 18:45:40.207339 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:40.207314    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1313ba4d-e524-4280-8e80-671e4491ed31-console-serving-cert\") pod \"console-79d487d845-9s8sg\" (UID: \"1313ba4d-e524-4280-8e80-671e4491ed31\") " pod="openshift-console/console-79d487d845-9s8sg"
Sep 16 18:45:40.207447 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:40.207348    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1313ba4d-e524-4280-8e80-671e4491ed31-trusted-ca-bundle\") pod \"console-79d487d845-9s8sg\" (UID: \"1313ba4d-e524-4280-8e80-671e4491ed31\") " pod="openshift-console/console-79d487d845-9s8sg"
Sep 16 18:45:40.207447 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:40.207366    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1313ba4d-e524-4280-8e80-671e4491ed31-console-oauth-config\") pod \"console-79d487d845-9s8sg\" (UID: \"1313ba4d-e524-4280-8e80-671e4491ed31\") " pod="openshift-console/console-79d487d845-9s8sg"
Sep 16 18:45:40.207514 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:40.207447    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q6q2\" (UniqueName: \"kubernetes.io/projected/1313ba4d-e524-4280-8e80-671e4491ed31-kube-api-access-5q6q2\") pod \"console-79d487d845-9s8sg\" (UID: \"1313ba4d-e524-4280-8e80-671e4491ed31\") " pod="openshift-console/console-79d487d845-9s8sg"
Sep 16 18:45:40.207514 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:40.207482    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1313ba4d-e524-4280-8e80-671e4491ed31-oauth-serving-cert\") pod \"console-79d487d845-9s8sg\" (UID: \"1313ba4d-e524-4280-8e80-671e4491ed31\") " pod="openshift-console/console-79d487d845-9s8sg"
Sep 16 18:45:40.207514 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:40.207502    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1313ba4d-e524-4280-8e80-671e4491ed31-console-config\") pod \"console-79d487d845-9s8sg\" (UID: \"1313ba4d-e524-4280-8e80-671e4491ed31\") " pod="openshift-console/console-79d487d845-9s8sg"
Sep 16 18:45:40.207610 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:40.207517    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1313ba4d-e524-4280-8e80-671e4491ed31-service-ca\") pod \"console-79d487d845-9s8sg\" (UID: \"1313ba4d-e524-4280-8e80-671e4491ed31\") " pod="openshift-console/console-79d487d845-9s8sg"
Sep 16 18:45:40.209911 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:40.209887    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79d487d845-9s8sg"]
Sep 16 18:45:40.307964 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:40.307929    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q6q2\" (UniqueName: \"kubernetes.io/projected/1313ba4d-e524-4280-8e80-671e4491ed31-kube-api-access-5q6q2\") pod \"console-79d487d845-9s8sg\" (UID: \"1313ba4d-e524-4280-8e80-671e4491ed31\") " pod="openshift-console/console-79d487d845-9s8sg"
Sep 16 18:45:40.307964 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:40.307969    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1313ba4d-e524-4280-8e80-671e4491ed31-oauth-serving-cert\") pod \"console-79d487d845-9s8sg\" (UID: \"1313ba4d-e524-4280-8e80-671e4491ed31\") " pod="openshift-console/console-79d487d845-9s8sg"
Sep 16 18:45:40.308219 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:40.307991    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1313ba4d-e524-4280-8e80-671e4491ed31-console-config\") pod \"console-79d487d845-9s8sg\" (UID: \"1313ba4d-e524-4280-8e80-671e4491ed31\") " pod="openshift-console/console-79d487d845-9s8sg"
Sep 16 18:45:40.308219 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:40.308008    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1313ba4d-e524-4280-8e80-671e4491ed31-service-ca\") pod \"console-79d487d845-9s8sg\" (UID: \"1313ba4d-e524-4280-8e80-671e4491ed31\") " pod="openshift-console/console-79d487d845-9s8sg"
Sep 16 18:45:40.308219 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:40.308045    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1313ba4d-e524-4280-8e80-671e4491ed31-console-serving-cert\") pod \"console-79d487d845-9s8sg\" (UID: \"1313ba4d-e524-4280-8e80-671e4491ed31\") " pod="openshift-console/console-79d487d845-9s8sg"
Sep 16 18:45:40.308219 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:40.308071    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1313ba4d-e524-4280-8e80-671e4491ed31-trusted-ca-bundle\") pod \"console-79d487d845-9s8sg\" (UID: \"1313ba4d-e524-4280-8e80-671e4491ed31\") " pod="openshift-console/console-79d487d845-9s8sg"
Sep 16 18:45:40.308219 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:40.308089    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1313ba4d-e524-4280-8e80-671e4491ed31-console-oauth-config\") pod \"console-79d487d845-9s8sg\" (UID: \"1313ba4d-e524-4280-8e80-671e4491ed31\") " pod="openshift-console/console-79d487d845-9s8sg"
Sep 16 18:45:40.308867 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:40.308840    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1313ba4d-e524-4280-8e80-671e4491ed31-console-config\") pod \"console-79d487d845-9s8sg\" (UID: \"1313ba4d-e524-4280-8e80-671e4491ed31\") " pod="openshift-console/console-79d487d845-9s8sg"
Sep 16 18:45:40.309000 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:40.308840    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1313ba4d-e524-4280-8e80-671e4491ed31-service-ca\") pod \"console-79d487d845-9s8sg\" (UID: \"1313ba4d-e524-4280-8e80-671e4491ed31\") " pod="openshift-console/console-79d487d845-9s8sg"
Sep 16 18:45:40.309064 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:40.309017    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1313ba4d-e524-4280-8e80-671e4491ed31-trusted-ca-bundle\") pod \"console-79d487d845-9s8sg\" (UID: \"1313ba4d-e524-4280-8e80-671e4491ed31\") " pod="openshift-console/console-79d487d845-9s8sg"
Sep 16 18:45:40.309362 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:40.309339    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1313ba4d-e524-4280-8e80-671e4491ed31-oauth-serving-cert\") pod \"console-79d487d845-9s8sg\" (UID: \"1313ba4d-e524-4280-8e80-671e4491ed31\") " pod="openshift-console/console-79d487d845-9s8sg"
Sep 16 18:45:40.310699 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:40.310679    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1313ba4d-e524-4280-8e80-671e4491ed31-console-serving-cert\") pod \"console-79d487d845-9s8sg\" (UID: \"1313ba4d-e524-4280-8e80-671e4491ed31\") " pod="openshift-console/console-79d487d845-9s8sg"
Sep 16 18:45:40.311078 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:40.311058    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1313ba4d-e524-4280-8e80-671e4491ed31-console-oauth-config\") pod \"console-79d487d845-9s8sg\" (UID: \"1313ba4d-e524-4280-8e80-671e4491ed31\") " pod="openshift-console/console-79d487d845-9s8sg"
Sep 16 18:45:40.317681 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:40.317662    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q6q2\" (UniqueName: \"kubernetes.io/projected/1313ba4d-e524-4280-8e80-671e4491ed31-kube-api-access-5q6q2\") pod \"console-79d487d845-9s8sg\" (UID: \"1313ba4d-e524-4280-8e80-671e4491ed31\") " pod="openshift-console/console-79d487d845-9s8sg"
Sep 16 18:45:40.502220 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:40.502121    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79d487d845-9s8sg"
Sep 16 18:45:40.776121 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:40.776047    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79d487d845-9s8sg"]
Sep 16 18:45:40.779451 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:45:40.779425    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1313ba4d_e524_4280_8e80_671e4491ed31.slice/crio-2c398d9efbdf0f0cae6bc03c56eb1fe5ce3cc55cdc552aa5d69f944f2383bb46 WatchSource:0}: Error finding container 2c398d9efbdf0f0cae6bc03c56eb1fe5ce3cc55cdc552aa5d69f944f2383bb46: Status 404 returned error can't find the container with id 2c398d9efbdf0f0cae6bc03c56eb1fe5ce3cc55cdc552aa5d69f944f2383bb46
Sep 16 18:45:41.317142 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:41.317104    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79d487d845-9s8sg" event={"ID":"1313ba4d-e524-4280-8e80-671e4491ed31","Type":"ContainerStarted","Data":"f8ddce5ffbd4d4657adfc65f14dda4a6893f60b1b5baea24a37c4cfdcc0808eb"}
Sep 16 18:45:41.317563 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:41.317171    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79d487d845-9s8sg" event={"ID":"1313ba4d-e524-4280-8e80-671e4491ed31","Type":"ContainerStarted","Data":"2c398d9efbdf0f0cae6bc03c56eb1fe5ce3cc55cdc552aa5d69f944f2383bb46"}
Sep 16 18:45:41.340442 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:41.340397    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-79d487d845-9s8sg" podStartSLOduration=1.340381381 podStartE2EDuration="1.340381381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:41.339798247 +0000 UTC m=+316.783659836" watchObservedRunningTime="2026-09-16 18:45:41.340381381 +0000 UTC m=+316.784242957"
Sep 16 18:45:50.502980 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:50.502942    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-79d487d845-9s8sg"
Sep 16 18:45:50.503413 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:50.503110    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-79d487d845-9s8sg"
Sep 16 18:45:50.508070 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:50.508047    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-79d487d845-9s8sg"
Sep 16 18:45:51.358276 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:51.358245    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-79d487d845-9s8sg"
Sep 16 18:45:51.433421 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:45:51.433379    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59bb745b6f-gh5tc"]
Sep 16 18:46:16.453616 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:46:16.453569    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-59bb745b6f-gh5tc" podUID="b18aa2f3-1cc3-4e45-97b7-122a66d979d2" containerName="console" containerID="cri-o://53ffd4fbc8f628d7ccb2690f490bb4a89f060f0f50c9312706e4e723f0b715f6" gracePeriod=15
Sep 16 18:46:16.682729 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:46:16.682706    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59bb745b6f-gh5tc"
Sep 16 18:46:16.816471 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:46:16.816359    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b18aa2f3-1cc3-4e45-97b7-122a66d979d2-oauth-serving-cert\") pod \"b18aa2f3-1cc3-4e45-97b7-122a66d979d2\" (UID: \"b18aa2f3-1cc3-4e45-97b7-122a66d979d2\") "
Sep 16 18:46:16.816471 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:46:16.816425    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dtmlw\" (UniqueName: \"kubernetes.io/projected/b18aa2f3-1cc3-4e45-97b7-122a66d979d2-kube-api-access-dtmlw\") pod \"b18aa2f3-1cc3-4e45-97b7-122a66d979d2\" (UID: \"b18aa2f3-1cc3-4e45-97b7-122a66d979d2\") "
Sep 16 18:46:16.816471 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:46:16.816464    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b18aa2f3-1cc3-4e45-97b7-122a66d979d2-console-config\") pod \"b18aa2f3-1cc3-4e45-97b7-122a66d979d2\" (UID: \"b18aa2f3-1cc3-4e45-97b7-122a66d979d2\") "
Sep 16 18:46:16.816756 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:46:16.816554    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b18aa2f3-1cc3-4e45-97b7-122a66d979d2-service-ca\") pod \"b18aa2f3-1cc3-4e45-97b7-122a66d979d2\" (UID: \"b18aa2f3-1cc3-4e45-97b7-122a66d979d2\") "
Sep 16 18:46:16.816756 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:46:16.816602    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b18aa2f3-1cc3-4e45-97b7-122a66d979d2-console-oauth-config\") pod \"b18aa2f3-1cc3-4e45-97b7-122a66d979d2\" (UID: \"b18aa2f3-1cc3-4e45-97b7-122a66d979d2\") "
Sep 16 18:46:16.816756 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:46:16.816627    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b18aa2f3-1cc3-4e45-97b7-122a66d979d2-trusted-ca-bundle\") pod \"b18aa2f3-1cc3-4e45-97b7-122a66d979d2\" (UID: \"b18aa2f3-1cc3-4e45-97b7-122a66d979d2\") "
Sep 16 18:46:16.816756 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:46:16.816669    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b18aa2f3-1cc3-4e45-97b7-122a66d979d2-console-serving-cert\") pod \"b18aa2f3-1cc3-4e45-97b7-122a66d979d2\" (UID: \"b18aa2f3-1cc3-4e45-97b7-122a66d979d2\") "
Sep 16 18:46:16.816947 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:46:16.816815    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b18aa2f3-1cc3-4e45-97b7-122a66d979d2-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b18aa2f3-1cc3-4e45-97b7-122a66d979d2" (UID: "b18aa2f3-1cc3-4e45-97b7-122a66d979d2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:46:16.816947 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:46:16.816840    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b18aa2f3-1cc3-4e45-97b7-122a66d979d2-console-config" (OuterVolumeSpecName: "console-config") pod "b18aa2f3-1cc3-4e45-97b7-122a66d979d2" (UID: "b18aa2f3-1cc3-4e45-97b7-122a66d979d2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:46:16.817103 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:46:16.817072    2586 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b18aa2f3-1cc3-4e45-97b7-122a66d979d2-oauth-serving-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 18:46:16.817264 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:46:16.817127    2586 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b18aa2f3-1cc3-4e45-97b7-122a66d979d2-console-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 18:46:16.817264 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:46:16.817061    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b18aa2f3-1cc3-4e45-97b7-122a66d979d2-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b18aa2f3-1cc3-4e45-97b7-122a66d979d2" (UID: "b18aa2f3-1cc3-4e45-97b7-122a66d979d2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:46:16.817371 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:46:16.817259    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b18aa2f3-1cc3-4e45-97b7-122a66d979d2-service-ca" (OuterVolumeSpecName: "service-ca") pod "b18aa2f3-1cc3-4e45-97b7-122a66d979d2" (UID: "b18aa2f3-1cc3-4e45-97b7-122a66d979d2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:46:16.818699 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:46:16.818673    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b18aa2f3-1cc3-4e45-97b7-122a66d979d2-kube-api-access-dtmlw" (OuterVolumeSpecName: "kube-api-access-dtmlw") pod "b18aa2f3-1cc3-4e45-97b7-122a66d979d2" (UID: "b18aa2f3-1cc3-4e45-97b7-122a66d979d2"). InnerVolumeSpecName "kube-api-access-dtmlw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:46:16.818784 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:46:16.818719    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b18aa2f3-1cc3-4e45-97b7-122a66d979d2-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b18aa2f3-1cc3-4e45-97b7-122a66d979d2" (UID: "b18aa2f3-1cc3-4e45-97b7-122a66d979d2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:46:16.818784 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:46:16.818751    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b18aa2f3-1cc3-4e45-97b7-122a66d979d2-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b18aa2f3-1cc3-4e45-97b7-122a66d979d2" (UID: "b18aa2f3-1cc3-4e45-97b7-122a66d979d2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:46:16.917881 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:46:16.917854    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtmlw\" (UniqueName: \"kubernetes.io/projected/b18aa2f3-1cc3-4e45-97b7-122a66d979d2-kube-api-access-dtmlw\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 18:46:16.917881 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:46:16.917879    2586 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b18aa2f3-1cc3-4e45-97b7-122a66d979d2-service-ca\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 18:46:16.918028 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:46:16.917888    2586 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b18aa2f3-1cc3-4e45-97b7-122a66d979d2-console-oauth-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 18:46:16.918028 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:46:16.917897    2586 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b18aa2f3-1cc3-4e45-97b7-122a66d979d2-trusted-ca-bundle\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 18:46:16.918028 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:46:16.917905    2586 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b18aa2f3-1cc3-4e45-97b7-122a66d979d2-console-serving-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 18:46:17.450103 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:46:17.449990    2586 generic.go:356] "Generic (PLEG): container finished" podID="b18aa2f3-1cc3-4e45-97b7-122a66d979d2" containerID="53ffd4fbc8f628d7ccb2690f490bb4a89f060f0f50c9312706e4e723f0b715f6" exitCode=0
Sep 16 18:46:17.450103 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:46:17.450052    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59bb745b6f-gh5tc"
Sep 16 18:46:17.450103 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:46:17.450074    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59bb745b6f-gh5tc" event={"ID":"b18aa2f3-1cc3-4e45-97b7-122a66d979d2","Type":"ContainerDied","Data":"53ffd4fbc8f628d7ccb2690f490bb4a89f060f0f50c9312706e4e723f0b715f6"}
Sep 16 18:46:17.450380 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:46:17.450160    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59bb745b6f-gh5tc" event={"ID":"b18aa2f3-1cc3-4e45-97b7-122a66d979d2","Type":"ContainerDied","Data":"8a65c8d76f2f43a62a85ad1c30b99a3379f0b68a4ad6236bb6314d36f768cd05"}
Sep 16 18:46:17.450380 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:46:17.450186    2586 scope.go:117] "RemoveContainer" containerID="53ffd4fbc8f628d7ccb2690f490bb4a89f060f0f50c9312706e4e723f0b715f6"
Sep 16 18:46:17.458626 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:46:17.458456    2586 scope.go:117] "RemoveContainer" containerID="53ffd4fbc8f628d7ccb2690f490bb4a89f060f0f50c9312706e4e723f0b715f6"
Sep 16 18:46:17.458840 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:46:17.458723    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53ffd4fbc8f628d7ccb2690f490bb4a89f060f0f50c9312706e4e723f0b715f6\": container with ID starting with 53ffd4fbc8f628d7ccb2690f490bb4a89f060f0f50c9312706e4e723f0b715f6 not found: ID does not exist" containerID="53ffd4fbc8f628d7ccb2690f490bb4a89f060f0f50c9312706e4e723f0b715f6"
Sep 16 18:46:17.458840 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:46:17.458748    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53ffd4fbc8f628d7ccb2690f490bb4a89f060f0f50c9312706e4e723f0b715f6"} err="failed to get container status \"53ffd4fbc8f628d7ccb2690f490bb4a89f060f0f50c9312706e4e723f0b715f6\": rpc error: code = NotFound desc = could not find container \"53ffd4fbc8f628d7ccb2690f490bb4a89f060f0f50c9312706e4e723f0b715f6\": container with ID starting with 53ffd4fbc8f628d7ccb2690f490bb4a89f060f0f50c9312706e4e723f0b715f6 not found: ID does not exist"
Sep 16 18:46:17.468572 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:46:17.468552    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59bb745b6f-gh5tc"]
Sep 16 18:46:17.471277 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:46:17.471259    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-59bb745b6f-gh5tc"]
Sep 16 18:46:19.091794 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:46:19.091757    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b18aa2f3-1cc3-4e45-97b7-122a66d979d2" path="/var/lib/kubelet/pods/b18aa2f3-1cc3-4e45-97b7-122a66d979d2/volumes"
Sep 16 18:47:56.042949 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:47:56.042875    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-656b4bf9b7-f8rl7"]
Sep 16 18:47:56.043394 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:47:56.043148    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b18aa2f3-1cc3-4e45-97b7-122a66d979d2" containerName="console"
Sep 16 18:47:56.043394 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:47:56.043159    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="b18aa2f3-1cc3-4e45-97b7-122a66d979d2" containerName="console"
Sep 16 18:47:56.043394 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:47:56.043211    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="b18aa2f3-1cc3-4e45-97b7-122a66d979d2" containerName="console"
Sep 16 18:47:56.044972 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:47:56.044954    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-656b4bf9b7-f8rl7"
Sep 16 18:47:56.100251 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:47:56.100206    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c973db92-be99-42ac-8a85-96d211f806a4-service-ca\") pod \"console-656b4bf9b7-f8rl7\" (UID: \"c973db92-be99-42ac-8a85-96d211f806a4\") " pod="openshift-console/console-656b4bf9b7-f8rl7"
Sep 16 18:47:56.100406 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:47:56.100261    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c973db92-be99-42ac-8a85-96d211f806a4-console-serving-cert\") pod \"console-656b4bf9b7-f8rl7\" (UID: \"c973db92-be99-42ac-8a85-96d211f806a4\") " pod="openshift-console/console-656b4bf9b7-f8rl7"
Sep 16 18:47:56.100406 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:47:56.100298    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c973db92-be99-42ac-8a85-96d211f806a4-console-config\") pod \"console-656b4bf9b7-f8rl7\" (UID: \"c973db92-be99-42ac-8a85-96d211f806a4\") " pod="openshift-console/console-656b4bf9b7-f8rl7"
Sep 16 18:47:56.100406 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:47:56.100350    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c973db92-be99-42ac-8a85-96d211f806a4-console-oauth-config\") pod \"console-656b4bf9b7-f8rl7\" (UID: \"c973db92-be99-42ac-8a85-96d211f806a4\") " pod="openshift-console/console-656b4bf9b7-f8rl7"
Sep 16 18:47:56.100406 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:47:56.100370    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c973db92-be99-42ac-8a85-96d211f806a4-trusted-ca-bundle\") pod \"console-656b4bf9b7-f8rl7\" (UID: \"c973db92-be99-42ac-8a85-96d211f806a4\") " pod="openshift-console/console-656b4bf9b7-f8rl7"
Sep 16 18:47:56.100406 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:47:56.100391    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8nzk\" (UniqueName: \"kubernetes.io/projected/c973db92-be99-42ac-8a85-96d211f806a4-kube-api-access-k8nzk\") pod \"console-656b4bf9b7-f8rl7\" (UID: \"c973db92-be99-42ac-8a85-96d211f806a4\") " pod="openshift-console/console-656b4bf9b7-f8rl7"
Sep 16 18:47:56.100629 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:47:56.100466    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c973db92-be99-42ac-8a85-96d211f806a4-oauth-serving-cert\") pod \"console-656b4bf9b7-f8rl7\" (UID: \"c973db92-be99-42ac-8a85-96d211f806a4\") " pod="openshift-console/console-656b4bf9b7-f8rl7"
Sep 16 18:47:56.130948 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:47:56.130914    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-656b4bf9b7-f8rl7"]
Sep 16 18:47:56.201623 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:47:56.201589    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c973db92-be99-42ac-8a85-96d211f806a4-console-oauth-config\") pod \"console-656b4bf9b7-f8rl7\" (UID: \"c973db92-be99-42ac-8a85-96d211f806a4\") " pod="openshift-console/console-656b4bf9b7-f8rl7"
Sep 16 18:47:56.201623 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:47:56.201624    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c973db92-be99-42ac-8a85-96d211f806a4-trusted-ca-bundle\") pod \"console-656b4bf9b7-f8rl7\" (UID: \"c973db92-be99-42ac-8a85-96d211f806a4\") " pod="openshift-console/console-656b4bf9b7-f8rl7"
Sep 16 18:47:56.201842 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:47:56.201646    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8nzk\" (UniqueName: \"kubernetes.io/projected/c973db92-be99-42ac-8a85-96d211f806a4-kube-api-access-k8nzk\") pod \"console-656b4bf9b7-f8rl7\" (UID: \"c973db92-be99-42ac-8a85-96d211f806a4\") " pod="openshift-console/console-656b4bf9b7-f8rl7"
Sep 16 18:47:56.201842 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:47:56.201669    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c973db92-be99-42ac-8a85-96d211f806a4-oauth-serving-cert\") pod \"console-656b4bf9b7-f8rl7\" (UID: \"c973db92-be99-42ac-8a85-96d211f806a4\") " pod="openshift-console/console-656b4bf9b7-f8rl7"
Sep 16 18:47:56.201842 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:47:56.201720    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c973db92-be99-42ac-8a85-96d211f806a4-service-ca\") pod \"console-656b4bf9b7-f8rl7\" (UID: \"c973db92-be99-42ac-8a85-96d211f806a4\") " pod="openshift-console/console-656b4bf9b7-f8rl7"
Sep 16 18:47:56.201842 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:47:56.201739    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c973db92-be99-42ac-8a85-96d211f806a4-console-serving-cert\") pod \"console-656b4bf9b7-f8rl7\" (UID: \"c973db92-be99-42ac-8a85-96d211f806a4\") " pod="openshift-console/console-656b4bf9b7-f8rl7"
Sep 16 18:47:56.201842 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:47:56.201780    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c973db92-be99-42ac-8a85-96d211f806a4-console-config\") pod \"console-656b4bf9b7-f8rl7\" (UID: \"c973db92-be99-42ac-8a85-96d211f806a4\") " pod="openshift-console/console-656b4bf9b7-f8rl7"
Sep 16 18:47:56.202566 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:47:56.202534    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c973db92-be99-42ac-8a85-96d211f806a4-oauth-serving-cert\") pod \"console-656b4bf9b7-f8rl7\" (UID: \"c973db92-be99-42ac-8a85-96d211f806a4\") " pod="openshift-console/console-656b4bf9b7-f8rl7"
Sep 16 18:47:56.202740 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:47:56.202604    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c973db92-be99-42ac-8a85-96d211f806a4-trusted-ca-bundle\") pod \"console-656b4bf9b7-f8rl7\" (UID: \"c973db92-be99-42ac-8a85-96d211f806a4\") " pod="openshift-console/console-656b4bf9b7-f8rl7"
Sep 16 18:47:56.202740 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:47:56.202708    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c973db92-be99-42ac-8a85-96d211f806a4-console-config\") pod \"console-656b4bf9b7-f8rl7\" (UID: \"c973db92-be99-42ac-8a85-96d211f806a4\") " pod="openshift-console/console-656b4bf9b7-f8rl7"
Sep 16 18:47:56.202843 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:47:56.202762    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c973db92-be99-42ac-8a85-96d211f806a4-service-ca\") pod \"console-656b4bf9b7-f8rl7\" (UID: \"c973db92-be99-42ac-8a85-96d211f806a4\") " pod="openshift-console/console-656b4bf9b7-f8rl7"
Sep 16 18:47:56.204398 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:47:56.204378    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c973db92-be99-42ac-8a85-96d211f806a4-console-serving-cert\") pod \"console-656b4bf9b7-f8rl7\" (UID: \"c973db92-be99-42ac-8a85-96d211f806a4\") " pod="openshift-console/console-656b4bf9b7-f8rl7"
Sep 16 18:47:56.204840 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:47:56.204823    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c973db92-be99-42ac-8a85-96d211f806a4-console-oauth-config\") pod \"console-656b4bf9b7-f8rl7\" (UID: \"c973db92-be99-42ac-8a85-96d211f806a4\") " pod="openshift-console/console-656b4bf9b7-f8rl7"
Sep 16 18:47:56.212153 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:47:56.212132    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8nzk\" (UniqueName: \"kubernetes.io/projected/c973db92-be99-42ac-8a85-96d211f806a4-kube-api-access-k8nzk\") pod \"console-656b4bf9b7-f8rl7\" (UID: \"c973db92-be99-42ac-8a85-96d211f806a4\") " pod="openshift-console/console-656b4bf9b7-f8rl7"
Sep 16 18:47:56.354545 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:47:56.354512    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-656b4bf9b7-f8rl7"
Sep 16 18:47:56.633059 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:47:56.632982    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-656b4bf9b7-f8rl7"]
Sep 16 18:47:56.636052 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:47:56.636022    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc973db92_be99_42ac_8a85_96d211f806a4.slice/crio-467e271ad118a4af5e6d21a79c719aefc6f897e1606d40bd7754fc5418e037dd WatchSource:0}: Error finding container 467e271ad118a4af5e6d21a79c719aefc6f897e1606d40bd7754fc5418e037dd: Status 404 returned error can't find the container with id 467e271ad118a4af5e6d21a79c719aefc6f897e1606d40bd7754fc5418e037dd
Sep 16 18:47:56.817847 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:47:56.817809    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-656b4bf9b7-f8rl7" event={"ID":"c973db92-be99-42ac-8a85-96d211f806a4","Type":"ContainerStarted","Data":"8612d9012f538b9e62932d2913ed16e2b27e03e8a72045b01567b4dbad31be41"}
Sep 16 18:47:56.818025 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:47:56.817856    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-656b4bf9b7-f8rl7" event={"ID":"c973db92-be99-42ac-8a85-96d211f806a4","Type":"ContainerStarted","Data":"467e271ad118a4af5e6d21a79c719aefc6f897e1606d40bd7754fc5418e037dd"}
Sep 16 18:47:56.841002 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:47:56.840951    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-656b4bf9b7-f8rl7" podStartSLOduration=0.840936452 podStartE2EDuration="840.936452ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:56.840136087 +0000 UTC m=+452.283997655" watchObservedRunningTime="2026-09-16 18:47:56.840936452 +0000 UTC m=+452.284798028"
Sep 16 18:47:59.444587 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:47:59.444550    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apv4xs"]
Sep 16 18:47:59.446893 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:47:59.446876    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apv4xs"
Sep 16 18:47:59.450195 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:47:59.450175    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 18:47:59.450866 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:47:59.450847    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 18:47:59.450926 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:47:59.450852    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-rssn7\""
Sep 16 18:47:59.458366 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:47:59.458344    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apv4xs"]
Sep 16 18:47:59.529437 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:47:59.529406    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8fe290e8-b54b-4c8a-a5d1-d653dbbc42d2-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apv4xs\" (UID: \"8fe290e8-b54b-4c8a-a5d1-d653dbbc42d2\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apv4xs"
Sep 16 18:47:59.529638 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:47:59.529461    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8fe290e8-b54b-4c8a-a5d1-d653dbbc42d2-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apv4xs\" (UID: \"8fe290e8-b54b-4c8a-a5d1-d653dbbc42d2\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apv4xs"
Sep 16 18:47:59.529638 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:47:59.529575    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk582\" (UniqueName: \"kubernetes.io/projected/8fe290e8-b54b-4c8a-a5d1-d653dbbc42d2-kube-api-access-gk582\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apv4xs\" (UID: \"8fe290e8-b54b-4c8a-a5d1-d653dbbc42d2\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apv4xs"
Sep 16 18:47:59.630555 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:47:59.630502    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk582\" (UniqueName: \"kubernetes.io/projected/8fe290e8-b54b-4c8a-a5d1-d653dbbc42d2-kube-api-access-gk582\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apv4xs\" (UID: \"8fe290e8-b54b-4c8a-a5d1-d653dbbc42d2\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apv4xs"
Sep 16 18:47:59.630708 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:47:59.630565    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8fe290e8-b54b-4c8a-a5d1-d653dbbc42d2-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apv4xs\" (UID: \"8fe290e8-b54b-4c8a-a5d1-d653dbbc42d2\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apv4xs"
Sep 16 18:47:59.630708 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:47:59.630593    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8fe290e8-b54b-4c8a-a5d1-d653dbbc42d2-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apv4xs\" (UID: \"8fe290e8-b54b-4c8a-a5d1-d653dbbc42d2\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apv4xs"
Sep 16 18:47:59.630939 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:47:59.630924    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8fe290e8-b54b-4c8a-a5d1-d653dbbc42d2-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apv4xs\" (UID: \"8fe290e8-b54b-4c8a-a5d1-d653dbbc42d2\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apv4xs"
Sep 16 18:47:59.630979 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:47:59.630955    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8fe290e8-b54b-4c8a-a5d1-d653dbbc42d2-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apv4xs\" (UID: \"8fe290e8-b54b-4c8a-a5d1-d653dbbc42d2\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apv4xs"
Sep 16 18:47:59.640753 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:47:59.640723    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk582\" (UniqueName: \"kubernetes.io/projected/8fe290e8-b54b-4c8a-a5d1-d653dbbc42d2-kube-api-access-gk582\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apv4xs\" (UID: \"8fe290e8-b54b-4c8a-a5d1-d653dbbc42d2\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apv4xs"
Sep 16 18:47:59.755884 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:47:59.755794    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apv4xs"
Sep 16 18:48:00.031555 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:00.031530    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apv4xs"]
Sep 16 18:48:00.033335 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:48:00.033305    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fe290e8_b54b_4c8a_a5d1_d653dbbc42d2.slice/crio-9259a4f9daf836d646affa1e015044a8e1e1820eb32b0e17a5db1d09a513487f WatchSource:0}: Error finding container 9259a4f9daf836d646affa1e015044a8e1e1820eb32b0e17a5db1d09a513487f: Status 404 returned error can't find the container with id 9259a4f9daf836d646affa1e015044a8e1e1820eb32b0e17a5db1d09a513487f
Sep 16 18:48:00.035360 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:00.035344    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:48:00.835009 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:00.834973    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apv4xs" event={"ID":"8fe290e8-b54b-4c8a-a5d1-d653dbbc42d2","Type":"ContainerStarted","Data":"9259a4f9daf836d646affa1e015044a8e1e1820eb32b0e17a5db1d09a513487f"}
Sep 16 18:48:05.857575 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:05.857538    2586 generic.go:356] "Generic (PLEG): container finished" podID="8fe290e8-b54b-4c8a-a5d1-d653dbbc42d2" containerID="7c96bb5c4f1ab30f1612f8f7849c4bb8782ee6e3a8fe72f5d85a9635240cfcff" exitCode=0
Sep 16 18:48:05.858043 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:05.857622    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apv4xs" event={"ID":"8fe290e8-b54b-4c8a-a5d1-d653dbbc42d2","Type":"ContainerDied","Data":"7c96bb5c4f1ab30f1612f8f7849c4bb8782ee6e3a8fe72f5d85a9635240cfcff"}
Sep 16 18:48:06.355099 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:06.355061    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-656b4bf9b7-f8rl7"
Sep 16 18:48:06.355309 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:06.355142    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-656b4bf9b7-f8rl7"
Sep 16 18:48:06.359730 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:06.359708    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-656b4bf9b7-f8rl7"
Sep 16 18:48:06.866324 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:06.866295    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-656b4bf9b7-f8rl7"
Sep 16 18:48:06.924119 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:06.924014    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79d487d845-9s8sg"]
Sep 16 18:48:08.871407 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:08.871376    2586 generic.go:356] "Generic (PLEG): container finished" podID="8fe290e8-b54b-4c8a-a5d1-d653dbbc42d2" containerID="a2d0768f0963a855c0dc188b683d949ecd6fff02eef422569a21040454573534" exitCode=0
Sep 16 18:48:08.871910 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:08.871474    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apv4xs" event={"ID":"8fe290e8-b54b-4c8a-a5d1-d653dbbc42d2","Type":"ContainerDied","Data":"a2d0768f0963a855c0dc188b683d949ecd6fff02eef422569a21040454573534"}
Sep 16 18:48:15.903486 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:15.903451    2586 generic.go:356] "Generic (PLEG): container finished" podID="8fe290e8-b54b-4c8a-a5d1-d653dbbc42d2" containerID="3ea5d090968257b814e4674d3da23ba989e4471f03fe1a16f1bd444b49b37fe2" exitCode=0
Sep 16 18:48:15.903850 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:15.903497    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apv4xs" event={"ID":"8fe290e8-b54b-4c8a-a5d1-d653dbbc42d2","Type":"ContainerDied","Data":"3ea5d090968257b814e4674d3da23ba989e4471f03fe1a16f1bd444b49b37fe2"}
Sep 16 18:48:17.031074 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:17.031051    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apv4xs"
Sep 16 18:48:17.184019 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:17.183922    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gk582\" (UniqueName: \"kubernetes.io/projected/8fe290e8-b54b-4c8a-a5d1-d653dbbc42d2-kube-api-access-gk582\") pod \"8fe290e8-b54b-4c8a-a5d1-d653dbbc42d2\" (UID: \"8fe290e8-b54b-4c8a-a5d1-d653dbbc42d2\") "
Sep 16 18:48:17.184019 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:17.184014    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8fe290e8-b54b-4c8a-a5d1-d653dbbc42d2-bundle\") pod \"8fe290e8-b54b-4c8a-a5d1-d653dbbc42d2\" (UID: \"8fe290e8-b54b-4c8a-a5d1-d653dbbc42d2\") "
Sep 16 18:48:17.184256 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:17.184050    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8fe290e8-b54b-4c8a-a5d1-d653dbbc42d2-util\") pod \"8fe290e8-b54b-4c8a-a5d1-d653dbbc42d2\" (UID: \"8fe290e8-b54b-4c8a-a5d1-d653dbbc42d2\") "
Sep 16 18:48:17.184881 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:17.184851    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fe290e8-b54b-4c8a-a5d1-d653dbbc42d2-bundle" (OuterVolumeSpecName: "bundle") pod "8fe290e8-b54b-4c8a-a5d1-d653dbbc42d2" (UID: "8fe290e8-b54b-4c8a-a5d1-d653dbbc42d2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:48:17.186133 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:17.186109    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fe290e8-b54b-4c8a-a5d1-d653dbbc42d2-kube-api-access-gk582" (OuterVolumeSpecName: "kube-api-access-gk582") pod "8fe290e8-b54b-4c8a-a5d1-d653dbbc42d2" (UID: "8fe290e8-b54b-4c8a-a5d1-d653dbbc42d2"). InnerVolumeSpecName "kube-api-access-gk582". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:48:17.188134 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:17.188106    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fe290e8-b54b-4c8a-a5d1-d653dbbc42d2-util" (OuterVolumeSpecName: "util") pod "8fe290e8-b54b-4c8a-a5d1-d653dbbc42d2" (UID: "8fe290e8-b54b-4c8a-a5d1-d653dbbc42d2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:48:17.284774 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:17.284740    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk582\" (UniqueName: \"kubernetes.io/projected/8fe290e8-b54b-4c8a-a5d1-d653dbbc42d2-kube-api-access-gk582\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 18:48:17.284774 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:17.284768    2586 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8fe290e8-b54b-4c8a-a5d1-d653dbbc42d2-bundle\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 18:48:17.284774 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:17.284777    2586 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8fe290e8-b54b-4c8a-a5d1-d653dbbc42d2-util\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 18:48:17.912970 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:17.912927    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apv4xs" event={"ID":"8fe290e8-b54b-4c8a-a5d1-d653dbbc42d2","Type":"ContainerDied","Data":"9259a4f9daf836d646affa1e015044a8e1e1820eb32b0e17a5db1d09a513487f"}
Sep 16 18:48:17.912970 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:17.912947    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1apv4xs"
Sep 16 18:48:17.913174 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:17.912990    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9259a4f9daf836d646affa1e015044a8e1e1820eb32b0e17a5db1d09a513487f"
Sep 16 18:48:22.636140 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:22.636102    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-xnlg2"]
Sep 16 18:48:22.636613 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:22.636427    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fe290e8-b54b-4c8a-a5d1-d653dbbc42d2" containerName="extract"
Sep 16 18:48:22.636613 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:22.636439    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fe290e8-b54b-4c8a-a5d1-d653dbbc42d2" containerName="extract"
Sep 16 18:48:22.636613 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:22.636452    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fe290e8-b54b-4c8a-a5d1-d653dbbc42d2" containerName="pull"
Sep 16 18:48:22.636613 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:22.636458    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fe290e8-b54b-4c8a-a5d1-d653dbbc42d2" containerName="pull"
Sep 16 18:48:22.636613 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:22.636467    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fe290e8-b54b-4c8a-a5d1-d653dbbc42d2" containerName="util"
Sep 16 18:48:22.636613 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:22.636473    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fe290e8-b54b-4c8a-a5d1-d653dbbc42d2" containerName="util"
Sep 16 18:48:22.636613 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:22.636518    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fe290e8-b54b-4c8a-a5d1-d653dbbc42d2" containerName="extract"
Sep 16 18:48:22.638378 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:22.638361    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-xnlg2"
Sep 16 18:48:22.640987 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:22.640959    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 16 18:48:22.640987 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:22.640984    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 16 18:48:22.641169 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:22.641002    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 16 18:48:22.641169 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:22.641074    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-w5555\""
Sep 16 18:48:22.653491 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:22.653467    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-xnlg2"]
Sep 16 18:48:22.734051 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:22.734015    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/3e103c61-cf8f-4a5f-8257-5fcf676525ef-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-xnlg2\" (UID: \"3e103c61-cf8f-4a5f-8257-5fcf676525ef\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-xnlg2"
Sep 16 18:48:22.734218 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:22.734080    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57ngn\" (UniqueName: \"kubernetes.io/projected/3e103c61-cf8f-4a5f-8257-5fcf676525ef-kube-api-access-57ngn\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-xnlg2\" (UID: \"3e103c61-cf8f-4a5f-8257-5fcf676525ef\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-xnlg2"
Sep 16 18:48:22.835183 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:22.835148    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/3e103c61-cf8f-4a5f-8257-5fcf676525ef-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-xnlg2\" (UID: \"3e103c61-cf8f-4a5f-8257-5fcf676525ef\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-xnlg2"
Sep 16 18:48:22.835361 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:22.835203    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57ngn\" (UniqueName: \"kubernetes.io/projected/3e103c61-cf8f-4a5f-8257-5fcf676525ef-kube-api-access-57ngn\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-xnlg2\" (UID: \"3e103c61-cf8f-4a5f-8257-5fcf676525ef\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-xnlg2"
Sep 16 18:48:22.837581 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:22.837552    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/3e103c61-cf8f-4a5f-8257-5fcf676525ef-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-xnlg2\" (UID: \"3e103c61-cf8f-4a5f-8257-5fcf676525ef\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-xnlg2"
Sep 16 18:48:22.850130 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:22.850103    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57ngn\" (UniqueName: \"kubernetes.io/projected/3e103c61-cf8f-4a5f-8257-5fcf676525ef-kube-api-access-57ngn\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-xnlg2\" (UID: \"3e103c61-cf8f-4a5f-8257-5fcf676525ef\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-xnlg2"
Sep 16 18:48:22.947717 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:22.947620    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-xnlg2"
Sep 16 18:48:23.225305 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:23.225199    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-xnlg2"]
Sep 16 18:48:23.228825 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:48:23.228798    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e103c61_cf8f_4a5f_8257_5fcf676525ef.slice/crio-f73cb56943fb36476cb78fb9a7c4ecf51edcebf9ae2d8f2d5784b81b63a08cb6 WatchSource:0}: Error finding container f73cb56943fb36476cb78fb9a7c4ecf51edcebf9ae2d8f2d5784b81b63a08cb6: Status 404 returned error can't find the container with id f73cb56943fb36476cb78fb9a7c4ecf51edcebf9ae2d8f2d5784b81b63a08cb6
Sep 16 18:48:23.938564 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:23.938523    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-xnlg2" event={"ID":"3e103c61-cf8f-4a5f-8257-5fcf676525ef","Type":"ContainerStarted","Data":"f73cb56943fb36476cb78fb9a7c4ecf51edcebf9ae2d8f2d5784b81b63a08cb6"}
Sep 16 18:48:26.954283 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:26.954242    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-xnlg2" event={"ID":"3e103c61-cf8f-4a5f-8257-5fcf676525ef","Type":"ContainerStarted","Data":"3eda89d5da8424d741e372edd4590bfdd770ad05b781c2a3e32b19ddb4166067"}
Sep 16 18:48:26.954778 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:26.954340    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-xnlg2"
Sep 16 18:48:26.988385 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:26.988328    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-xnlg2" podStartSLOduration=2.044991337 podStartE2EDuration="4.988309337s" totalImagesPullingTime="2.943318s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:26.987548853 +0000 UTC m=+482.431410449" watchObservedRunningTime="2026-09-16 18:48:26.988309337 +0000 UTC m=+482.432170912"
Sep 16 18:48:28.159833 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:28.159787    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-dpwbt"]
Sep 16 18:48:28.162351 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:28.162334    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-dpwbt"
Sep 16 18:48:28.165954 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:28.165930    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 16 18:48:28.166074 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:28.165992    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-xpkh2\""
Sep 16 18:48:28.167148 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:28.167125    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 16 18:48:28.177809 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:28.177784    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-dpwbt"]
Sep 16 18:48:28.180579 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:28.180552    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b3fdf125-aa62-4bc9-9ff9-d6f837a2584b-certificates\") pod \"keda-admission-6c4d864b86-dpwbt\" (UID: \"b3fdf125-aa62-4bc9-9ff9-d6f837a2584b\") " pod="openshift-keda/keda-admission-6c4d864b86-dpwbt"
Sep 16 18:48:28.180785 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:28.180738    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm2bx\" (UniqueName: \"kubernetes.io/projected/b3fdf125-aa62-4bc9-9ff9-d6f837a2584b-kube-api-access-zm2bx\") pod \"keda-admission-6c4d864b86-dpwbt\" (UID: \"b3fdf125-aa62-4bc9-9ff9-d6f837a2584b\") " pod="openshift-keda/keda-admission-6c4d864b86-dpwbt"
Sep 16 18:48:28.281522 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:28.281488    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b3fdf125-aa62-4bc9-9ff9-d6f837a2584b-certificates\") pod \"keda-admission-6c4d864b86-dpwbt\" (UID: \"b3fdf125-aa62-4bc9-9ff9-d6f837a2584b\") " pod="openshift-keda/keda-admission-6c4d864b86-dpwbt"
Sep 16 18:48:28.281680 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:28.281547    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm2bx\" (UniqueName: \"kubernetes.io/projected/b3fdf125-aa62-4bc9-9ff9-d6f837a2584b-kube-api-access-zm2bx\") pod \"keda-admission-6c4d864b86-dpwbt\" (UID: \"b3fdf125-aa62-4bc9-9ff9-d6f837a2584b\") " pod="openshift-keda/keda-admission-6c4d864b86-dpwbt"
Sep 16 18:48:28.281680 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:48:28.281657    2586 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Sep 16 18:48:28.281752 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:48:28.281688    2586 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-6c4d864b86-dpwbt: secret "keda-admission-webhooks-certs" not found
Sep 16 18:48:28.281785 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:48:28.281753    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3fdf125-aa62-4bc9-9ff9-d6f837a2584b-certificates podName:b3fdf125-aa62-4bc9-9ff9-d6f837a2584b nodeName:}" failed. No retries permitted until 2026-09-16 18:48:28.781733835 +0000 UTC m=+484.225595396 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/b3fdf125-aa62-4bc9-9ff9-d6f837a2584b-certificates") pod "keda-admission-6c4d864b86-dpwbt" (UID: "b3fdf125-aa62-4bc9-9ff9-d6f837a2584b") : secret "keda-admission-webhooks-certs" not found
Sep 16 18:48:28.291811 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:28.291783    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm2bx\" (UniqueName: \"kubernetes.io/projected/b3fdf125-aa62-4bc9-9ff9-d6f837a2584b-kube-api-access-zm2bx\") pod \"keda-admission-6c4d864b86-dpwbt\" (UID: \"b3fdf125-aa62-4bc9-9ff9-d6f837a2584b\") " pod="openshift-keda/keda-admission-6c4d864b86-dpwbt"
Sep 16 18:48:28.785080 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:28.785050    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b3fdf125-aa62-4bc9-9ff9-d6f837a2584b-certificates\") pod \"keda-admission-6c4d864b86-dpwbt\" (UID: \"b3fdf125-aa62-4bc9-9ff9-d6f837a2584b\") " pod="openshift-keda/keda-admission-6c4d864b86-dpwbt"
Sep 16 18:48:28.787407 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:28.787389    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b3fdf125-aa62-4bc9-9ff9-d6f837a2584b-certificates\") pod \"keda-admission-6c4d864b86-dpwbt\" (UID: \"b3fdf125-aa62-4bc9-9ff9-d6f837a2584b\") " pod="openshift-keda/keda-admission-6c4d864b86-dpwbt"
Sep 16 18:48:29.072525 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:29.072427    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-dpwbt"
Sep 16 18:48:29.357088 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:29.357051    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-dpwbt"]
Sep 16 18:48:29.362027 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:48:29.361999    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3fdf125_aa62_4bc9_9ff9_d6f837a2584b.slice/crio-b7e59bad2354595c76500393935ffadbd62eb4452a81ef6f3efd80e133f3139e WatchSource:0}: Error finding container b7e59bad2354595c76500393935ffadbd62eb4452a81ef6f3efd80e133f3139e: Status 404 returned error can't find the container with id b7e59bad2354595c76500393935ffadbd62eb4452a81ef6f3efd80e133f3139e
Sep 16 18:48:29.973055 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:29.973020    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-dpwbt" event={"ID":"b3fdf125-aa62-4bc9-9ff9-d6f837a2584b","Type":"ContainerStarted","Data":"b7e59bad2354595c76500393935ffadbd62eb4452a81ef6f3efd80e133f3139e"}
Sep 16 18:48:31.948216 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:31.948173    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-79d487d845-9s8sg" podUID="1313ba4d-e524-4280-8e80-671e4491ed31" containerName="console" containerID="cri-o://f8ddce5ffbd4d4657adfc65f14dda4a6893f60b1b5baea24a37c4cfdcc0808eb" gracePeriod=15
Sep 16 18:48:32.200206 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:32.200150    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79d487d845-9s8sg"
Sep 16 18:48:32.208305 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:32.208279    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1313ba4d-e524-4280-8e80-671e4491ed31-trusted-ca-bundle\") pod \"1313ba4d-e524-4280-8e80-671e4491ed31\" (UID: \"1313ba4d-e524-4280-8e80-671e4491ed31\") "
Sep 16 18:48:32.208414 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:32.208326    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5q6q2\" (UniqueName: \"kubernetes.io/projected/1313ba4d-e524-4280-8e80-671e4491ed31-kube-api-access-5q6q2\") pod \"1313ba4d-e524-4280-8e80-671e4491ed31\" (UID: \"1313ba4d-e524-4280-8e80-671e4491ed31\") "
Sep 16 18:48:32.208414 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:32.208353    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1313ba4d-e524-4280-8e80-671e4491ed31-service-ca\") pod \"1313ba4d-e524-4280-8e80-671e4491ed31\" (UID: \"1313ba4d-e524-4280-8e80-671e4491ed31\") "
Sep 16 18:48:32.208414 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:32.208381    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1313ba4d-e524-4280-8e80-671e4491ed31-console-serving-cert\") pod \"1313ba4d-e524-4280-8e80-671e4491ed31\" (UID: \"1313ba4d-e524-4280-8e80-671e4491ed31\") "
Sep 16 18:48:32.208534 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:32.208436    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1313ba4d-e524-4280-8e80-671e4491ed31-console-config\") pod \"1313ba4d-e524-4280-8e80-671e4491ed31\" (UID: \"1313ba4d-e524-4280-8e80-671e4491ed31\") "
Sep 16 18:48:32.208582 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:32.208564    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1313ba4d-e524-4280-8e80-671e4491ed31-console-oauth-config\") pod \"1313ba4d-e524-4280-8e80-671e4491ed31\" (UID: \"1313ba4d-e524-4280-8e80-671e4491ed31\") "
Sep 16 18:48:32.208633 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:32.208610    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1313ba4d-e524-4280-8e80-671e4491ed31-oauth-serving-cert\") pod \"1313ba4d-e524-4280-8e80-671e4491ed31\" (UID: \"1313ba4d-e524-4280-8e80-671e4491ed31\") "
Sep 16 18:48:32.208741 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:32.208716    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1313ba4d-e524-4280-8e80-671e4491ed31-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "1313ba4d-e524-4280-8e80-671e4491ed31" (UID: "1313ba4d-e524-4280-8e80-671e4491ed31"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:48:32.208856 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:32.208819    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1313ba4d-e524-4280-8e80-671e4491ed31-console-config" (OuterVolumeSpecName: "console-config") pod "1313ba4d-e524-4280-8e80-671e4491ed31" (UID: "1313ba4d-e524-4280-8e80-671e4491ed31"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:48:32.209029 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:32.208872    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1313ba4d-e524-4280-8e80-671e4491ed31-service-ca" (OuterVolumeSpecName: "service-ca") pod "1313ba4d-e524-4280-8e80-671e4491ed31" (UID: "1313ba4d-e524-4280-8e80-671e4491ed31"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:48:32.209029 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:32.208879    2586 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1313ba4d-e524-4280-8e80-671e4491ed31-trusted-ca-bundle\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 18:48:32.209029 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:32.208987    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1313ba4d-e524-4280-8e80-671e4491ed31-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1313ba4d-e524-4280-8e80-671e4491ed31" (UID: "1313ba4d-e524-4280-8e80-671e4491ed31"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:48:32.210535 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:32.210510    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1313ba4d-e524-4280-8e80-671e4491ed31-kube-api-access-5q6q2" (OuterVolumeSpecName: "kube-api-access-5q6q2") pod "1313ba4d-e524-4280-8e80-671e4491ed31" (UID: "1313ba4d-e524-4280-8e80-671e4491ed31"). InnerVolumeSpecName "kube-api-access-5q6q2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:48:32.210610 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:32.210552    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1313ba4d-e524-4280-8e80-671e4491ed31-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1313ba4d-e524-4280-8e80-671e4491ed31" (UID: "1313ba4d-e524-4280-8e80-671e4491ed31"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:48:32.210610 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:32.210590    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1313ba4d-e524-4280-8e80-671e4491ed31-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1313ba4d-e524-4280-8e80-671e4491ed31" (UID: "1313ba4d-e524-4280-8e80-671e4491ed31"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:48:32.309849 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:32.309820    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q6q2\" (UniqueName: \"kubernetes.io/projected/1313ba4d-e524-4280-8e80-671e4491ed31-kube-api-access-5q6q2\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 18:48:32.309849 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:32.309848    2586 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1313ba4d-e524-4280-8e80-671e4491ed31-service-ca\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 18:48:32.309849 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:32.309858    2586 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1313ba4d-e524-4280-8e80-671e4491ed31-console-serving-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 18:48:32.310063 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:32.309867    2586 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1313ba4d-e524-4280-8e80-671e4491ed31-console-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 18:48:32.310063 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:32.309875    2586 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1313ba4d-e524-4280-8e80-671e4491ed31-console-oauth-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 18:48:32.310063 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:32.309884    2586 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1313ba4d-e524-4280-8e80-671e4491ed31-oauth-serving-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 18:48:32.986897 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:32.986861    2586 generic.go:356] "Generic (PLEG): container finished" podID="1313ba4d-e524-4280-8e80-671e4491ed31" containerID="f8ddce5ffbd4d4657adfc65f14dda4a6893f60b1b5baea24a37c4cfdcc0808eb" exitCode=0
Sep 16 18:48:32.987386 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:32.986907    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79d487d845-9s8sg"
Sep 16 18:48:32.987386 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:32.986908    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79d487d845-9s8sg" event={"ID":"1313ba4d-e524-4280-8e80-671e4491ed31","Type":"ContainerDied","Data":"f8ddce5ffbd4d4657adfc65f14dda4a6893f60b1b5baea24a37c4cfdcc0808eb"}
Sep 16 18:48:32.987386 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:32.987053    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79d487d845-9s8sg" event={"ID":"1313ba4d-e524-4280-8e80-671e4491ed31","Type":"ContainerDied","Data":"2c398d9efbdf0f0cae6bc03c56eb1fe5ce3cc55cdc552aa5d69f944f2383bb46"}
Sep 16 18:48:32.987386 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:32.987090    2586 scope.go:117] "RemoveContainer" containerID="f8ddce5ffbd4d4657adfc65f14dda4a6893f60b1b5baea24a37c4cfdcc0808eb"
Sep 16 18:48:32.997619 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:32.997354    2586 scope.go:117] "RemoveContainer" containerID="f8ddce5ffbd4d4657adfc65f14dda4a6893f60b1b5baea24a37c4cfdcc0808eb"
Sep 16 18:48:32.997819 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:48:32.997797    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8ddce5ffbd4d4657adfc65f14dda4a6893f60b1b5baea24a37c4cfdcc0808eb\": container with ID starting with f8ddce5ffbd4d4657adfc65f14dda4a6893f60b1b5baea24a37c4cfdcc0808eb not found: ID does not exist" containerID="f8ddce5ffbd4d4657adfc65f14dda4a6893f60b1b5baea24a37c4cfdcc0808eb"
Sep 16 18:48:32.997885 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:32.997826    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8ddce5ffbd4d4657adfc65f14dda4a6893f60b1b5baea24a37c4cfdcc0808eb"} err="failed to get container status \"f8ddce5ffbd4d4657adfc65f14dda4a6893f60b1b5baea24a37c4cfdcc0808eb\": rpc error: code = NotFound desc = could not find container \"f8ddce5ffbd4d4657adfc65f14dda4a6893f60b1b5baea24a37c4cfdcc0808eb\": container with ID starting with f8ddce5ffbd4d4657adfc65f14dda4a6893f60b1b5baea24a37c4cfdcc0808eb not found: ID does not exist"
Sep 16 18:48:33.016135 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:33.016104    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79d487d845-9s8sg"]
Sep 16 18:48:33.017363 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:33.017346    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-79d487d845-9s8sg"]
Sep 16 18:48:33.091922 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:33.091889    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1313ba4d-e524-4280-8e80-671e4491ed31" path="/var/lib/kubelet/pods/1313ba4d-e524-4280-8e80-671e4491ed31/volumes"
Sep 16 18:48:42.026520 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:42.026480    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-dpwbt" event={"ID":"b3fdf125-aa62-4bc9-9ff9-d6f837a2584b","Type":"ContainerStarted","Data":"37c182d49a9c27dbb9321427e1e33be60ea487c12eee49206c85d0dba7d194d9"}
Sep 16 18:48:42.026987 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:42.026588    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-dpwbt"
Sep 16 18:48:42.044596 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:42.044548    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-dpwbt" podStartSLOduration=2.078345015 podStartE2EDuration="14.044533651s" totalImagesPullingTime="11.966188636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:42.044083226 +0000 UTC m=+497.487944802" watchObservedRunningTime="2026-09-16 18:48:42.044533651 +0000 UTC m=+497.488395228"
Sep 16 18:48:47.965380 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:48:47.965348    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-xnlg2"
Sep 16 18:49:03.033833 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:49:03.033800    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-dpwbt"
Sep 16 18:49:47.321274 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:49:47.321240    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-599b79d756-r94fw"]
Sep 16 18:49:47.321843 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:49:47.321540    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1313ba4d-e524-4280-8e80-671e4491ed31" containerName="console"
Sep 16 18:49:47.321843 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:49:47.321552    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="1313ba4d-e524-4280-8e80-671e4491ed31" containerName="console"
Sep 16 18:49:47.321843 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:49:47.321604    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="1313ba4d-e524-4280-8e80-671e4491ed31" containerName="console"
Sep 16 18:49:47.325521 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:49:47.325501    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-599b79d756-r94fw"
Sep 16 18:49:47.328359 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:49:47.328330    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 18:49:47.328467 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:49:47.328330    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 16 18:49:47.328467 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:49:47.328443    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-bt5mj\""
Sep 16 18:49:47.328575 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:49:47.328336    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 18:49:47.336244 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:49:47.336203    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-599b79d756-r94fw"]
Sep 16 18:49:47.358866 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:49:47.358835    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-xssm2"]
Sep 16 18:49:47.362110 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:49:47.362090    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-xssm2"
Sep 16 18:49:47.364478 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:49:47.364458    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 16 18:49:47.364593 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:49:47.364575    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-2q5x2\""
Sep 16 18:49:47.373304 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:49:47.373280    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-xssm2"]
Sep 16 18:49:47.413905 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:49:47.413869    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aea89c8a-49af-4a43-a722-8097fb7aef34-cert\") pod \"kserve-controller-manager-599b79d756-r94fw\" (UID: \"aea89c8a-49af-4a43-a722-8097fb7aef34\") " pod="kserve/kserve-controller-manager-599b79d756-r94fw"
Sep 16 18:49:47.414058 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:49:47.413975    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/12682bbb-5ad5-44a3-b405-f8840e7b2524-data\") pod \"seaweedfs-64568bcd49-xssm2\" (UID: \"12682bbb-5ad5-44a3-b405-f8840e7b2524\") " pod="kserve/seaweedfs-64568bcd49-xssm2"
Sep 16 18:49:47.414058 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:49:47.413994    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkv5t\" (UniqueName: \"kubernetes.io/projected/12682bbb-5ad5-44a3-b405-f8840e7b2524-kube-api-access-fkv5t\") pod \"seaweedfs-64568bcd49-xssm2\" (UID: \"12682bbb-5ad5-44a3-b405-f8840e7b2524\") " pod="kserve/seaweedfs-64568bcd49-xssm2"
Sep 16 18:49:47.414058 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:49:47.414017    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5k2k\" (UniqueName: \"kubernetes.io/projected/aea89c8a-49af-4a43-a722-8097fb7aef34-kube-api-access-t5k2k\") pod \"kserve-controller-manager-599b79d756-r94fw\" (UID: \"aea89c8a-49af-4a43-a722-8097fb7aef34\") " pod="kserve/kserve-controller-manager-599b79d756-r94fw"
Sep 16 18:49:47.515338 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:49:47.515305    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/12682bbb-5ad5-44a3-b405-f8840e7b2524-data\") pod \"seaweedfs-64568bcd49-xssm2\" (UID: \"12682bbb-5ad5-44a3-b405-f8840e7b2524\") " pod="kserve/seaweedfs-64568bcd49-xssm2"
Sep 16 18:49:47.515338 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:49:47.515340    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkv5t\" (UniqueName: \"kubernetes.io/projected/12682bbb-5ad5-44a3-b405-f8840e7b2524-kube-api-access-fkv5t\") pod \"seaweedfs-64568bcd49-xssm2\" (UID: \"12682bbb-5ad5-44a3-b405-f8840e7b2524\") " pod="kserve/seaweedfs-64568bcd49-xssm2"
Sep 16 18:49:47.515546 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:49:47.515363    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5k2k\" (UniqueName: \"kubernetes.io/projected/aea89c8a-49af-4a43-a722-8097fb7aef34-kube-api-access-t5k2k\") pod \"kserve-controller-manager-599b79d756-r94fw\" (UID: \"aea89c8a-49af-4a43-a722-8097fb7aef34\") " pod="kserve/kserve-controller-manager-599b79d756-r94fw"
Sep 16 18:49:47.515546 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:49:47.515482    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aea89c8a-49af-4a43-a722-8097fb7aef34-cert\") pod \"kserve-controller-manager-599b79d756-r94fw\" (UID: \"aea89c8a-49af-4a43-a722-8097fb7aef34\") " pod="kserve/kserve-controller-manager-599b79d756-r94fw"
Sep 16 18:49:47.515666 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:49:47.515647    2586 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 16 18:49:47.515761 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:49:47.515747    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aea89c8a-49af-4a43-a722-8097fb7aef34-cert podName:aea89c8a-49af-4a43-a722-8097fb7aef34 nodeName:}" failed. No retries permitted until 2026-09-16 18:49:48.015715842 +0000 UTC m=+563.459577410 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/aea89c8a-49af-4a43-a722-8097fb7aef34-cert") pod "kserve-controller-manager-599b79d756-r94fw" (UID: "aea89c8a-49af-4a43-a722-8097fb7aef34") : secret "kserve-webhook-server-cert" not found
Sep 16 18:49:47.515827 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:49:47.515762    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/12682bbb-5ad5-44a3-b405-f8840e7b2524-data\") pod \"seaweedfs-64568bcd49-xssm2\" (UID: \"12682bbb-5ad5-44a3-b405-f8840e7b2524\") " pod="kserve/seaweedfs-64568bcd49-xssm2"
Sep 16 18:49:47.527359 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:49:47.527334    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5k2k\" (UniqueName: \"kubernetes.io/projected/aea89c8a-49af-4a43-a722-8097fb7aef34-kube-api-access-t5k2k\") pod \"kserve-controller-manager-599b79d756-r94fw\" (UID: \"aea89c8a-49af-4a43-a722-8097fb7aef34\") " pod="kserve/kserve-controller-manager-599b79d756-r94fw"
Sep 16 18:49:47.528638 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:49:47.528613    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkv5t\" (UniqueName: \"kubernetes.io/projected/12682bbb-5ad5-44a3-b405-f8840e7b2524-kube-api-access-fkv5t\") pod \"seaweedfs-64568bcd49-xssm2\" (UID: \"12682bbb-5ad5-44a3-b405-f8840e7b2524\") " pod="kserve/seaweedfs-64568bcd49-xssm2"
Sep 16 18:49:47.671222 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:49:47.671186    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-xssm2"
Sep 16 18:49:47.946658 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:49:47.946635    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-xssm2"]
Sep 16 18:49:47.949369 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:49:47.949342    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12682bbb_5ad5_44a3_b405_f8840e7b2524.slice/crio-c9e82003b00f8db2311abc94a0c817713094475bc2171639422128ba41da924a WatchSource:0}: Error finding container c9e82003b00f8db2311abc94a0c817713094475bc2171639422128ba41da924a: Status 404 returned error can't find the container with id c9e82003b00f8db2311abc94a0c817713094475bc2171639422128ba41da924a
Sep 16 18:49:48.019640 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:49:48.019603    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aea89c8a-49af-4a43-a722-8097fb7aef34-cert\") pod \"kserve-controller-manager-599b79d756-r94fw\" (UID: \"aea89c8a-49af-4a43-a722-8097fb7aef34\") " pod="kserve/kserve-controller-manager-599b79d756-r94fw"
Sep 16 18:49:48.022116 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:49:48.022088    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aea89c8a-49af-4a43-a722-8097fb7aef34-cert\") pod \"kserve-controller-manager-599b79d756-r94fw\" (UID: \"aea89c8a-49af-4a43-a722-8097fb7aef34\") " pod="kserve/kserve-controller-manager-599b79d756-r94fw"
Sep 16 18:49:48.237105 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:49:48.237019    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-599b79d756-r94fw"
Sep 16 18:49:48.302813 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:49:48.302779    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-xssm2" event={"ID":"12682bbb-5ad5-44a3-b405-f8840e7b2524","Type":"ContainerStarted","Data":"c9e82003b00f8db2311abc94a0c817713094475bc2171639422128ba41da924a"}
Sep 16 18:49:48.514265 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:49:48.512332    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-599b79d756-r94fw"]
Sep 16 18:49:49.309627 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:49:49.309568    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-599b79d756-r94fw" event={"ID":"aea89c8a-49af-4a43-a722-8097fb7aef34","Type":"ContainerStarted","Data":"b37847a97b3e14dda2f3bf7deeb1763cbdc0d118542508e4cec50ee224421d16"}
Sep 16 18:49:52.324920 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:49:52.324882    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-xssm2" event={"ID":"12682bbb-5ad5-44a3-b405-f8840e7b2524","Type":"ContainerStarted","Data":"92beecd06d4f2b2a5f623fe7b7e093cd868e72fe340dd77b14c0410bee94665b"}
Sep 16 18:49:52.325343 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:49:52.325010    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-xssm2"
Sep 16 18:49:52.326465 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:49:52.326446    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-599b79d756-r94fw" event={"ID":"aea89c8a-49af-4a43-a722-8097fb7aef34","Type":"ContainerStarted","Data":"776ab85b845aa116fafbc43bea13564e778e2c8f5b445cff58170a2b79cb1474"}
Sep 16 18:49:52.326560 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:49:52.326543    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-599b79d756-r94fw"
Sep 16 18:49:52.344751 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:49:52.344703    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-xssm2" podStartSLOduration=1.130864421 podStartE2EDuration="5.344686991s" totalImagesPullingTime="4.21382257s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:52.343159248 +0000 UTC m=+567.787020825" watchObservedRunningTime="2026-09-16 18:49:52.344686991 +0000 UTC m=+567.788548567"
Sep 16 18:49:52.360505 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:49:52.360449    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-599b79d756-r94fw" podStartSLOduration=1.799871119 podStartE2EDuration="5.360436991s" totalImagesPullingTime="3.560565872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:52.359901261 +0000 UTC m=+567.803762837" watchObservedRunningTime="2026-09-16 18:49:52.360436991 +0000 UTC m=+567.804298567"
Sep 16 18:49:58.332415 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:49:58.332380    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-xssm2"
Sep 16 18:50:23.334901 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:50:23.334872    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-599b79d756-r94fw"
Sep 16 18:50:30.739162 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:50:30.739120    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-599b79d756-r94fw"]
Sep 16 18:50:30.739717 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:50:30.739346    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-599b79d756-r94fw" podUID="aea89c8a-49af-4a43-a722-8097fb7aef34" containerName="manager" containerID="cri-o://776ab85b845aa116fafbc43bea13564e778e2c8f5b445cff58170a2b79cb1474" gracePeriod=10
Sep 16 18:50:30.767924 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:50:30.767893    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-599b79d756-tcz7f"]
Sep 16 18:50:30.770147 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:50:30.770131    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-599b79d756-tcz7f"
Sep 16 18:50:30.782521 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:50:30.782494    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-599b79d756-tcz7f"]
Sep 16 18:50:30.859110 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:50:30.859082    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztpnl\" (UniqueName: \"kubernetes.io/projected/23b125e0-f1ef-49c6-8a71-beec63cdd445-kube-api-access-ztpnl\") pod \"kserve-controller-manager-599b79d756-tcz7f\" (UID: \"23b125e0-f1ef-49c6-8a71-beec63cdd445\") " pod="kserve/kserve-controller-manager-599b79d756-tcz7f"
Sep 16 18:50:30.859258 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:50:30.859183    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/23b125e0-f1ef-49c6-8a71-beec63cdd445-cert\") pod \"kserve-controller-manager-599b79d756-tcz7f\" (UID: \"23b125e0-f1ef-49c6-8a71-beec63cdd445\") " pod="kserve/kserve-controller-manager-599b79d756-tcz7f"
Sep 16 18:50:30.960208 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:50:30.960172    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/23b125e0-f1ef-49c6-8a71-beec63cdd445-cert\") pod \"kserve-controller-manager-599b79d756-tcz7f\" (UID: \"23b125e0-f1ef-49c6-8a71-beec63cdd445\") " pod="kserve/kserve-controller-manager-599b79d756-tcz7f"
Sep 16 18:50:30.960424 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:50:30.960249    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztpnl\" (UniqueName: \"kubernetes.io/projected/23b125e0-f1ef-49c6-8a71-beec63cdd445-kube-api-access-ztpnl\") pod \"kserve-controller-manager-599b79d756-tcz7f\" (UID: \"23b125e0-f1ef-49c6-8a71-beec63cdd445\") " pod="kserve/kserve-controller-manager-599b79d756-tcz7f"
Sep 16 18:50:30.962634 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:50:30.962610    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/23b125e0-f1ef-49c6-8a71-beec63cdd445-cert\") pod \"kserve-controller-manager-599b79d756-tcz7f\" (UID: \"23b125e0-f1ef-49c6-8a71-beec63cdd445\") " pod="kserve/kserve-controller-manager-599b79d756-tcz7f"
Sep 16 18:50:30.970698 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:50:30.970670    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztpnl\" (UniqueName: \"kubernetes.io/projected/23b125e0-f1ef-49c6-8a71-beec63cdd445-kube-api-access-ztpnl\") pod \"kserve-controller-manager-599b79d756-tcz7f\" (UID: \"23b125e0-f1ef-49c6-8a71-beec63cdd445\") " pod="kserve/kserve-controller-manager-599b79d756-tcz7f"
Sep 16 18:50:30.987264 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:50:30.987223    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-599b79d756-r94fw"
Sep 16 18:50:31.060731 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:50:31.060645    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aea89c8a-49af-4a43-a722-8097fb7aef34-cert\") pod \"aea89c8a-49af-4a43-a722-8097fb7aef34\" (UID: \"aea89c8a-49af-4a43-a722-8097fb7aef34\") "
Sep 16 18:50:31.060731 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:50:31.060727    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5k2k\" (UniqueName: \"kubernetes.io/projected/aea89c8a-49af-4a43-a722-8097fb7aef34-kube-api-access-t5k2k\") pod \"aea89c8a-49af-4a43-a722-8097fb7aef34\" (UID: \"aea89c8a-49af-4a43-a722-8097fb7aef34\") "
Sep 16 18:50:31.062795 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:50:31.062757    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aea89c8a-49af-4a43-a722-8097fb7aef34-kube-api-access-t5k2k" (OuterVolumeSpecName: "kube-api-access-t5k2k") pod "aea89c8a-49af-4a43-a722-8097fb7aef34" (UID: "aea89c8a-49af-4a43-a722-8097fb7aef34"). InnerVolumeSpecName "kube-api-access-t5k2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:50:31.062907 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:50:31.062830    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aea89c8a-49af-4a43-a722-8097fb7aef34-cert" (OuterVolumeSpecName: "cert") pod "aea89c8a-49af-4a43-a722-8097fb7aef34" (UID: "aea89c8a-49af-4a43-a722-8097fb7aef34"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:50:31.113016 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:50:31.112985    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-599b79d756-tcz7f"
Sep 16 18:50:31.162856 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:50:31.162825    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5k2k\" (UniqueName: \"kubernetes.io/projected/aea89c8a-49af-4a43-a722-8097fb7aef34-kube-api-access-t5k2k\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 18:50:31.163078 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:50:31.163030    2586 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aea89c8a-49af-4a43-a722-8097fb7aef34-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 18:50:31.382186 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:50:31.382073    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-599b79d756-tcz7f"]
Sep 16 18:50:31.384737 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:50:31.384715    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23b125e0_f1ef_49c6_8a71_beec63cdd445.slice/crio-3113bc14cc3b1067d54b4f8a59efbe24f2deb76e8b5c6ec0231ba1d9520e7f92 WatchSource:0}: Error finding container 3113bc14cc3b1067d54b4f8a59efbe24f2deb76e8b5c6ec0231ba1d9520e7f92: Status 404 returned error can't find the container with id 3113bc14cc3b1067d54b4f8a59efbe24f2deb76e8b5c6ec0231ba1d9520e7f92
Sep 16 18:50:31.489509 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:50:31.489472    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-599b79d756-tcz7f" event={"ID":"23b125e0-f1ef-49c6-8a71-beec63cdd445","Type":"ContainerStarted","Data":"3113bc14cc3b1067d54b4f8a59efbe24f2deb76e8b5c6ec0231ba1d9520e7f92"}
Sep 16 18:50:31.490708 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:50:31.490685    2586 generic.go:356] "Generic (PLEG): container finished" podID="aea89c8a-49af-4a43-a722-8097fb7aef34" containerID="776ab85b845aa116fafbc43bea13564e778e2c8f5b445cff58170a2b79cb1474" exitCode=0
Sep 16 18:50:31.490825 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:50:31.490726    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-599b79d756-r94fw"
Sep 16 18:50:31.490825 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:50:31.490763    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-599b79d756-r94fw" event={"ID":"aea89c8a-49af-4a43-a722-8097fb7aef34","Type":"ContainerDied","Data":"776ab85b845aa116fafbc43bea13564e778e2c8f5b445cff58170a2b79cb1474"}
Sep 16 18:50:31.490825 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:50:31.490809    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-599b79d756-r94fw" event={"ID":"aea89c8a-49af-4a43-a722-8097fb7aef34","Type":"ContainerDied","Data":"b37847a97b3e14dda2f3bf7deeb1763cbdc0d118542508e4cec50ee224421d16"}
Sep 16 18:50:31.490926 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:50:31.490832    2586 scope.go:117] "RemoveContainer" containerID="776ab85b845aa116fafbc43bea13564e778e2c8f5b445cff58170a2b79cb1474"
Sep 16 18:50:31.500284 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:50:31.500264    2586 scope.go:117] "RemoveContainer" containerID="776ab85b845aa116fafbc43bea13564e778e2c8f5b445cff58170a2b79cb1474"
Sep 16 18:50:31.500759 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:50:31.500737    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"776ab85b845aa116fafbc43bea13564e778e2c8f5b445cff58170a2b79cb1474\": container with ID starting with 776ab85b845aa116fafbc43bea13564e778e2c8f5b445cff58170a2b79cb1474 not found: ID does not exist" containerID="776ab85b845aa116fafbc43bea13564e778e2c8f5b445cff58170a2b79cb1474"
Sep 16 18:50:31.500831 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:50:31.500771    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"776ab85b845aa116fafbc43bea13564e778e2c8f5b445cff58170a2b79cb1474"} err="failed to get container status \"776ab85b845aa116fafbc43bea13564e778e2c8f5b445cff58170a2b79cb1474\": rpc error: code = NotFound desc = could not find container \"776ab85b845aa116fafbc43bea13564e778e2c8f5b445cff58170a2b79cb1474\": container with ID starting with 776ab85b845aa116fafbc43bea13564e778e2c8f5b445cff58170a2b79cb1474 not found: ID does not exist"
Sep 16 18:50:31.510788 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:50:31.510767    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-599b79d756-r94fw"]
Sep 16 18:50:31.515759 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:50:31.515740    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-599b79d756-r94fw"]
Sep 16 18:50:32.496479 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:50:32.496442    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-599b79d756-tcz7f" event={"ID":"23b125e0-f1ef-49c6-8a71-beec63cdd445","Type":"ContainerStarted","Data":"42a6af698d71146eb7f176fbd9ab0943f3d80a01306c467fd3addeb5b39c4b17"}
Sep 16 18:50:32.496947 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:50:32.496509    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-599b79d756-tcz7f"
Sep 16 18:50:32.520718 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:50:32.520663    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-599b79d756-tcz7f" podStartSLOduration=2.172455066 podStartE2EDuration="2.520647587s" totalImagesPullingTime="348.192521ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:50:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:50:32.519301317 +0000 UTC m=+607.963162894" watchObservedRunningTime="2026-09-16 18:50:32.520647587 +0000 UTC m=+607.964509185"
Sep 16 18:50:33.092277 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:50:33.092219    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aea89c8a-49af-4a43-a722-8097fb7aef34" path="/var/lib/kubelet/pods/aea89c8a-49af-4a43-a722-8097fb7aef34/volumes"
Sep 16 18:51:03.505920 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:51:03.505884    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-599b79d756-tcz7f"
Sep 16 18:51:04.874443 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:51:04.874411    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-psx8f"]
Sep 16 18:51:04.874788 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:51:04.874708    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aea89c8a-49af-4a43-a722-8097fb7aef34" containerName="manager"
Sep 16 18:51:04.874788 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:51:04.874720    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="aea89c8a-49af-4a43-a722-8097fb7aef34" containerName="manager"
Sep 16 18:51:04.874788 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:51:04.874771    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="aea89c8a-49af-4a43-a722-8097fb7aef34" containerName="manager"
Sep 16 18:51:04.877474 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:51:04.877454    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-psx8f"
Sep 16 18:51:04.880631 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:51:04.880608    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 16 18:51:04.880751 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:51:04.880699    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 16 18:51:04.880813 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:51:04.880635    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-mktgj\""
Sep 16 18:51:04.892269 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:51:04.892240    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-psx8f"]
Sep 16 18:51:05.039914 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:51:05.039876    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38f003c8-ea78-4260-8fc0-4dc2d1415936-metrics-certs\") pod \"odh-model-controller-7b966d7984-psx8f\" (UID: \"38f003c8-ea78-4260-8fc0-4dc2d1415936\") " pod="kserve/odh-model-controller-7b966d7984-psx8f"
Sep 16 18:51:05.040100 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:51:05.039928    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/38f003c8-ea78-4260-8fc0-4dc2d1415936-cert\") pod \"odh-model-controller-7b966d7984-psx8f\" (UID: \"38f003c8-ea78-4260-8fc0-4dc2d1415936\") " pod="kserve/odh-model-controller-7b966d7984-psx8f"
Sep 16 18:51:05.040100 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:51:05.039950    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27cb2\" (UniqueName: \"kubernetes.io/projected/38f003c8-ea78-4260-8fc0-4dc2d1415936-kube-api-access-27cb2\") pod \"odh-model-controller-7b966d7984-psx8f\" (UID: \"38f003c8-ea78-4260-8fc0-4dc2d1415936\") " pod="kserve/odh-model-controller-7b966d7984-psx8f"
Sep 16 18:51:05.140873 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:51:05.140793    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38f003c8-ea78-4260-8fc0-4dc2d1415936-metrics-certs\") pod \"odh-model-controller-7b966d7984-psx8f\" (UID: \"38f003c8-ea78-4260-8fc0-4dc2d1415936\") " pod="kserve/odh-model-controller-7b966d7984-psx8f"
Sep 16 18:51:05.140873 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:51:05.140841    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/38f003c8-ea78-4260-8fc0-4dc2d1415936-cert\") pod \"odh-model-controller-7b966d7984-psx8f\" (UID: \"38f003c8-ea78-4260-8fc0-4dc2d1415936\") " pod="kserve/odh-model-controller-7b966d7984-psx8f"
Sep 16 18:51:05.140873 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:51:05.140870    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27cb2\" (UniqueName: \"kubernetes.io/projected/38f003c8-ea78-4260-8fc0-4dc2d1415936-kube-api-access-27cb2\") pod \"odh-model-controller-7b966d7984-psx8f\" (UID: \"38f003c8-ea78-4260-8fc0-4dc2d1415936\") " pod="kserve/odh-model-controller-7b966d7984-psx8f"
Sep 16 18:51:05.143401 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:51:05.143373    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/38f003c8-ea78-4260-8fc0-4dc2d1415936-cert\") pod \"odh-model-controller-7b966d7984-psx8f\" (UID: \"38f003c8-ea78-4260-8fc0-4dc2d1415936\") " pod="kserve/odh-model-controller-7b966d7984-psx8f"
Sep 16 18:51:05.143515 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:51:05.143373    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38f003c8-ea78-4260-8fc0-4dc2d1415936-metrics-certs\") pod \"odh-model-controller-7b966d7984-psx8f\" (UID: \"38f003c8-ea78-4260-8fc0-4dc2d1415936\") " pod="kserve/odh-model-controller-7b966d7984-psx8f"
Sep 16 18:51:05.149588 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:51:05.149562    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27cb2\" (UniqueName: \"kubernetes.io/projected/38f003c8-ea78-4260-8fc0-4dc2d1415936-kube-api-access-27cb2\") pod \"odh-model-controller-7b966d7984-psx8f\" (UID: \"38f003c8-ea78-4260-8fc0-4dc2d1415936\") " pod="kserve/odh-model-controller-7b966d7984-psx8f"
Sep 16 18:51:05.188609 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:51:05.188573    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-psx8f"
Sep 16 18:51:05.461967 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:51:05.461845    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-psx8f"]
Sep 16 18:51:05.465821 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:51:05.465787    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38f003c8_ea78_4260_8fc0_4dc2d1415936.slice/crio-8fae5b458990e0c3f651e5cdef97367cc4b0e59600a7bace8ef2980c14360255 WatchSource:0}: Error finding container 8fae5b458990e0c3f651e5cdef97367cc4b0e59600a7bace8ef2980c14360255: Status 404 returned error can't find the container with id 8fae5b458990e0c3f651e5cdef97367cc4b0e59600a7bace8ef2980c14360255
Sep 16 18:51:05.637882 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:51:05.637849    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-psx8f" event={"ID":"38f003c8-ea78-4260-8fc0-4dc2d1415936","Type":"ContainerStarted","Data":"8fae5b458990e0c3f651e5cdef97367cc4b0e59600a7bace8ef2980c14360255"}
Sep 16 18:51:08.655537 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:51:08.655498    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-psx8f" event={"ID":"38f003c8-ea78-4260-8fc0-4dc2d1415936","Type":"ContainerStarted","Data":"dfb71f3dd7b9f9eb83ab2a2ed954711854f6a1fcbe3f4acb3bf81b35c38a567e"}
Sep 16 18:51:08.656053 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:51:08.655643    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-psx8f"
Sep 16 18:51:08.675492 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:51:08.675436    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-psx8f" podStartSLOduration=2.036354073 podStartE2EDuration="4.675420087s" totalImagesPullingTime="2.639066014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:51:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:51:08.673762352 +0000 UTC m=+644.117623942" watchObservedRunningTime="2026-09-16 18:51:08.675420087 +0000 UTC m=+644.119281662"
Sep 16 18:51:19.663180 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:51:19.663144    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-psx8f"
Sep 16 18:52:39.311184 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:39.311095    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-cwwq7"]
Sep 16 18:52:39.314534 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:39.314512    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-cwwq7"
Sep 16 18:52:39.319249 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:39.319213    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 16 18:52:39.325220 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:39.325197    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-cwwq7"]
Sep 16 18:52:39.431533 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:39.431504    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkqj7\" (UniqueName: \"kubernetes.io/projected/5853d455-3e49-4519-9864-de53ac3f9900-kube-api-access-nkqj7\") pod \"seaweedfs-tls-custom-6644bf5d6c-cwwq7\" (UID: \"5853d455-3e49-4519-9864-de53ac3f9900\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-cwwq7"
Sep 16 18:52:39.431533 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:39.431539    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5853d455-3e49-4519-9864-de53ac3f9900-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-cwwq7\" (UID: \"5853d455-3e49-4519-9864-de53ac3f9900\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-cwwq7"
Sep 16 18:52:39.532629 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:39.532592    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkqj7\" (UniqueName: \"kubernetes.io/projected/5853d455-3e49-4519-9864-de53ac3f9900-kube-api-access-nkqj7\") pod \"seaweedfs-tls-custom-6644bf5d6c-cwwq7\" (UID: \"5853d455-3e49-4519-9864-de53ac3f9900\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-cwwq7"
Sep 16 18:52:39.532785 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:39.532634    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5853d455-3e49-4519-9864-de53ac3f9900-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-cwwq7\" (UID: \"5853d455-3e49-4519-9864-de53ac3f9900\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-cwwq7"
Sep 16 18:52:39.533044 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:39.533023    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5853d455-3e49-4519-9864-de53ac3f9900-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-cwwq7\" (UID: \"5853d455-3e49-4519-9864-de53ac3f9900\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-cwwq7"
Sep 16 18:52:39.542989 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:39.542968    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkqj7\" (UniqueName: \"kubernetes.io/projected/5853d455-3e49-4519-9864-de53ac3f9900-kube-api-access-nkqj7\") pod \"seaweedfs-tls-custom-6644bf5d6c-cwwq7\" (UID: \"5853d455-3e49-4519-9864-de53ac3f9900\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-cwwq7"
Sep 16 18:52:39.623928 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:39.623901    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-cwwq7"
Sep 16 18:52:39.894102 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:39.894076    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-cwwq7"]
Sep 16 18:52:39.896922 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:52:39.896884    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5853d455_3e49_4519_9864_de53ac3f9900.slice/crio-8a9193acbaaf27911624a1804ecbae5e5198e91d018f7d1eb47c9e48a81d2345 WatchSource:0}: Error finding container 8a9193acbaaf27911624a1804ecbae5e5198e91d018f7d1eb47c9e48a81d2345: Status 404 returned error can't find the container with id 8a9193acbaaf27911624a1804ecbae5e5198e91d018f7d1eb47c9e48a81d2345
Sep 16 18:52:40.045358 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:40.045324    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-cwwq7" event={"ID":"5853d455-3e49-4519-9864-de53ac3f9900","Type":"ContainerStarted","Data":"8a9193acbaaf27911624a1804ecbae5e5198e91d018f7d1eb47c9e48a81d2345"}
Sep 16 18:52:41.051497 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:41.051462    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-cwwq7" event={"ID":"5853d455-3e49-4519-9864-de53ac3f9900","Type":"ContainerStarted","Data":"89a4bd25d867a3657e0acd3bfc85928f024df5a4029c8b9c8f9977bb67a780ee"}
Sep 16 18:52:41.070321 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:41.070275    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-cwwq7" podStartSLOduration=1.753134448 podStartE2EDuration="2.070260705s" totalImagesPullingTime="317.126257ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:52:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:52:41.069524687 +0000 UTC m=+736.513386264" watchObservedRunningTime="2026-09-16 18:52:41.070260705 +0000 UTC m=+736.514122280"
Sep 16 18:52:42.442138 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:42.442102    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-cwwq7"]
Sep 16 18:52:43.063410 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:43.063366    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-cwwq7" podUID="5853d455-3e49-4519-9864-de53ac3f9900" containerName="seaweedfs-tls-custom" containerID="cri-o://89a4bd25d867a3657e0acd3bfc85928f024df5a4029c8b9c8f9977bb67a780ee" gracePeriod=30
Sep 16 18:52:44.304320 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:44.304295    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-cwwq7"
Sep 16 18:52:44.374219 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:44.374185    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5853d455-3e49-4519-9864-de53ac3f9900-data\") pod \"5853d455-3e49-4519-9864-de53ac3f9900\" (UID: \"5853d455-3e49-4519-9864-de53ac3f9900\") "
Sep 16 18:52:44.374411 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:44.374312    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nkqj7\" (UniqueName: \"kubernetes.io/projected/5853d455-3e49-4519-9864-de53ac3f9900-kube-api-access-nkqj7\") pod \"5853d455-3e49-4519-9864-de53ac3f9900\" (UID: \"5853d455-3e49-4519-9864-de53ac3f9900\") "
Sep 16 18:52:44.375453 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:44.375425    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5853d455-3e49-4519-9864-de53ac3f9900-data" (OuterVolumeSpecName: "data") pod "5853d455-3e49-4519-9864-de53ac3f9900" (UID: "5853d455-3e49-4519-9864-de53ac3f9900"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:52:44.376405 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:44.376372    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5853d455-3e49-4519-9864-de53ac3f9900-kube-api-access-nkqj7" (OuterVolumeSpecName: "kube-api-access-nkqj7") pod "5853d455-3e49-4519-9864-de53ac3f9900" (UID: "5853d455-3e49-4519-9864-de53ac3f9900"). InnerVolumeSpecName "kube-api-access-nkqj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:52:44.475193 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:44.475158    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkqj7\" (UniqueName: \"kubernetes.io/projected/5853d455-3e49-4519-9864-de53ac3f9900-kube-api-access-nkqj7\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 18:52:44.475193 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:44.475190    2586 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5853d455-3e49-4519-9864-de53ac3f9900-data\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 18:52:45.072541 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:45.072506    2586 generic.go:356] "Generic (PLEG): container finished" podID="5853d455-3e49-4519-9864-de53ac3f9900" containerID="89a4bd25d867a3657e0acd3bfc85928f024df5a4029c8b9c8f9977bb67a780ee" exitCode=0
Sep 16 18:52:45.072749 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:45.072547    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-cwwq7"
Sep 16 18:52:45.072749 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:45.072584    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-cwwq7" event={"ID":"5853d455-3e49-4519-9864-de53ac3f9900","Type":"ContainerDied","Data":"89a4bd25d867a3657e0acd3bfc85928f024df5a4029c8b9c8f9977bb67a780ee"}
Sep 16 18:52:45.072749 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:45.072664    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-cwwq7" event={"ID":"5853d455-3e49-4519-9864-de53ac3f9900","Type":"ContainerDied","Data":"8a9193acbaaf27911624a1804ecbae5e5198e91d018f7d1eb47c9e48a81d2345"}
Sep 16 18:52:45.072749 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:45.072695    2586 scope.go:117] "RemoveContainer" containerID="89a4bd25d867a3657e0acd3bfc85928f024df5a4029c8b9c8f9977bb67a780ee"
Sep 16 18:52:45.082468 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:45.082448    2586 scope.go:117] "RemoveContainer" containerID="89a4bd25d867a3657e0acd3bfc85928f024df5a4029c8b9c8f9977bb67a780ee"
Sep 16 18:52:45.082753 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:52:45.082734    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89a4bd25d867a3657e0acd3bfc85928f024df5a4029c8b9c8f9977bb67a780ee\": container with ID starting with 89a4bd25d867a3657e0acd3bfc85928f024df5a4029c8b9c8f9977bb67a780ee not found: ID does not exist" containerID="89a4bd25d867a3657e0acd3bfc85928f024df5a4029c8b9c8f9977bb67a780ee"
Sep 16 18:52:45.082804 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:45.082762    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89a4bd25d867a3657e0acd3bfc85928f024df5a4029c8b9c8f9977bb67a780ee"} err="failed to get container status \"89a4bd25d867a3657e0acd3bfc85928f024df5a4029c8b9c8f9977bb67a780ee\": rpc error: code = NotFound desc = could not find container \"89a4bd25d867a3657e0acd3bfc85928f024df5a4029c8b9c8f9977bb67a780ee\": container with ID starting with 89a4bd25d867a3657e0acd3bfc85928f024df5a4029c8b9c8f9977bb67a780ee not found: ID does not exist"
Sep 16 18:52:45.099531 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:45.099505    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-cwwq7"]
Sep 16 18:52:45.109474 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:45.109449    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-cwwq7"]
Sep 16 18:52:45.154747 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:45.154714    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-4szz7"]
Sep 16 18:52:45.155033 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:45.155021    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5853d455-3e49-4519-9864-de53ac3f9900" containerName="seaweedfs-tls-custom"
Sep 16 18:52:45.155075 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:45.155034    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="5853d455-3e49-4519-9864-de53ac3f9900" containerName="seaweedfs-tls-custom"
Sep 16 18:52:45.155113 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:45.155084    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="5853d455-3e49-4519-9864-de53ac3f9900" containerName="seaweedfs-tls-custom"
Sep 16 18:52:45.159424 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:45.159404    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-4szz7"
Sep 16 18:52:45.161776 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:45.161750    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 16 18:52:45.161893 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:45.161883    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom\""
Sep 16 18:52:45.172013 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:45.171990    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-4szz7"]
Sep 16 18:52:45.281765 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:45.281724    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5g85\" (UniqueName: \"kubernetes.io/projected/4ff37a2e-7be7-4578-9b29-949089c4f8a2-kube-api-access-w5g85\") pod \"seaweedfs-tls-custom-8b44d767d-4szz7\" (UID: \"4ff37a2e-7be7-4578-9b29-949089c4f8a2\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-4szz7"
Sep 16 18:52:45.281950 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:45.281794    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4ff37a2e-7be7-4578-9b29-949089c4f8a2-data\") pod \"seaweedfs-tls-custom-8b44d767d-4szz7\" (UID: \"4ff37a2e-7be7-4578-9b29-949089c4f8a2\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-4szz7"
Sep 16 18:52:45.281950 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:45.281850    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/4ff37a2e-7be7-4578-9b29-949089c4f8a2-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-4szz7\" (UID: \"4ff37a2e-7be7-4578-9b29-949089c4f8a2\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-4szz7"
Sep 16 18:52:45.382965 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:45.382937    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/4ff37a2e-7be7-4578-9b29-949089c4f8a2-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-4szz7\" (UID: \"4ff37a2e-7be7-4578-9b29-949089c4f8a2\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-4szz7"
Sep 16 18:52:45.383363 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:45.382977    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5g85\" (UniqueName: \"kubernetes.io/projected/4ff37a2e-7be7-4578-9b29-949089c4f8a2-kube-api-access-w5g85\") pod \"seaweedfs-tls-custom-8b44d767d-4szz7\" (UID: \"4ff37a2e-7be7-4578-9b29-949089c4f8a2\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-4szz7"
Sep 16 18:52:45.383363 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:45.383022    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4ff37a2e-7be7-4578-9b29-949089c4f8a2-data\") pod \"seaweedfs-tls-custom-8b44d767d-4szz7\" (UID: \"4ff37a2e-7be7-4578-9b29-949089c4f8a2\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-4szz7"
Sep 16 18:52:45.383494 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:45.383455    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4ff37a2e-7be7-4578-9b29-949089c4f8a2-data\") pod \"seaweedfs-tls-custom-8b44d767d-4szz7\" (UID: \"4ff37a2e-7be7-4578-9b29-949089c4f8a2\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-4szz7"
Sep 16 18:52:45.385453 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:45.385425    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/4ff37a2e-7be7-4578-9b29-949089c4f8a2-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-4szz7\" (UID: \"4ff37a2e-7be7-4578-9b29-949089c4f8a2\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-4szz7"
Sep 16 18:52:45.395109 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:45.395087    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5g85\" (UniqueName: \"kubernetes.io/projected/4ff37a2e-7be7-4578-9b29-949089c4f8a2-kube-api-access-w5g85\") pod \"seaweedfs-tls-custom-8b44d767d-4szz7\" (UID: \"4ff37a2e-7be7-4578-9b29-949089c4f8a2\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-4szz7"
Sep 16 18:52:45.468162 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:45.468128    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-4szz7"
Sep 16 18:52:45.740902 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:45.740819    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-4szz7"]
Sep 16 18:52:45.744346 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:52:45.744317    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ff37a2e_7be7_4578_9b29_949089c4f8a2.slice/crio-c8fb8973edbf2d324ec551f5f6305cbac3dc755b539d50594773ccfdf2558cfc WatchSource:0}: Error finding container c8fb8973edbf2d324ec551f5f6305cbac3dc755b539d50594773ccfdf2558cfc: Status 404 returned error can't find the container with id c8fb8973edbf2d324ec551f5f6305cbac3dc755b539d50594773ccfdf2558cfc
Sep 16 18:52:46.079452 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:46.079418    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-4szz7" event={"ID":"4ff37a2e-7be7-4578-9b29-949089c4f8a2","Type":"ContainerStarted","Data":"166a27eb76e9bbae3eb8ad4afc9423f93f55bbbc7df205bc28894062afb40475"}
Sep 16 18:52:46.079621 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:46.079470    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-4szz7" event={"ID":"4ff37a2e-7be7-4578-9b29-949089c4f8a2","Type":"ContainerStarted","Data":"c8fb8973edbf2d324ec551f5f6305cbac3dc755b539d50594773ccfdf2558cfc"}
Sep 16 18:52:46.102871 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:46.102818    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-8b44d767d-4szz7" podStartSLOduration=0.858523271 podStartE2EDuration="1.102800083s" totalImagesPullingTime="244.276812ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:52:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:52:46.102121083 +0000 UTC m=+741.545982660" watchObservedRunningTime="2026-09-16 18:52:46.102800083 +0000 UTC m=+741.546661658"
Sep 16 18:52:47.092080 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:47.092045    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5853d455-3e49-4519-9864-de53ac3f9900" path="/var/lib/kubelet/pods/5853d455-3e49-4519-9864-de53ac3f9900/volumes"
Sep 16 18:52:58.287546 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:58.287514    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-lbdvm"]
Sep 16 18:52:58.290948 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:58.290928    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-lbdvm"
Sep 16 18:52:58.293041 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:58.293008    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving\""
Sep 16 18:52:58.293173 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:58.293153    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Sep 16 18:52:58.313204 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:58.313178    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-lbdvm"]
Sep 16 18:52:58.393585 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:58.393547    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/40aa28e7-7214-41ef-8483-14a7429bc0b9-data\") pod \"seaweedfs-tls-serving-67b7477c6d-lbdvm\" (UID: \"40aa28e7-7214-41ef-8483-14a7429bc0b9\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-lbdvm"
Sep 16 18:52:58.393749 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:58.393676    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/40aa28e7-7214-41ef-8483-14a7429bc0b9-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-lbdvm\" (UID: \"40aa28e7-7214-41ef-8483-14a7429bc0b9\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-lbdvm"
Sep 16 18:52:58.393749 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:58.393719    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnp85\" (UniqueName: \"kubernetes.io/projected/40aa28e7-7214-41ef-8483-14a7429bc0b9-kube-api-access-pnp85\") pod \"seaweedfs-tls-serving-67b7477c6d-lbdvm\" (UID: \"40aa28e7-7214-41ef-8483-14a7429bc0b9\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-lbdvm"
Sep 16 18:52:58.495086 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:58.495056    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/40aa28e7-7214-41ef-8483-14a7429bc0b9-data\") pod \"seaweedfs-tls-serving-67b7477c6d-lbdvm\" (UID: \"40aa28e7-7214-41ef-8483-14a7429bc0b9\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-lbdvm"
Sep 16 18:52:58.495272 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:58.495107    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/40aa28e7-7214-41ef-8483-14a7429bc0b9-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-lbdvm\" (UID: \"40aa28e7-7214-41ef-8483-14a7429bc0b9\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-lbdvm"
Sep 16 18:52:58.495272 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:58.495131    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnp85\" (UniqueName: \"kubernetes.io/projected/40aa28e7-7214-41ef-8483-14a7429bc0b9-kube-api-access-pnp85\") pod \"seaweedfs-tls-serving-67b7477c6d-lbdvm\" (UID: \"40aa28e7-7214-41ef-8483-14a7429bc0b9\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-lbdvm"
Sep 16 18:52:58.495524 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:58.495501    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/40aa28e7-7214-41ef-8483-14a7429bc0b9-data\") pod \"seaweedfs-tls-serving-67b7477c6d-lbdvm\" (UID: \"40aa28e7-7214-41ef-8483-14a7429bc0b9\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-lbdvm"
Sep 16 18:52:58.497493 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:58.497474    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/40aa28e7-7214-41ef-8483-14a7429bc0b9-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-lbdvm\" (UID: \"40aa28e7-7214-41ef-8483-14a7429bc0b9\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-lbdvm"
Sep 16 18:52:58.505673 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:58.505650    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnp85\" (UniqueName: \"kubernetes.io/projected/40aa28e7-7214-41ef-8483-14a7429bc0b9-kube-api-access-pnp85\") pod \"seaweedfs-tls-serving-67b7477c6d-lbdvm\" (UID: \"40aa28e7-7214-41ef-8483-14a7429bc0b9\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-lbdvm"
Sep 16 18:52:58.600280 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:58.600249    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-lbdvm"
Sep 16 18:52:58.871148 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:58.871108    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-lbdvm"]
Sep 16 18:52:59.135465 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:52:59.135427    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-lbdvm" event={"ID":"40aa28e7-7214-41ef-8483-14a7429bc0b9","Type":"ContainerStarted","Data":"78e81d18f05852016e8782a02e08c9046ea7423b05f1a32fbd76d4bf3545d6f8"}
Sep 16 18:53:00.146545 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:00.146501    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-lbdvm" event={"ID":"40aa28e7-7214-41ef-8483-14a7429bc0b9","Type":"ContainerStarted","Data":"3e17d4e13d352de4eb7f0468297051f1b07071bf532c716e10dc29bd2836f883"}
Sep 16 18:53:00.175458 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:00.175410    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-serving-67b7477c6d-lbdvm" podStartSLOduration=1.91965991 podStartE2EDuration="2.175395743s" totalImagesPullingTime="255.735833ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:52:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:53:00.175162365 +0000 UTC m=+755.619023952" watchObservedRunningTime="2026-09-16 18:53:00.175395743 +0000 UTC m=+755.619257363"
Sep 16 18:53:24.211224 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:24.211183    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-28115f-predictor-57bb46966f-jx4sx"]
Sep 16 18:53:24.216670 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:24.216643    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-28115f-predictor-57bb46966f-jx4sx"
Sep 16 18:53:24.219143 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:24.219123    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-28115f-predictor-serving-cert\""
Sep 16 18:53:24.219558 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:24.219535    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 18:53:24.219835 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:24.219813    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-28115f-kube-rbac-proxy-sar-config\""
Sep 16 18:53:24.220136 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:24.220117    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 18:53:24.220136 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:24.220128    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-28115f-dockercfg-fvp7n\""
Sep 16 18:53:24.220314 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:24.220129    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:53:24.220314 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:24.220132    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-28115f\""
Sep 16 18:53:24.226735 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:24.226707    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-28115f-predictor-57bb46966f-jx4sx"]
Sep 16 18:53:24.313295 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:24.313257    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae207379-aa2c-4b5e-b959-b4562ec3623e-proxy-tls\") pod \"isvc-init-fail-28115f-predictor-57bb46966f-jx4sx\" (UID: \"ae207379-aa2c-4b5e-b959-b4562ec3623e\") " pod="kserve-ci-e2e-test/isvc-init-fail-28115f-predictor-57bb46966f-jx4sx"
Sep 16 18:53:24.313473 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:24.313308    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjzpq\" (UniqueName: \"kubernetes.io/projected/ae207379-aa2c-4b5e-b959-b4562ec3623e-kube-api-access-wjzpq\") pod \"isvc-init-fail-28115f-predictor-57bb46966f-jx4sx\" (UID: \"ae207379-aa2c-4b5e-b959-b4562ec3623e\") " pod="kserve-ci-e2e-test/isvc-init-fail-28115f-predictor-57bb46966f-jx4sx"
Sep 16 18:53:24.313473 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:24.313392    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae207379-aa2c-4b5e-b959-b4562ec3623e-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-28115f-predictor-57bb46966f-jx4sx\" (UID: \"ae207379-aa2c-4b5e-b959-b4562ec3623e\") " pod="kserve-ci-e2e-test/isvc-init-fail-28115f-predictor-57bb46966f-jx4sx"
Sep 16 18:53:24.313473 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:24.313420    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae207379-aa2c-4b5e-b959-b4562ec3623e-kserve-provision-location\") pod \"isvc-init-fail-28115f-predictor-57bb46966f-jx4sx\" (UID: \"ae207379-aa2c-4b5e-b959-b4562ec3623e\") " pod="kserve-ci-e2e-test/isvc-init-fail-28115f-predictor-57bb46966f-jx4sx"
Sep 16 18:53:24.313473 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:24.313456    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae207379-aa2c-4b5e-b959-b4562ec3623e-cabundle-cert\") pod \"isvc-init-fail-28115f-predictor-57bb46966f-jx4sx\" (UID: \"ae207379-aa2c-4b5e-b959-b4562ec3623e\") " pod="kserve-ci-e2e-test/isvc-init-fail-28115f-predictor-57bb46966f-jx4sx"
Sep 16 18:53:24.413892 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:24.413860    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae207379-aa2c-4b5e-b959-b4562ec3623e-proxy-tls\") pod \"isvc-init-fail-28115f-predictor-57bb46966f-jx4sx\" (UID: \"ae207379-aa2c-4b5e-b959-b4562ec3623e\") " pod="kserve-ci-e2e-test/isvc-init-fail-28115f-predictor-57bb46966f-jx4sx"
Sep 16 18:53:24.413892 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:24.413894    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjzpq\" (UniqueName: \"kubernetes.io/projected/ae207379-aa2c-4b5e-b959-b4562ec3623e-kube-api-access-wjzpq\") pod \"isvc-init-fail-28115f-predictor-57bb46966f-jx4sx\" (UID: \"ae207379-aa2c-4b5e-b959-b4562ec3623e\") " pod="kserve-ci-e2e-test/isvc-init-fail-28115f-predictor-57bb46966f-jx4sx"
Sep 16 18:53:24.414151 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:53:24.414025    2586 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-28115f-predictor-serving-cert: secret "isvc-init-fail-28115f-predictor-serving-cert" not found
Sep 16 18:53:24.414151 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:53:24.414099    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae207379-aa2c-4b5e-b959-b4562ec3623e-proxy-tls podName:ae207379-aa2c-4b5e-b959-b4562ec3623e nodeName:}" failed. No retries permitted until 2026-09-16 18:53:24.91407754 +0000 UTC m=+780.357939098 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ae207379-aa2c-4b5e-b959-b4562ec3623e-proxy-tls") pod "isvc-init-fail-28115f-predictor-57bb46966f-jx4sx" (UID: "ae207379-aa2c-4b5e-b959-b4562ec3623e") : secret "isvc-init-fail-28115f-predictor-serving-cert" not found
Sep 16 18:53:24.414151 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:24.414026    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae207379-aa2c-4b5e-b959-b4562ec3623e-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-28115f-predictor-57bb46966f-jx4sx\" (UID: \"ae207379-aa2c-4b5e-b959-b4562ec3623e\") " pod="kserve-ci-e2e-test/isvc-init-fail-28115f-predictor-57bb46966f-jx4sx"
Sep 16 18:53:24.414151 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:24.414137    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae207379-aa2c-4b5e-b959-b4562ec3623e-kserve-provision-location\") pod \"isvc-init-fail-28115f-predictor-57bb46966f-jx4sx\" (UID: \"ae207379-aa2c-4b5e-b959-b4562ec3623e\") " pod="kserve-ci-e2e-test/isvc-init-fail-28115f-predictor-57bb46966f-jx4sx"
Sep 16 18:53:24.414356 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:24.414183    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae207379-aa2c-4b5e-b959-b4562ec3623e-cabundle-cert\") pod \"isvc-init-fail-28115f-predictor-57bb46966f-jx4sx\" (UID: \"ae207379-aa2c-4b5e-b959-b4562ec3623e\") " pod="kserve-ci-e2e-test/isvc-init-fail-28115f-predictor-57bb46966f-jx4sx"
Sep 16 18:53:24.414598 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:24.414577    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae207379-aa2c-4b5e-b959-b4562ec3623e-kserve-provision-location\") pod \"isvc-init-fail-28115f-predictor-57bb46966f-jx4sx\" (UID: \"ae207379-aa2c-4b5e-b959-b4562ec3623e\") " pod="kserve-ci-e2e-test/isvc-init-fail-28115f-predictor-57bb46966f-jx4sx"
Sep 16 18:53:24.414737 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:24.414719    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae207379-aa2c-4b5e-b959-b4562ec3623e-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-28115f-predictor-57bb46966f-jx4sx\" (UID: \"ae207379-aa2c-4b5e-b959-b4562ec3623e\") " pod="kserve-ci-e2e-test/isvc-init-fail-28115f-predictor-57bb46966f-jx4sx"
Sep 16 18:53:24.414821 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:24.414803    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae207379-aa2c-4b5e-b959-b4562ec3623e-cabundle-cert\") pod \"isvc-init-fail-28115f-predictor-57bb46966f-jx4sx\" (UID: \"ae207379-aa2c-4b5e-b959-b4562ec3623e\") " pod="kserve-ci-e2e-test/isvc-init-fail-28115f-predictor-57bb46966f-jx4sx"
Sep 16 18:53:24.425169 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:24.425150    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjzpq\" (UniqueName: \"kubernetes.io/projected/ae207379-aa2c-4b5e-b959-b4562ec3623e-kube-api-access-wjzpq\") pod \"isvc-init-fail-28115f-predictor-57bb46966f-jx4sx\" (UID: \"ae207379-aa2c-4b5e-b959-b4562ec3623e\") " pod="kserve-ci-e2e-test/isvc-init-fail-28115f-predictor-57bb46966f-jx4sx"
Sep 16 18:53:24.920081 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:24.920045    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae207379-aa2c-4b5e-b959-b4562ec3623e-proxy-tls\") pod \"isvc-init-fail-28115f-predictor-57bb46966f-jx4sx\" (UID: \"ae207379-aa2c-4b5e-b959-b4562ec3623e\") " pod="kserve-ci-e2e-test/isvc-init-fail-28115f-predictor-57bb46966f-jx4sx"
Sep 16 18:53:24.922555 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:24.922531    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae207379-aa2c-4b5e-b959-b4562ec3623e-proxy-tls\") pod \"isvc-init-fail-28115f-predictor-57bb46966f-jx4sx\" (UID: \"ae207379-aa2c-4b5e-b959-b4562ec3623e\") " pod="kserve-ci-e2e-test/isvc-init-fail-28115f-predictor-57bb46966f-jx4sx"
Sep 16 18:53:25.131585 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:25.131556    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-28115f-dockercfg-fvp7n\""
Sep 16 18:53:25.139482 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:25.139458    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-28115f-predictor-57bb46966f-jx4sx"
Sep 16 18:53:25.417619 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:25.417593    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-28115f-predictor-57bb46966f-jx4sx"]
Sep 16 18:53:25.420030 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:53:25.419992    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae207379_aa2c_4b5e_b959_b4562ec3623e.slice/crio-b0e2db3f1b4c10fee39b70001c06507a5e3174e41e37a758096455c14e462359 WatchSource:0}: Error finding container b0e2db3f1b4c10fee39b70001c06507a5e3174e41e37a758096455c14e462359: Status 404 returned error can't find the container with id b0e2db3f1b4c10fee39b70001c06507a5e3174e41e37a758096455c14e462359
Sep 16 18:53:25.421997 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:25.421977    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:53:26.271564 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:26.271529    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-28115f-predictor-57bb46966f-jx4sx" event={"ID":"ae207379-aa2c-4b5e-b959-b4562ec3623e","Type":"ContainerStarted","Data":"b0e2db3f1b4c10fee39b70001c06507a5e3174e41e37a758096455c14e462359"}
Sep 16 18:53:28.749028 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:28.749002    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-28115f\""
Sep 16 18:53:29.291307 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:29.291264    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-28115f-predictor-57bb46966f-jx4sx" event={"ID":"ae207379-aa2c-4b5e-b959-b4562ec3623e","Type":"ContainerStarted","Data":"cb8d2ba13f68e1580573a065a42b9b2b5edd18e2b7d29770c04d4efa69f0dea4"}
Sep 16 18:53:35.319482 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:35.319451    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-28115f-predictor-57bb46966f-jx4sx_ae207379-aa2c-4b5e-b959-b4562ec3623e/storage-initializer/0.log"
Sep 16 18:53:35.319847 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:35.319492    2586 generic.go:356] "Generic (PLEG): container finished" podID="ae207379-aa2c-4b5e-b959-b4562ec3623e" containerID="cb8d2ba13f68e1580573a065a42b9b2b5edd18e2b7d29770c04d4efa69f0dea4" exitCode=1
Sep 16 18:53:35.319847 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:35.319525    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-28115f-predictor-57bb46966f-jx4sx" event={"ID":"ae207379-aa2c-4b5e-b959-b4562ec3623e","Type":"ContainerDied","Data":"cb8d2ba13f68e1580573a065a42b9b2b5edd18e2b7d29770c04d4efa69f0dea4"}
Sep 16 18:53:36.325622 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:36.325594    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-28115f-predictor-57bb46966f-jx4sx_ae207379-aa2c-4b5e-b959-b4562ec3623e/storage-initializer/0.log"
Sep 16 18:53:36.325979 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:36.325670    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-28115f-predictor-57bb46966f-jx4sx" event={"ID":"ae207379-aa2c-4b5e-b959-b4562ec3623e","Type":"ContainerStarted","Data":"8749b2fb0ad733d5d419027855a226fc97ed06c45a1784c057359c8d82ec5c25"}
Sep 16 18:53:38.336472 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:38.336444    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-28115f-predictor-57bb46966f-jx4sx_ae207379-aa2c-4b5e-b959-b4562ec3623e/storage-initializer/1.log"
Sep 16 18:53:38.336962 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:38.336940    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-28115f-predictor-57bb46966f-jx4sx_ae207379-aa2c-4b5e-b959-b4562ec3623e/storage-initializer/0.log"
Sep 16 18:53:38.337009 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:38.336986    2586 generic.go:356] "Generic (PLEG): container finished" podID="ae207379-aa2c-4b5e-b959-b4562ec3623e" containerID="8749b2fb0ad733d5d419027855a226fc97ed06c45a1784c057359c8d82ec5c25" exitCode=1
Sep 16 18:53:38.337082 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:38.337062    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-28115f-predictor-57bb46966f-jx4sx" event={"ID":"ae207379-aa2c-4b5e-b959-b4562ec3623e","Type":"ContainerDied","Data":"8749b2fb0ad733d5d419027855a226fc97ed06c45a1784c057359c8d82ec5c25"}
Sep 16 18:53:38.337138 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:38.337126    2586 scope.go:117] "RemoveContainer" containerID="cb8d2ba13f68e1580573a065a42b9b2b5edd18e2b7d29770c04d4efa69f0dea4"
Sep 16 18:53:38.337518 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:38.337502    2586 scope.go:117] "RemoveContainer" containerID="cb8d2ba13f68e1580573a065a42b9b2b5edd18e2b7d29770c04d4efa69f0dea4"
Sep 16 18:53:38.348138 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:53:38.348108    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-init-fail-28115f-predictor-57bb46966f-jx4sx_kserve-ci-e2e-test(ae207379-aa2c-4b5e-b959-b4562ec3623e)\"" pod="kserve-ci-e2e-test/isvc-init-fail-28115f-predictor-57bb46966f-jx4sx" podUID="ae207379-aa2c-4b5e-b959-b4562ec3623e"
Sep 16 18:53:38.348369 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:53:38.348343    2586 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-init-fail-28115f-predictor-57bb46966f-jx4sx_kserve-ci-e2e-test_ae207379-aa2c-4b5e-b959-b4562ec3623e_0 in pod sandbox b0e2db3f1b4c10fee39b70001c06507a5e3174e41e37a758096455c14e462359 from index: no such id: 'cb8d2ba13f68e1580573a065a42b9b2b5edd18e2b7d29770c04d4efa69f0dea4'" containerID="cb8d2ba13f68e1580573a065a42b9b2b5edd18e2b7d29770c04d4efa69f0dea4"
Sep 16 18:53:38.348438 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:38.348379    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb8d2ba13f68e1580573a065a42b9b2b5edd18e2b7d29770c04d4efa69f0dea4"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-init-fail-28115f-predictor-57bb46966f-jx4sx_kserve-ci-e2e-test_ae207379-aa2c-4b5e-b959-b4562ec3623e_0 in pod sandbox b0e2db3f1b4c10fee39b70001c06507a5e3174e41e37a758096455c14e462359 from index: no such id: 'cb8d2ba13f68e1580573a065a42b9b2b5edd18e2b7d29770c04d4efa69f0dea4'"
Sep 16 18:53:38.869567 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:38.869523    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-28115f-predictor-57bb46966f-jx4sx"]
Sep 16 18:53:39.343697 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:39.343619    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-28115f-predictor-57bb46966f-jx4sx_ae207379-aa2c-4b5e-b959-b4562ec3623e/storage-initializer/1.log"
Sep 16 18:53:39.479993 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:39.479971    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-28115f-predictor-57bb46966f-jx4sx_ae207379-aa2c-4b5e-b959-b4562ec3623e/storage-initializer/1.log"
Sep 16 18:53:39.483566 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:39.483548    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-28115f-predictor-57bb46966f-jx4sx"
Sep 16 18:53:39.552412 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:39.552376    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae207379-aa2c-4b5e-b959-b4562ec3623e-cabundle-cert\") pod \"ae207379-aa2c-4b5e-b959-b4562ec3623e\" (UID: \"ae207379-aa2c-4b5e-b959-b4562ec3623e\") "
Sep 16 18:53:39.552594 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:39.552449    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae207379-aa2c-4b5e-b959-b4562ec3623e-kserve-provision-location\") pod \"ae207379-aa2c-4b5e-b959-b4562ec3623e\" (UID: \"ae207379-aa2c-4b5e-b959-b4562ec3623e\") "
Sep 16 18:53:39.552594 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:39.552507    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae207379-aa2c-4b5e-b959-b4562ec3623e-proxy-tls\") pod \"ae207379-aa2c-4b5e-b959-b4562ec3623e\" (UID: \"ae207379-aa2c-4b5e-b959-b4562ec3623e\") "
Sep 16 18:53:39.552594 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:39.552560    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae207379-aa2c-4b5e-b959-b4562ec3623e-kube-rbac-proxy-sar-config\") pod \"ae207379-aa2c-4b5e-b959-b4562ec3623e\" (UID: \"ae207379-aa2c-4b5e-b959-b4562ec3623e\") "
Sep 16 18:53:39.552594 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:39.552592    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wjzpq\" (UniqueName: \"kubernetes.io/projected/ae207379-aa2c-4b5e-b959-b4562ec3623e-kube-api-access-wjzpq\") pod \"ae207379-aa2c-4b5e-b959-b4562ec3623e\" (UID: \"ae207379-aa2c-4b5e-b959-b4562ec3623e\") "
Sep 16 18:53:39.552791 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:39.552740    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae207379-aa2c-4b5e-b959-b4562ec3623e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ae207379-aa2c-4b5e-b959-b4562ec3623e" (UID: "ae207379-aa2c-4b5e-b959-b4562ec3623e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:53:39.552848 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:39.552798    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae207379-aa2c-4b5e-b959-b4562ec3623e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ae207379-aa2c-4b5e-b959-b4562ec3623e" (UID: "ae207379-aa2c-4b5e-b959-b4562ec3623e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:53:39.552906 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:39.552844    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae207379-aa2c-4b5e-b959-b4562ec3623e-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 18:53:39.552970 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:39.552898    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae207379-aa2c-4b5e-b959-b4562ec3623e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ae207379-aa2c-4b5e-b959-b4562ec3623e" (UID: "ae207379-aa2c-4b5e-b959-b4562ec3623e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:53:39.554718 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:39.554685    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae207379-aa2c-4b5e-b959-b4562ec3623e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ae207379-aa2c-4b5e-b959-b4562ec3623e" (UID: "ae207379-aa2c-4b5e-b959-b4562ec3623e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:53:39.554824 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:39.554788    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae207379-aa2c-4b5e-b959-b4562ec3623e-kube-api-access-wjzpq" (OuterVolumeSpecName: "kube-api-access-wjzpq") pod "ae207379-aa2c-4b5e-b959-b4562ec3623e" (UID: "ae207379-aa2c-4b5e-b959-b4562ec3623e"). InnerVolumeSpecName "kube-api-access-wjzpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:53:39.654180 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:39.654141    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae207379-aa2c-4b5e-b959-b4562ec3623e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 18:53:39.654180 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:39.654180    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjzpq\" (UniqueName: \"kubernetes.io/projected/ae207379-aa2c-4b5e-b959-b4562ec3623e-kube-api-access-wjzpq\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 18:53:39.654392 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:39.654196    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae207379-aa2c-4b5e-b959-b4562ec3623e-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 18:53:39.654392 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:39.654211    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae207379-aa2c-4b5e-b959-b4562ec3623e-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 18:53:40.349451 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:40.349421    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-28115f-predictor-57bb46966f-jx4sx_ae207379-aa2c-4b5e-b959-b4562ec3623e/storage-initializer/1.log"
Sep 16 18:53:40.349831 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:40.349485    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-28115f-predictor-57bb46966f-jx4sx" event={"ID":"ae207379-aa2c-4b5e-b959-b4562ec3623e","Type":"ContainerDied","Data":"b0e2db3f1b4c10fee39b70001c06507a5e3174e41e37a758096455c14e462359"}
Sep 16 18:53:40.349831 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:40.349525    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-28115f-predictor-57bb46966f-jx4sx"
Sep 16 18:53:40.349831 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:40.349537    2586 scope.go:117] "RemoveContainer" containerID="8749b2fb0ad733d5d419027855a226fc97ed06c45a1784c057359c8d82ec5c25"
Sep 16 18:53:40.395815 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:40.395783    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-28115f-predictor-57bb46966f-jx4sx"]
Sep 16 18:53:40.402362 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:40.402337    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-28115f-predictor-57bb46966f-jx4sx"]
Sep 16 18:53:41.091703 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:41.091668    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae207379-aa2c-4b5e-b959-b4562ec3623e" path="/var/lib/kubelet/pods/ae207379-aa2c-4b5e-b959-b4562ec3623e/volumes"
Sep 16 18:53:44.065533 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:44.065498    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hv478"]
Sep 16 18:53:44.065886 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:44.065824    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae207379-aa2c-4b5e-b959-b4562ec3623e" containerName="storage-initializer"
Sep 16 18:53:44.065886 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:44.065835    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae207379-aa2c-4b5e-b959-b4562ec3623e" containerName="storage-initializer"
Sep 16 18:53:44.065949 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:44.065894    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae207379-aa2c-4b5e-b959-b4562ec3623e" containerName="storage-initializer"
Sep 16 18:53:44.065949 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:44.065901    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae207379-aa2c-4b5e-b959-b4562ec3623e" containerName="storage-initializer"
Sep 16 18:53:44.066354 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:44.065959    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae207379-aa2c-4b5e-b959-b4562ec3623e" containerName="storage-initializer"
Sep 16 18:53:44.066354 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:44.065966    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae207379-aa2c-4b5e-b959-b4562ec3623e" containerName="storage-initializer"
Sep 16 18:53:44.070596 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:44.070574    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hv478"
Sep 16 18:53:44.073180 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:44.073156    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 18:53:44.073445 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:44.073425    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 18:53:44.073546 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:44.073189    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:53:44.073546 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:44.073189    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
Sep 16 18:53:44.073638 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:44.073209    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
Sep 16 18:53:44.073638 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:44.073188    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 18:53:44.074089 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:44.074072    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-pv9ct\""
Sep 16 18:53:44.081385 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:44.081364    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hv478"]
Sep 16 18:53:44.194982 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:44.194947    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9170f668-fc9d-47e2-b0e2-20a26036ed6b-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hv478\" (UID: \"9170f668-fc9d-47e2-b0e2-20a26036ed6b\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hv478"
Sep 16 18:53:44.195156 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:44.195079    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9170f668-fc9d-47e2-b0e2-20a26036ed6b-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hv478\" (UID: \"9170f668-fc9d-47e2-b0e2-20a26036ed6b\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hv478"
Sep 16 18:53:44.195156 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:44.195112    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9170f668-fc9d-47e2-b0e2-20a26036ed6b-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hv478\" (UID: \"9170f668-fc9d-47e2-b0e2-20a26036ed6b\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hv478"
Sep 16 18:53:44.195156 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:44.195138    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hp6z\" (UniqueName: \"kubernetes.io/projected/9170f668-fc9d-47e2-b0e2-20a26036ed6b-kube-api-access-5hp6z\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hv478\" (UID: \"9170f668-fc9d-47e2-b0e2-20a26036ed6b\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hv478"
Sep 16 18:53:44.195343 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:44.195161    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9170f668-fc9d-47e2-b0e2-20a26036ed6b-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hv478\" (UID: \"9170f668-fc9d-47e2-b0e2-20a26036ed6b\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hv478"
Sep 16 18:53:44.296040 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:44.295997    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9170f668-fc9d-47e2-b0e2-20a26036ed6b-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hv478\" (UID: \"9170f668-fc9d-47e2-b0e2-20a26036ed6b\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hv478"
Sep 16 18:53:44.296198 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:44.296135    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9170f668-fc9d-47e2-b0e2-20a26036ed6b-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hv478\" (UID: \"9170f668-fc9d-47e2-b0e2-20a26036ed6b\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hv478"
Sep 16 18:53:44.296198 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:44.296167    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9170f668-fc9d-47e2-b0e2-20a26036ed6b-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hv478\" (UID: \"9170f668-fc9d-47e2-b0e2-20a26036ed6b\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hv478"
Sep 16 18:53:44.296354 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:44.296209    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hp6z\" (UniqueName: \"kubernetes.io/projected/9170f668-fc9d-47e2-b0e2-20a26036ed6b-kube-api-access-5hp6z\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hv478\" (UID: \"9170f668-fc9d-47e2-b0e2-20a26036ed6b\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hv478"
Sep 16 18:53:44.296354 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:44.296267    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9170f668-fc9d-47e2-b0e2-20a26036ed6b-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hv478\" (UID: \"9170f668-fc9d-47e2-b0e2-20a26036ed6b\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hv478"
Sep 16 18:53:44.296638 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:44.296612    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9170f668-fc9d-47e2-b0e2-20a26036ed6b-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hv478\" (UID: \"9170f668-fc9d-47e2-b0e2-20a26036ed6b\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hv478"
Sep 16 18:53:44.296832 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:44.296810    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9170f668-fc9d-47e2-b0e2-20a26036ed6b-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hv478\" (UID: \"9170f668-fc9d-47e2-b0e2-20a26036ed6b\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hv478"
Sep 16 18:53:44.296926 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:44.296907    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9170f668-fc9d-47e2-b0e2-20a26036ed6b-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hv478\" (UID: \"9170f668-fc9d-47e2-b0e2-20a26036ed6b\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hv478"
Sep 16 18:53:44.298867 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:44.298845    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9170f668-fc9d-47e2-b0e2-20a26036ed6b-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hv478\" (UID: \"9170f668-fc9d-47e2-b0e2-20a26036ed6b\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hv478"
Sep 16 18:53:44.307675 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:44.307656    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hp6z\" (UniqueName: \"kubernetes.io/projected/9170f668-fc9d-47e2-b0e2-20a26036ed6b-kube-api-access-5hp6z\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-hv478\" (UID: \"9170f668-fc9d-47e2-b0e2-20a26036ed6b\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hv478"
Sep 16 18:53:44.382001 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:44.381975    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hv478"
Sep 16 18:53:44.657406 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:44.657379    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hv478"]
Sep 16 18:53:44.660292 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:53:44.660265    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9170f668_fc9d_47e2_b0e2_20a26036ed6b.slice/crio-23f086deac37f75f7d3f60948821f1df4e50be7792eb197198be7d50f25dc4c4 WatchSource:0}: Error finding container 23f086deac37f75f7d3f60948821f1df4e50be7792eb197198be7d50f25dc4c4: Status 404 returned error can't find the container with id 23f086deac37f75f7d3f60948821f1df4e50be7792eb197198be7d50f25dc4c4
Sep 16 18:53:45.374084 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:45.374048    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hv478" event={"ID":"9170f668-fc9d-47e2-b0e2-20a26036ed6b","Type":"ContainerStarted","Data":"f927f6b2c8a158d25ec0039e6c29ca4a62f961a82b3191d705ce442099df347e"}
Sep 16 18:53:45.374454 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:45.374102    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hv478" event={"ID":"9170f668-fc9d-47e2-b0e2-20a26036ed6b","Type":"ContainerStarted","Data":"23f086deac37f75f7d3f60948821f1df4e50be7792eb197198be7d50f25dc4c4"}
Sep 16 18:53:49.393151 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:49.393116    2586 generic.go:356] "Generic (PLEG): container finished" podID="9170f668-fc9d-47e2-b0e2-20a26036ed6b" containerID="f927f6b2c8a158d25ec0039e6c29ca4a62f961a82b3191d705ce442099df347e" exitCode=0
Sep 16 18:53:49.393557 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:53:49.393186    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hv478" event={"ID":"9170f668-fc9d-47e2-b0e2-20a26036ed6b","Type":"ContainerDied","Data":"f927f6b2c8a158d25ec0039e6c29ca4a62f961a82b3191d705ce442099df347e"}
Sep 16 18:54:11.521456 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:54:11.521423    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hv478" event={"ID":"9170f668-fc9d-47e2-b0e2-20a26036ed6b","Type":"ContainerStarted","Data":"b490b6a5fe464cd15654eeaa7d5507bfab3ef36ad063f1ef1b5328a19212e63c"}
Sep 16 18:54:14.548677 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:54:14.548641    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hv478" event={"ID":"9170f668-fc9d-47e2-b0e2-20a26036ed6b","Type":"ContainerStarted","Data":"44d7c12e83cac87e6ed90514aed18add1accc5549506a50badc6f6689e5d3dcc"}
Sep 16 18:54:14.549113 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:54:14.548739    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hv478"
Sep 16 18:54:14.549113 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:54:14.548772    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hv478"
Sep 16 18:54:14.549918 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:54:14.549888    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hv478" podUID="9170f668-fc9d-47e2-b0e2-20a26036ed6b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 18:54:14.574257 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:54:14.574174    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hv478" podStartSLOduration=1.951600231 podStartE2EDuration="30.574159359s" totalImagesPullingTime="24.783923116s" totalInitContainerRuntime="3.838636012s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:53:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:54:14.572579483 +0000 UTC m=+830.016441060" watchObservedRunningTime="2026-09-16 18:54:14.574159359 +0000 UTC m=+830.018020934"
Sep 16 18:54:15.553161 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:54:15.553122    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hv478" podUID="9170f668-fc9d-47e2-b0e2-20a26036ed6b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 18:54:20.558583 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:54:20.558552    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hv478"
Sep 16 18:54:20.559033 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:54:20.559005    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hv478" podUID="9170f668-fc9d-47e2-b0e2-20a26036ed6b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 18:54:30.559116 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:54:30.559078    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hv478" podUID="9170f668-fc9d-47e2-b0e2-20a26036ed6b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 18:54:40.559010 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:54:40.558970    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hv478" podUID="9170f668-fc9d-47e2-b0e2-20a26036ed6b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 18:54:50.559316 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:54:50.559274    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hv478" podUID="9170f668-fc9d-47e2-b0e2-20a26036ed6b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 18:55:00.559078 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:00.559035    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hv478" podUID="9170f668-fc9d-47e2-b0e2-20a26036ed6b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 18:55:10.559268 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:10.559202    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hv478" podUID="9170f668-fc9d-47e2-b0e2-20a26036ed6b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 18:55:20.559134 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:20.559040    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hv478" podUID="9170f668-fc9d-47e2-b0e2-20a26036ed6b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 18:55:22.086934 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:22.086889    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hv478" podUID="9170f668-fc9d-47e2-b0e2-20a26036ed6b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 18:55:32.088065 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:32.088032    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hv478"
Sep 16 18:55:34.288280 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:34.288223    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hv478"]
Sep 16 18:55:34.288663 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:34.288476    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hv478" podUID="9170f668-fc9d-47e2-b0e2-20a26036ed6b" containerName="kserve-container" containerID="cri-o://b490b6a5fe464cd15654eeaa7d5507bfab3ef36ad063f1ef1b5328a19212e63c" gracePeriod=30
Sep 16 18:55:34.288663 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:34.288529    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hv478" podUID="9170f668-fc9d-47e2-b0e2-20a26036ed6b" containerName="kube-rbac-proxy" containerID="cri-o://44d7c12e83cac87e6ed90514aed18add1accc5549506a50badc6f6689e5d3dcc" gracePeriod=30
Sep 16 18:55:34.915458 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:34.915424    2586 generic.go:356] "Generic (PLEG): container finished" podID="9170f668-fc9d-47e2-b0e2-20a26036ed6b" containerID="44d7c12e83cac87e6ed90514aed18add1accc5549506a50badc6f6689e5d3dcc" exitCode=2
Sep 16 18:55:34.915627 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:34.915497    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hv478" event={"ID":"9170f668-fc9d-47e2-b0e2-20a26036ed6b","Type":"ContainerDied","Data":"44d7c12e83cac87e6ed90514aed18add1accc5549506a50badc6f6689e5d3dcc"}
Sep 16 18:55:35.554163 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:35.554117    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hv478" podUID="9170f668-fc9d-47e2-b0e2-20a26036ed6b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Sep 16 18:55:38.936784 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:38.936749    2586 generic.go:356] "Generic (PLEG): container finished" podID="9170f668-fc9d-47e2-b0e2-20a26036ed6b" containerID="b490b6a5fe464cd15654eeaa7d5507bfab3ef36ad063f1ef1b5328a19212e63c" exitCode=0
Sep 16 18:55:38.937132 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:38.936808    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hv478" event={"ID":"9170f668-fc9d-47e2-b0e2-20a26036ed6b","Type":"ContainerDied","Data":"b490b6a5fe464cd15654eeaa7d5507bfab3ef36ad063f1ef1b5328a19212e63c"}
Sep 16 18:55:39.034213 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:39.034189    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hv478"
Sep 16 18:55:39.098514 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:39.098494    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9170f668-fc9d-47e2-b0e2-20a26036ed6b-cabundle-cert\") pod \"9170f668-fc9d-47e2-b0e2-20a26036ed6b\" (UID: \"9170f668-fc9d-47e2-b0e2-20a26036ed6b\") "
Sep 16 18:55:39.098821 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:39.098800    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9170f668-fc9d-47e2-b0e2-20a26036ed6b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9170f668-fc9d-47e2-b0e2-20a26036ed6b" (UID: "9170f668-fc9d-47e2-b0e2-20a26036ed6b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:55:39.199074 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:39.199045    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hp6z\" (UniqueName: \"kubernetes.io/projected/9170f668-fc9d-47e2-b0e2-20a26036ed6b-kube-api-access-5hp6z\") pod \"9170f668-fc9d-47e2-b0e2-20a26036ed6b\" (UID: \"9170f668-fc9d-47e2-b0e2-20a26036ed6b\") "
Sep 16 18:55:39.199257 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:39.199088    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9170f668-fc9d-47e2-b0e2-20a26036ed6b-kserve-provision-location\") pod \"9170f668-fc9d-47e2-b0e2-20a26036ed6b\" (UID: \"9170f668-fc9d-47e2-b0e2-20a26036ed6b\") "
Sep 16 18:55:39.199257 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:39.199112    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9170f668-fc9d-47e2-b0e2-20a26036ed6b-kube-rbac-proxy-sar-config\") pod \"9170f668-fc9d-47e2-b0e2-20a26036ed6b\" (UID: \"9170f668-fc9d-47e2-b0e2-20a26036ed6b\") "
Sep 16 18:55:39.199257 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:39.199142    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9170f668-fc9d-47e2-b0e2-20a26036ed6b-proxy-tls\") pod \"9170f668-fc9d-47e2-b0e2-20a26036ed6b\" (UID: \"9170f668-fc9d-47e2-b0e2-20a26036ed6b\") "
Sep 16 18:55:39.199434 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:39.199412    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9170f668-fc9d-47e2-b0e2-20a26036ed6b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9170f668-fc9d-47e2-b0e2-20a26036ed6b" (UID: "9170f668-fc9d-47e2-b0e2-20a26036ed6b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:39.199484 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:39.199431    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9170f668-fc9d-47e2-b0e2-20a26036ed6b-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 18:55:39.199552 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:39.199528    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9170f668-fc9d-47e2-b0e2-20a26036ed6b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9170f668-fc9d-47e2-b0e2-20a26036ed6b" (UID: "9170f668-fc9d-47e2-b0e2-20a26036ed6b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:55:39.201249 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:39.201201    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9170f668-fc9d-47e2-b0e2-20a26036ed6b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9170f668-fc9d-47e2-b0e2-20a26036ed6b" (UID: "9170f668-fc9d-47e2-b0e2-20a26036ed6b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:55:39.201249 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:39.201221    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9170f668-fc9d-47e2-b0e2-20a26036ed6b-kube-api-access-5hp6z" (OuterVolumeSpecName: "kube-api-access-5hp6z") pod "9170f668-fc9d-47e2-b0e2-20a26036ed6b" (UID: "9170f668-fc9d-47e2-b0e2-20a26036ed6b"). InnerVolumeSpecName "kube-api-access-5hp6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:55:39.300333 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:39.300298    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9170f668-fc9d-47e2-b0e2-20a26036ed6b-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 18:55:39.300333 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:39.300327    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9170f668-fc9d-47e2-b0e2-20a26036ed6b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 18:55:39.300333 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:39.300337    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9170f668-fc9d-47e2-b0e2-20a26036ed6b-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 18:55:39.300546 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:39.300348    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hp6z\" (UniqueName: \"kubernetes.io/projected/9170f668-fc9d-47e2-b0e2-20a26036ed6b-kube-api-access-5hp6z\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 18:55:39.478442 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:39.478362    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-4wjgg"]
Sep 16 18:55:39.478743 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:39.478730    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9170f668-fc9d-47e2-b0e2-20a26036ed6b" containerName="kserve-container"
Sep 16 18:55:39.478785 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:39.478745    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="9170f668-fc9d-47e2-b0e2-20a26036ed6b" containerName="kserve-container"
Sep 16 18:55:39.478785 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:39.478759    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9170f668-fc9d-47e2-b0e2-20a26036ed6b" containerName="kube-rbac-proxy"
Sep 16 18:55:39.478785 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:39.478764    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="9170f668-fc9d-47e2-b0e2-20a26036ed6b" containerName="kube-rbac-proxy"
Sep 16 18:55:39.478785 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:39.478783    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9170f668-fc9d-47e2-b0e2-20a26036ed6b" containerName="storage-initializer"
Sep 16 18:55:39.478905 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:39.478792    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="9170f668-fc9d-47e2-b0e2-20a26036ed6b" containerName="storage-initializer"
Sep 16 18:55:39.478905 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:39.478842    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="9170f668-fc9d-47e2-b0e2-20a26036ed6b" containerName="kube-rbac-proxy"
Sep 16 18:55:39.478905 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:39.478851    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="9170f668-fc9d-47e2-b0e2-20a26036ed6b" containerName="kserve-container"
Sep 16 18:55:39.481148 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:39.481132    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-4wjgg"
Sep 16 18:55:39.483407 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:39.483385    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
Sep 16 18:55:39.483552 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:39.483536    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
Sep 16 18:55:39.495165 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:39.495141    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-4wjgg"]
Sep 16 18:55:39.502393 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:39.502366    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66c61900-ff5c-499c-a9b2-395f6a806f7b-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-4wjgg\" (UID: \"66c61900-ff5c-499c-a9b2-395f6a806f7b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-4wjgg"
Sep 16 18:55:39.502502 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:39.502406    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss92m\" (UniqueName: \"kubernetes.io/projected/66c61900-ff5c-499c-a9b2-395f6a806f7b-kube-api-access-ss92m\") pod \"isvc-predictive-xgboost-predictor-695f978978-4wjgg\" (UID: \"66c61900-ff5c-499c-a9b2-395f6a806f7b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-4wjgg"
Sep 16 18:55:39.502502 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:39.502426    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66c61900-ff5c-499c-a9b2-395f6a806f7b-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-4wjgg\" (UID: \"66c61900-ff5c-499c-a9b2-395f6a806f7b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-4wjgg"
Sep 16 18:55:39.502502 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:39.502471    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66c61900-ff5c-499c-a9b2-395f6a806f7b-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-4wjgg\" (UID: \"66c61900-ff5c-499c-a9b2-395f6a806f7b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-4wjgg"
Sep 16 18:55:39.502676 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:39.502505    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66c61900-ff5c-499c-a9b2-395f6a806f7b-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-4wjgg\" (UID: \"66c61900-ff5c-499c-a9b2-395f6a806f7b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-4wjgg"
Sep 16 18:55:39.603643 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:39.603613    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66c61900-ff5c-499c-a9b2-395f6a806f7b-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-4wjgg\" (UID: \"66c61900-ff5c-499c-a9b2-395f6a806f7b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-4wjgg"
Sep 16 18:55:39.603836 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:39.603659    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66c61900-ff5c-499c-a9b2-395f6a806f7b-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-4wjgg\" (UID: \"66c61900-ff5c-499c-a9b2-395f6a806f7b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-4wjgg"
Sep 16 18:55:39.603836 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:39.603690    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss92m\" (UniqueName: \"kubernetes.io/projected/66c61900-ff5c-499c-a9b2-395f6a806f7b-kube-api-access-ss92m\") pod \"isvc-predictive-xgboost-predictor-695f978978-4wjgg\" (UID: \"66c61900-ff5c-499c-a9b2-395f6a806f7b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-4wjgg"
Sep 16 18:55:39.603836 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:39.603718    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66c61900-ff5c-499c-a9b2-395f6a806f7b-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-4wjgg\" (UID: \"66c61900-ff5c-499c-a9b2-395f6a806f7b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-4wjgg"
Sep 16 18:55:39.603836 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:39.603749    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66c61900-ff5c-499c-a9b2-395f6a806f7b-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-4wjgg\" (UID: \"66c61900-ff5c-499c-a9b2-395f6a806f7b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-4wjgg"
Sep 16 18:55:39.604199 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:39.604172    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66c61900-ff5c-499c-a9b2-395f6a806f7b-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-4wjgg\" (UID: \"66c61900-ff5c-499c-a9b2-395f6a806f7b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-4wjgg"
Sep 16 18:55:39.604470 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:39.604450    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66c61900-ff5c-499c-a9b2-395f6a806f7b-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-4wjgg\" (UID: \"66c61900-ff5c-499c-a9b2-395f6a806f7b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-4wjgg"
Sep 16 18:55:39.604532 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:39.604475    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66c61900-ff5c-499c-a9b2-395f6a806f7b-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-4wjgg\" (UID: \"66c61900-ff5c-499c-a9b2-395f6a806f7b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-4wjgg"
Sep 16 18:55:39.606210 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:39.606192    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66c61900-ff5c-499c-a9b2-395f6a806f7b-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-4wjgg\" (UID: \"66c61900-ff5c-499c-a9b2-395f6a806f7b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-4wjgg"
Sep 16 18:55:39.614217 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:39.614192    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss92m\" (UniqueName: \"kubernetes.io/projected/66c61900-ff5c-499c-a9b2-395f6a806f7b-kube-api-access-ss92m\") pod \"isvc-predictive-xgboost-predictor-695f978978-4wjgg\" (UID: \"66c61900-ff5c-499c-a9b2-395f6a806f7b\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-4wjgg"
Sep 16 18:55:39.792813 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:39.792727    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-4wjgg"
Sep 16 18:55:39.943445 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:39.943408    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hv478" event={"ID":"9170f668-fc9d-47e2-b0e2-20a26036ed6b","Type":"ContainerDied","Data":"23f086deac37f75f7d3f60948821f1df4e50be7792eb197198be7d50f25dc4c4"}
Sep 16 18:55:39.943801 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:39.943453    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hv478"
Sep 16 18:55:39.943801 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:39.943481    2586 scope.go:117] "RemoveContainer" containerID="44d7c12e83cac87e6ed90514aed18add1accc5549506a50badc6f6689e5d3dcc"
Sep 16 18:55:39.951642 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:39.951625    2586 scope.go:117] "RemoveContainer" containerID="b490b6a5fe464cd15654eeaa7d5507bfab3ef36ad063f1ef1b5328a19212e63c"
Sep 16 18:55:39.960762 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:39.960738    2586 scope.go:117] "RemoveContainer" containerID="f927f6b2c8a158d25ec0039e6c29ca4a62f961a82b3191d705ce442099df347e"
Sep 16 18:55:39.969397 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:39.969371    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hv478"]
Sep 16 18:55:39.973356 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:39.973333    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-hv478"]
Sep 16 18:55:40.068302 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:40.068204    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-4wjgg"]
Sep 16 18:55:40.071585 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:55:40.071545    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66c61900_ff5c_499c_a9b2_395f6a806f7b.slice/crio-2e01c835191d47941b4fa757c144a601860a5a109e91637e64b0c8997732bf04 WatchSource:0}: Error finding container 2e01c835191d47941b4fa757c144a601860a5a109e91637e64b0c8997732bf04: Status 404 returned error can't find the container with id 2e01c835191d47941b4fa757c144a601860a5a109e91637e64b0c8997732bf04
Sep 16 18:55:40.949957 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:40.949923    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-4wjgg" event={"ID":"66c61900-ff5c-499c-a9b2-395f6a806f7b","Type":"ContainerStarted","Data":"524d7089f99abeb55b7b970b0deb0a9da1fd702952b69f665031e7bfd77d9a46"}
Sep 16 18:55:40.950361 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:40.949992    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-4wjgg" event={"ID":"66c61900-ff5c-499c-a9b2-395f6a806f7b","Type":"ContainerStarted","Data":"2e01c835191d47941b4fa757c144a601860a5a109e91637e64b0c8997732bf04"}
Sep 16 18:55:41.092408 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:41.092375    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9170f668-fc9d-47e2-b0e2-20a26036ed6b" path="/var/lib/kubelet/pods/9170f668-fc9d-47e2-b0e2-20a26036ed6b/volumes"
Sep 16 18:55:44.969505 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:44.969471    2586 generic.go:356] "Generic (PLEG): container finished" podID="66c61900-ff5c-499c-a9b2-395f6a806f7b" containerID="524d7089f99abeb55b7b970b0deb0a9da1fd702952b69f665031e7bfd77d9a46" exitCode=0
Sep 16 18:55:44.969890 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:44.969533    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-4wjgg" event={"ID":"66c61900-ff5c-499c-a9b2-395f6a806f7b","Type":"ContainerDied","Data":"524d7089f99abeb55b7b970b0deb0a9da1fd702952b69f665031e7bfd77d9a46"}
Sep 16 18:55:45.976157 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:45.976122    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-4wjgg" event={"ID":"66c61900-ff5c-499c-a9b2-395f6a806f7b","Type":"ContainerStarted","Data":"19f45e03e0a798e887acbeaa9ed5071a33629851d26139007cd7f13f646a83a7"}
Sep 16 18:55:45.976579 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:45.976182    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-4wjgg" event={"ID":"66c61900-ff5c-499c-a9b2-395f6a806f7b","Type":"ContainerStarted","Data":"81ea9ed131a6aff65e3059d3ebcb41d553c9b3ef9624750de85392fafd739fe6"}
Sep 16 18:55:45.976579 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:45.976483    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-4wjgg"
Sep 16 18:55:45.976664 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:45.976644    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-4wjgg"
Sep 16 18:55:45.977972 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:45.977943    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-4wjgg" podUID="66c61900-ff5c-499c-a9b2-395f6a806f7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 16 18:55:46.002164 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:46.002123    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-4wjgg" podStartSLOduration=2.95564161 podStartE2EDuration="7.002110328s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.046468718s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:55:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:55:46.00145005 +0000 UTC m=+921.445311629" watchObservedRunningTime="2026-09-16 18:55:46.002110328 +0000 UTC m=+921.445971904"
Sep 16 18:55:46.982065 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:46.982026    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-4wjgg" podUID="66c61900-ff5c-499c-a9b2-395f6a806f7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 16 18:55:51.987112 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:51.987076    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-4wjgg"
Sep 16 18:55:51.987608 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:55:51.987581    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-4wjgg" podUID="66c61900-ff5c-499c-a9b2-395f6a806f7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 16 18:56:01.988139 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:56:01.988096    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-4wjgg" podUID="66c61900-ff5c-499c-a9b2-395f6a806f7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 16 18:56:11.987701 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:56:11.987653    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-4wjgg" podUID="66c61900-ff5c-499c-a9b2-395f6a806f7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 16 18:56:21.987854 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:56:21.987808    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-4wjgg" podUID="66c61900-ff5c-499c-a9b2-395f6a806f7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 16 18:56:31.988274 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:56:31.988213    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-4wjgg" podUID="66c61900-ff5c-499c-a9b2-395f6a806f7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 16 18:56:41.988470 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:56:41.988425    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-4wjgg" podUID="66c61900-ff5c-499c-a9b2-395f6a806f7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 16 18:56:44.500855 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:56:44.500821    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kmhd5"]
Sep 16 18:56:44.505726 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:56:44.505703    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kmhd5"
Sep 16 18:56:44.509982 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:56:44.509961    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Sep 16 18:56:44.510103 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:56:44.510036    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Sep 16 18:56:44.520065 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:56:44.520038    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kmhd5"]
Sep 16 18:56:44.544417 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:56:44.544389    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcrwz\" (UniqueName: \"kubernetes.io/projected/99c64e29-e71f-4f90-ac1a-43e016a420bd-kube-api-access-gcrwz\") pod \"message-dumper-predictor-6575bcfdfd-kmhd5\" (UID: \"99c64e29-e71f-4f90-ac1a-43e016a420bd\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kmhd5"
Sep 16 18:56:44.544536 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:56:44.544424    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99c64e29-e71f-4f90-ac1a-43e016a420bd-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-kmhd5\" (UID: \"99c64e29-e71f-4f90-ac1a-43e016a420bd\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kmhd5"
Sep 16 18:56:44.544536 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:56:44.544504    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99c64e29-e71f-4f90-ac1a-43e016a420bd-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-kmhd5\" (UID: \"99c64e29-e71f-4f90-ac1a-43e016a420bd\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kmhd5"
Sep 16 18:56:44.645466 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:56:44.645428    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99c64e29-e71f-4f90-ac1a-43e016a420bd-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-kmhd5\" (UID: \"99c64e29-e71f-4f90-ac1a-43e016a420bd\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kmhd5"
Sep 16 18:56:44.645630 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:56:44.645500    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcrwz\" (UniqueName: \"kubernetes.io/projected/99c64e29-e71f-4f90-ac1a-43e016a420bd-kube-api-access-gcrwz\") pod \"message-dumper-predictor-6575bcfdfd-kmhd5\" (UID: \"99c64e29-e71f-4f90-ac1a-43e016a420bd\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kmhd5"
Sep 16 18:56:44.645630 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:56:44.645520    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99c64e29-e71f-4f90-ac1a-43e016a420bd-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-kmhd5\" (UID: \"99c64e29-e71f-4f90-ac1a-43e016a420bd\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kmhd5"
Sep 16 18:56:44.646084 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:56:44.646065    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99c64e29-e71f-4f90-ac1a-43e016a420bd-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-kmhd5\" (UID: \"99c64e29-e71f-4f90-ac1a-43e016a420bd\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kmhd5"
Sep 16 18:56:44.647887 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:56:44.647868    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99c64e29-e71f-4f90-ac1a-43e016a420bd-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-kmhd5\" (UID: \"99c64e29-e71f-4f90-ac1a-43e016a420bd\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kmhd5"
Sep 16 18:56:44.655712 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:56:44.655680    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcrwz\" (UniqueName: \"kubernetes.io/projected/99c64e29-e71f-4f90-ac1a-43e016a420bd-kube-api-access-gcrwz\") pod \"message-dumper-predictor-6575bcfdfd-kmhd5\" (UID: \"99c64e29-e71f-4f90-ac1a-43e016a420bd\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kmhd5"
Sep 16 18:56:44.816115 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:56:44.816036    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kmhd5"
Sep 16 18:56:45.093064 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:56:45.093044    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kmhd5"]
Sep 16 18:56:45.094982 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:56:45.094955    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99c64e29_e71f_4f90_ac1a_43e016a420bd.slice/crio-d68ad0305296aa14474ea33184d6ee45d09eac7e1d5758db11803011bbddd0e3 WatchSource:0}: Error finding container d68ad0305296aa14474ea33184d6ee45d09eac7e1d5758db11803011bbddd0e3: Status 404 returned error can't find the container with id d68ad0305296aa14474ea33184d6ee45d09eac7e1d5758db11803011bbddd0e3
Sep 16 18:56:45.246081 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:56:45.246044    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kmhd5" event={"ID":"99c64e29-e71f-4f90-ac1a-43e016a420bd","Type":"ContainerStarted","Data":"d68ad0305296aa14474ea33184d6ee45d09eac7e1d5758db11803011bbddd0e3"}
Sep 16 18:56:47.256946 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:56:47.256861    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kmhd5" event={"ID":"99c64e29-e71f-4f90-ac1a-43e016a420bd","Type":"ContainerStarted","Data":"6deda5eef23f44796f663de63f3850101554efdc691d4b349a4df3a615ddda36"}
Sep 16 18:56:47.257341 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:56:47.256963    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kmhd5" event={"ID":"99c64e29-e71f-4f90-ac1a-43e016a420bd","Type":"ContainerStarted","Data":"ecf5ac2433b8d290cf8cc1ae872fbcaa986ad0a9c38d54c870d0b88d21ac96f2"}
Sep 16 18:56:47.257341 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:56:47.257113    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kmhd5"
Sep 16 18:56:47.257341 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:56:47.257172    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kmhd5"
Sep 16 18:56:47.259044 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:56:47.259023    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kmhd5"
Sep 16 18:56:47.333827 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:56:47.333754    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kmhd5" podStartSLOduration=2.214128847 podStartE2EDuration="3.333732636s" totalImagesPullingTime="1.119603789s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:56:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:56:47.299918243 +0000 UTC m=+982.743779818" watchObservedRunningTime="2026-09-16 18:56:47.333732636 +0000 UTC m=+982.777594214"
Sep 16 18:56:51.988453 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:56:51.988414    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-4wjgg" podUID="66c61900-ff5c-499c-a9b2-395f6a806f7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 16 18:56:54.270652 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:56:54.270624    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kmhd5"
Sep 16 18:57:01.988829 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:01.988797    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-4wjgg"
Sep 16 18:57:09.558220 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:09.558184    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-4wjgg"]
Sep 16 18:57:09.558632 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:09.558438    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-4wjgg" podUID="66c61900-ff5c-499c-a9b2-395f6a806f7b" containerName="kserve-container" containerID="cri-o://81ea9ed131a6aff65e3059d3ebcb41d553c9b3ef9624750de85392fafd739fe6" gracePeriod=30
Sep 16 18:57:09.558632 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:09.558474    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-4wjgg" podUID="66c61900-ff5c-499c-a9b2-395f6a806f7b" containerName="kube-rbac-proxy" containerID="cri-o://19f45e03e0a798e887acbeaa9ed5071a33629851d26139007cd7f13f646a83a7" gracePeriod=30
Sep 16 18:57:10.364685 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:10.364654    2586 generic.go:356] "Generic (PLEG): container finished" podID="66c61900-ff5c-499c-a9b2-395f6a806f7b" containerID="19f45e03e0a798e887acbeaa9ed5071a33629851d26139007cd7f13f646a83a7" exitCode=2
Sep 16 18:57:10.364862 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:10.364717    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-4wjgg" event={"ID":"66c61900-ff5c-499c-a9b2-395f6a806f7b","Type":"ContainerDied","Data":"19f45e03e0a798e887acbeaa9ed5071a33629851d26139007cd7f13f646a83a7"}
Sep 16 18:57:11.982645 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:11.982597    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-4wjgg" podUID="66c61900-ff5c-499c-a9b2-395f6a806f7b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Sep 16 18:57:11.988044 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:11.988012    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-4wjgg" podUID="66c61900-ff5c-499c-a9b2-395f6a806f7b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 16 18:57:14.208641 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:14.208618    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-4wjgg"
Sep 16 18:57:14.304583 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:14.304480    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66c61900-ff5c-499c-a9b2-395f6a806f7b-cabundle-cert\") pod \"66c61900-ff5c-499c-a9b2-395f6a806f7b\" (UID: \"66c61900-ff5c-499c-a9b2-395f6a806f7b\") "
Sep 16 18:57:14.304583 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:14.304519    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ss92m\" (UniqueName: \"kubernetes.io/projected/66c61900-ff5c-499c-a9b2-395f6a806f7b-kube-api-access-ss92m\") pod \"66c61900-ff5c-499c-a9b2-395f6a806f7b\" (UID: \"66c61900-ff5c-499c-a9b2-395f6a806f7b\") "
Sep 16 18:57:14.304583 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:14.304587    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66c61900-ff5c-499c-a9b2-395f6a806f7b-kserve-provision-location\") pod \"66c61900-ff5c-499c-a9b2-395f6a806f7b\" (UID: \"66c61900-ff5c-499c-a9b2-395f6a806f7b\") "
Sep 16 18:57:14.304828 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:14.304610    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66c61900-ff5c-499c-a9b2-395f6a806f7b-proxy-tls\") pod \"66c61900-ff5c-499c-a9b2-395f6a806f7b\" (UID: \"66c61900-ff5c-499c-a9b2-395f6a806f7b\") "
Sep 16 18:57:14.304828 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:14.304655    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66c61900-ff5c-499c-a9b2-395f6a806f7b-kube-rbac-proxy-sar-config\") pod \"66c61900-ff5c-499c-a9b2-395f6a806f7b\" (UID: \"66c61900-ff5c-499c-a9b2-395f6a806f7b\") "
Sep 16 18:57:14.305034 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:14.304918    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66c61900-ff5c-499c-a9b2-395f6a806f7b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "66c61900-ff5c-499c-a9b2-395f6a806f7b" (UID: "66c61900-ff5c-499c-a9b2-395f6a806f7b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:57:14.305087 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:14.304955    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66c61900-ff5c-499c-a9b2-395f6a806f7b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "66c61900-ff5c-499c-a9b2-395f6a806f7b" (UID: "66c61900-ff5c-499c-a9b2-395f6a806f7b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:57:14.305147 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:14.305086    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66c61900-ff5c-499c-a9b2-395f6a806f7b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "66c61900-ff5c-499c-a9b2-395f6a806f7b" (UID: "66c61900-ff5c-499c-a9b2-395f6a806f7b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:57:14.306708 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:14.306680    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66c61900-ff5c-499c-a9b2-395f6a806f7b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "66c61900-ff5c-499c-a9b2-395f6a806f7b" (UID: "66c61900-ff5c-499c-a9b2-395f6a806f7b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:57:14.306842 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:14.306823    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66c61900-ff5c-499c-a9b2-395f6a806f7b-kube-api-access-ss92m" (OuterVolumeSpecName: "kube-api-access-ss92m") pod "66c61900-ff5c-499c-a9b2-395f6a806f7b" (UID: "66c61900-ff5c-499c-a9b2-395f6a806f7b"). InnerVolumeSpecName "kube-api-access-ss92m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:57:14.384781 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:14.384747    2586 generic.go:356] "Generic (PLEG): container finished" podID="66c61900-ff5c-499c-a9b2-395f6a806f7b" containerID="81ea9ed131a6aff65e3059d3ebcb41d553c9b3ef9624750de85392fafd739fe6" exitCode=0
Sep 16 18:57:14.384948 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:14.384816    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-4wjgg"
Sep 16 18:57:14.384948 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:14.384823    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-4wjgg" event={"ID":"66c61900-ff5c-499c-a9b2-395f6a806f7b","Type":"ContainerDied","Data":"81ea9ed131a6aff65e3059d3ebcb41d553c9b3ef9624750de85392fafd739fe6"}
Sep 16 18:57:14.384948 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:14.384909    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-4wjgg" event={"ID":"66c61900-ff5c-499c-a9b2-395f6a806f7b","Type":"ContainerDied","Data":"2e01c835191d47941b4fa757c144a601860a5a109e91637e64b0c8997732bf04"}
Sep 16 18:57:14.384948 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:14.384937    2586 scope.go:117] "RemoveContainer" containerID="19f45e03e0a798e887acbeaa9ed5071a33629851d26139007cd7f13f646a83a7"
Sep 16 18:57:14.394059 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:14.394039    2586 scope.go:117] "RemoveContainer" containerID="81ea9ed131a6aff65e3059d3ebcb41d553c9b3ef9624750de85392fafd739fe6"
Sep 16 18:57:14.401337 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:14.401322    2586 scope.go:117] "RemoveContainer" containerID="524d7089f99abeb55b7b970b0deb0a9da1fd702952b69f665031e7bfd77d9a46"
Sep 16 18:57:14.405583 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:14.405564    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66c61900-ff5c-499c-a9b2-395f6a806f7b-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 18:57:14.405653 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:14.405586    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66c61900-ff5c-499c-a9b2-395f6a806f7b-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 18:57:14.405653 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:14.405596    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66c61900-ff5c-499c-a9b2-395f6a806f7b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 18:57:14.405653 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:14.405604    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66c61900-ff5c-499c-a9b2-395f6a806f7b-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 18:57:14.405653 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:14.405613    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss92m\" (UniqueName: \"kubernetes.io/projected/66c61900-ff5c-499c-a9b2-395f6a806f7b-kube-api-access-ss92m\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 18:57:14.408441 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:14.408416    2586 scope.go:117] "RemoveContainer" containerID="19f45e03e0a798e887acbeaa9ed5071a33629851d26139007cd7f13f646a83a7"
Sep 16 18:57:14.408684 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:57:14.408663    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19f45e03e0a798e887acbeaa9ed5071a33629851d26139007cd7f13f646a83a7\": container with ID starting with 19f45e03e0a798e887acbeaa9ed5071a33629851d26139007cd7f13f646a83a7 not found: ID does not exist" containerID="19f45e03e0a798e887acbeaa9ed5071a33629851d26139007cd7f13f646a83a7"
Sep 16 18:57:14.408738 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:14.408694    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19f45e03e0a798e887acbeaa9ed5071a33629851d26139007cd7f13f646a83a7"} err="failed to get container status \"19f45e03e0a798e887acbeaa9ed5071a33629851d26139007cd7f13f646a83a7\": rpc error: code = NotFound desc = could not find container \"19f45e03e0a798e887acbeaa9ed5071a33629851d26139007cd7f13f646a83a7\": container with ID starting with 19f45e03e0a798e887acbeaa9ed5071a33629851d26139007cd7f13f646a83a7 not found: ID does not exist"
Sep 16 18:57:14.408738 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:14.408711    2586 scope.go:117] "RemoveContainer" containerID="81ea9ed131a6aff65e3059d3ebcb41d553c9b3ef9624750de85392fafd739fe6"
Sep 16 18:57:14.408968 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:57:14.408953    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81ea9ed131a6aff65e3059d3ebcb41d553c9b3ef9624750de85392fafd739fe6\": container with ID starting with 81ea9ed131a6aff65e3059d3ebcb41d553c9b3ef9624750de85392fafd739fe6 not found: ID does not exist" containerID="81ea9ed131a6aff65e3059d3ebcb41d553c9b3ef9624750de85392fafd739fe6"
Sep 16 18:57:14.409018 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:14.408971    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81ea9ed131a6aff65e3059d3ebcb41d553c9b3ef9624750de85392fafd739fe6"} err="failed to get container status \"81ea9ed131a6aff65e3059d3ebcb41d553c9b3ef9624750de85392fafd739fe6\": rpc error: code = NotFound desc = could not find container \"81ea9ed131a6aff65e3059d3ebcb41d553c9b3ef9624750de85392fafd739fe6\": container with ID starting with 81ea9ed131a6aff65e3059d3ebcb41d553c9b3ef9624750de85392fafd739fe6 not found: ID does not exist"
Sep 16 18:57:14.409018 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:14.408983    2586 scope.go:117] "RemoveContainer" containerID="524d7089f99abeb55b7b970b0deb0a9da1fd702952b69f665031e7bfd77d9a46"
Sep 16 18:57:14.409201 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:57:14.409185    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"524d7089f99abeb55b7b970b0deb0a9da1fd702952b69f665031e7bfd77d9a46\": container with ID starting with 524d7089f99abeb55b7b970b0deb0a9da1fd702952b69f665031e7bfd77d9a46 not found: ID does not exist" containerID="524d7089f99abeb55b7b970b0deb0a9da1fd702952b69f665031e7bfd77d9a46"
Sep 16 18:57:14.409252 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:14.409207    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"524d7089f99abeb55b7b970b0deb0a9da1fd702952b69f665031e7bfd77d9a46"} err="failed to get container status \"524d7089f99abeb55b7b970b0deb0a9da1fd702952b69f665031e7bfd77d9a46\": rpc error: code = NotFound desc = could not find container \"524d7089f99abeb55b7b970b0deb0a9da1fd702952b69f665031e7bfd77d9a46\": container with ID starting with 524d7089f99abeb55b7b970b0deb0a9da1fd702952b69f665031e7bfd77d9a46 not found: ID does not exist"
Sep 16 18:57:14.412411 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:14.412391    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-4wjgg"]
Sep 16 18:57:14.424356 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:14.424335    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-4wjgg"]
Sep 16 18:57:14.820621 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:14.820582    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p"]
Sep 16 18:57:14.821079 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:14.821065    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66c61900-ff5c-499c-a9b2-395f6a806f7b" containerName="kserve-container"
Sep 16 18:57:14.821124 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:14.821082    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="66c61900-ff5c-499c-a9b2-395f6a806f7b" containerName="kserve-container"
Sep 16 18:57:14.821124 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:14.821093    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66c61900-ff5c-499c-a9b2-395f6a806f7b" containerName="kube-rbac-proxy"
Sep 16 18:57:14.821124 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:14.821102    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="66c61900-ff5c-499c-a9b2-395f6a806f7b" containerName="kube-rbac-proxy"
Sep 16 18:57:14.821216 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:14.821133    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66c61900-ff5c-499c-a9b2-395f6a806f7b" containerName="storage-initializer"
Sep 16 18:57:14.821216 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:14.821143    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="66c61900-ff5c-499c-a9b2-395f6a806f7b" containerName="storage-initializer"
Sep 16 18:57:14.821311 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:14.821216    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="66c61900-ff5c-499c-a9b2-395f6a806f7b" containerName="kserve-container"
Sep 16 18:57:14.821311 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:14.821253    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="66c61900-ff5c-499c-a9b2-395f6a806f7b" containerName="kube-rbac-proxy"
Sep 16 18:57:14.826119 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:14.826095    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p"
Sep 16 18:57:14.829253 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:14.829219    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
Sep 16 18:57:14.829373 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:14.829309    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 18:57:14.829373 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:14.829322    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:57:14.829821 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:14.829804    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
Sep 16 18:57:14.837637 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:14.837612    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p"]
Sep 16 18:57:14.910855 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:14.910820    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p\" (UID: \"7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p"
Sep 16 18:57:14.911038 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:14.910864    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p\" (UID: \"7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p"
Sep 16 18:57:14.911038 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:14.910885    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p\" (UID: \"7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p"
Sep 16 18:57:14.911038 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:14.910987    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p\" (UID: \"7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p"
Sep 16 18:57:14.911194 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:14.911037    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq99x\" (UniqueName: \"kubernetes.io/projected/7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007-kube-api-access-gq99x\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p\" (UID: \"7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p"
Sep 16 18:57:15.012062 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:15.012028    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p\" (UID: \"7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p"
Sep 16 18:57:15.012062 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:15.012061    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p\" (UID: \"7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p"
Sep 16 18:57:15.012362 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:15.012092    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p\" (UID: \"7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p"
Sep 16 18:57:15.012362 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:15.012284    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq99x\" (UniqueName: \"kubernetes.io/projected/7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007-kube-api-access-gq99x\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p\" (UID: \"7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p"
Sep 16 18:57:15.012469 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:15.012419    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p\" (UID: \"7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p"
Sep 16 18:57:15.012537 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:15.012517    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p\" (UID: \"7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p"
Sep 16 18:57:15.012600 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:57:15.012553    2586 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-serving-cert: secret "isvc-predictive-lightgbm-predictor-serving-cert" not found
Sep 16 18:57:15.012667 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:57:15.012612    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007-proxy-tls podName:7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007 nodeName:}" failed. No retries permitted until 2026-09-16 18:57:15.512597218 +0000 UTC m=+1010.956458772 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007-proxy-tls") pod "isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p" (UID: "7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007") : secret "isvc-predictive-lightgbm-predictor-serving-cert" not found
Sep 16 18:57:15.012883 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:15.012862    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p\" (UID: \"7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p"
Sep 16 18:57:15.012883 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:15.012875    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p\" (UID: \"7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p"
Sep 16 18:57:15.023856 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:15.023838    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq99x\" (UniqueName: \"kubernetes.io/projected/7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007-kube-api-access-gq99x\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p\" (UID: \"7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p"
Sep 16 18:57:15.093968 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:15.093906    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66c61900-ff5c-499c-a9b2-395f6a806f7b" path="/var/lib/kubelet/pods/66c61900-ff5c-499c-a9b2-395f6a806f7b/volumes"
Sep 16 18:57:15.516416 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:15.516386    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p\" (UID: \"7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p"
Sep 16 18:57:15.518800 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:15.518779    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p\" (UID: \"7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p"
Sep 16 18:57:15.736803 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:15.736765    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p"
Sep 16 18:57:16.024425 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:16.024399    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p"]
Sep 16 18:57:16.026339 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:57:16.026309    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ec1fa1e_f8da_45f3_b4d1_4f7877e1e007.slice/crio-a5de0170914fc528bc724888288f57981bb2cfdae24bc4bb678ae101ddc8c197 WatchSource:0}: Error finding container a5de0170914fc528bc724888288f57981bb2cfdae24bc4bb678ae101ddc8c197: Status 404 returned error can't find the container with id a5de0170914fc528bc724888288f57981bb2cfdae24bc4bb678ae101ddc8c197
Sep 16 18:57:16.398793 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:16.398753    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p" event={"ID":"7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007","Type":"ContainerStarted","Data":"6bd078c2b8e57c28b0f804d954dc475330780518b499e9f50af08222ff372c93"}
Sep 16 18:57:16.398969 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:16.398831    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p" event={"ID":"7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007","Type":"ContainerStarted","Data":"a5de0170914fc528bc724888288f57981bb2cfdae24bc4bb678ae101ddc8c197"}
Sep 16 18:57:20.418193 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:20.418158    2586 generic.go:356] "Generic (PLEG): container finished" podID="7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007" containerID="6bd078c2b8e57c28b0f804d954dc475330780518b499e9f50af08222ff372c93" exitCode=0
Sep 16 18:57:20.418585 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:20.418212    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p" event={"ID":"7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007","Type":"ContainerDied","Data":"6bd078c2b8e57c28b0f804d954dc475330780518b499e9f50af08222ff372c93"}
Sep 16 18:57:21.425218 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:21.425181    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p" event={"ID":"7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007","Type":"ContainerStarted","Data":"940bd5505b1d1ded9040429b6601644a4c02ce82347b0eb329db2437f4367a58"}
Sep 16 18:57:21.425602 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:21.425264    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p" event={"ID":"7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007","Type":"ContainerStarted","Data":"8f4aee9980bd759bb497f8a597573bb5bc4b4c1fd5f45ee29769476c5fb4ba6e"}
Sep 16 18:57:21.425602 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:21.425464    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p"
Sep 16 18:57:21.456065 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:21.456014    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p" podStartSLOduration=3.713294802 podStartE2EDuration="7.455997927s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.742703125s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:57:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:57:21.454025519 +0000 UTC m=+1016.897887108" watchObservedRunningTime="2026-09-16 18:57:21.455997927 +0000 UTC m=+1016.899859502"
Sep 16 18:57:22.430575 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:22.430535    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p"
Sep 16 18:57:22.432268 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:22.432220    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p" podUID="7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 18:57:23.434622 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:23.434575    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p" podUID="7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 18:57:28.439263 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:28.439222    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p"
Sep 16 18:57:28.439924 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:28.439895    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p" podUID="7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 18:57:38.440117 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:38.440074    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p" podUID="7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 18:57:48.440413 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:48.440371    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p" podUID="7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 18:57:58.440654 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:57:58.440613    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p" podUID="7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 18:58:08.440598 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:08.440550    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p" podUID="7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 18:58:18.439880 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:18.439792    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p" podUID="7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 18:58:28.440831 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:28.440788    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p" podUID="7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 18:58:29.491481 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:29.491450    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-kmhd5_99c64e29-e71f-4f90-ac1a-43e016a420bd/kserve-container/0.log"
Sep 16 18:58:29.748463 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:29.748392    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kmhd5"]
Sep 16 18:58:29.748657 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:29.748629    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kmhd5" podUID="99c64e29-e71f-4f90-ac1a-43e016a420bd" containerName="kserve-container" containerID="cri-o://ecf5ac2433b8d290cf8cc1ae872fbcaa986ad0a9c38d54c870d0b88d21ac96f2" gracePeriod=30
Sep 16 18:58:29.748657 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:29.748649    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kmhd5" podUID="99c64e29-e71f-4f90-ac1a-43e016a420bd" containerName="kube-rbac-proxy" containerID="cri-o://6deda5eef23f44796f663de63f3850101554efdc691d4b349a4df3a615ddda36" gracePeriod=30
Sep 16 18:58:29.991798 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:29.991773    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kmhd5"
Sep 16 18:58:30.043266 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:30.043169    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gcrwz\" (UniqueName: \"kubernetes.io/projected/99c64e29-e71f-4f90-ac1a-43e016a420bd-kube-api-access-gcrwz\") pod \"99c64e29-e71f-4f90-ac1a-43e016a420bd\" (UID: \"99c64e29-e71f-4f90-ac1a-43e016a420bd\") "
Sep 16 18:58:30.043266 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:30.043264    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99c64e29-e71f-4f90-ac1a-43e016a420bd-proxy-tls\") pod \"99c64e29-e71f-4f90-ac1a-43e016a420bd\" (UID: \"99c64e29-e71f-4f90-ac1a-43e016a420bd\") "
Sep 16 18:58:30.043503 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:30.043401    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99c64e29-e71f-4f90-ac1a-43e016a420bd-kube-rbac-proxy-sar-config\") pod \"99c64e29-e71f-4f90-ac1a-43e016a420bd\" (UID: \"99c64e29-e71f-4f90-ac1a-43e016a420bd\") "
Sep 16 18:58:30.043947 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:30.043916    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99c64e29-e71f-4f90-ac1a-43e016a420bd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "99c64e29-e71f-4f90-ac1a-43e016a420bd" (UID: "99c64e29-e71f-4f90-ac1a-43e016a420bd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:58:30.045853 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:30.045830    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99c64e29-e71f-4f90-ac1a-43e016a420bd-kube-api-access-gcrwz" (OuterVolumeSpecName: "kube-api-access-gcrwz") pod "99c64e29-e71f-4f90-ac1a-43e016a420bd" (UID: "99c64e29-e71f-4f90-ac1a-43e016a420bd"). InnerVolumeSpecName "kube-api-access-gcrwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:58:30.045955 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:30.045938    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99c64e29-e71f-4f90-ac1a-43e016a420bd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "99c64e29-e71f-4f90-ac1a-43e016a420bd" (UID: "99c64e29-e71f-4f90-ac1a-43e016a420bd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:58:30.136432 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:30.136390    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s2xwx"]
Sep 16 18:58:30.136735 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:30.136721    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99c64e29-e71f-4f90-ac1a-43e016a420bd" containerName="kube-rbac-proxy"
Sep 16 18:58:30.136735 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:30.136734    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="99c64e29-e71f-4f90-ac1a-43e016a420bd" containerName="kube-rbac-proxy"
Sep 16 18:58:30.136860 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:30.136745    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99c64e29-e71f-4f90-ac1a-43e016a420bd" containerName="kserve-container"
Sep 16 18:58:30.136860 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:30.136752    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="99c64e29-e71f-4f90-ac1a-43e016a420bd" containerName="kserve-container"
Sep 16 18:58:30.136860 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:30.136811    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="99c64e29-e71f-4f90-ac1a-43e016a420bd" containerName="kube-rbac-proxy"
Sep 16 18:58:30.136860 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:30.136819    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="99c64e29-e71f-4f90-ac1a-43e016a420bd" containerName="kserve-container"
Sep 16 18:58:30.139883 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:30.139862    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s2xwx"
Sep 16 18:58:30.143961 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:30.143935    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a0d46bc-616b-40ea-8e1a-686bdb35d568-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-s2xwx\" (UID: \"7a0d46bc-616b-40ea-8e1a-686bdb35d568\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s2xwx"
Sep 16 18:58:30.144066 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:30.143975    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a0d46bc-616b-40ea-8e1a-686bdb35d568-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-s2xwx\" (UID: \"7a0d46bc-616b-40ea-8e1a-686bdb35d568\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s2xwx"
Sep 16 18:58:30.144066 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:30.144007    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr8n9\" (UniqueName: \"kubernetes.io/projected/7a0d46bc-616b-40ea-8e1a-686bdb35d568-kube-api-access-qr8n9\") pod \"isvc-autogluon-v1-predictor-574b5b7495-s2xwx\" (UID: \"7a0d46bc-616b-40ea-8e1a-686bdb35d568\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s2xwx"
Sep 16 18:58:30.144183 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:30.144092    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a0d46bc-616b-40ea-8e1a-686bdb35d568-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-s2xwx\" (UID: \"7a0d46bc-616b-40ea-8e1a-686bdb35d568\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s2xwx"
Sep 16 18:58:30.144183 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:30.144127    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a0d46bc-616b-40ea-8e1a-686bdb35d568-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-s2xwx\" (UID: \"7a0d46bc-616b-40ea-8e1a-686bdb35d568\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s2xwx"
Sep 16 18:58:30.144280 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:30.144202    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcrwz\" (UniqueName: \"kubernetes.io/projected/99c64e29-e71f-4f90-ac1a-43e016a420bd-kube-api-access-gcrwz\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 18:58:30.144280 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:30.144213    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99c64e29-e71f-4f90-ac1a-43e016a420bd-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 18:58:30.144280 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:30.144223    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99c64e29-e71f-4f90-ac1a-43e016a420bd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 18:58:30.147405 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:30.147385    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Sep 16 18:58:30.148359 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:30.148342    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Sep 16 18:58:30.158159 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:30.158134    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s2xwx"]
Sep 16 18:58:30.244807 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:30.244769    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a0d46bc-616b-40ea-8e1a-686bdb35d568-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-s2xwx\" (UID: \"7a0d46bc-616b-40ea-8e1a-686bdb35d568\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s2xwx"
Sep 16 18:58:30.244994 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:30.244826    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr8n9\" (UniqueName: \"kubernetes.io/projected/7a0d46bc-616b-40ea-8e1a-686bdb35d568-kube-api-access-qr8n9\") pod \"isvc-autogluon-v1-predictor-574b5b7495-s2xwx\" (UID: \"7a0d46bc-616b-40ea-8e1a-686bdb35d568\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s2xwx"
Sep 16 18:58:30.244994 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:30.244886    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a0d46bc-616b-40ea-8e1a-686bdb35d568-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-s2xwx\" (UID: \"7a0d46bc-616b-40ea-8e1a-686bdb35d568\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s2xwx"
Sep 16 18:58:30.244994 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:30.244914    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a0d46bc-616b-40ea-8e1a-686bdb35d568-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-s2xwx\" (UID: \"7a0d46bc-616b-40ea-8e1a-686bdb35d568\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s2xwx"
Sep 16 18:58:30.244994 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:30.244968    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a0d46bc-616b-40ea-8e1a-686bdb35d568-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-s2xwx\" (UID: \"7a0d46bc-616b-40ea-8e1a-686bdb35d568\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s2xwx"
Sep 16 18:58:30.245203 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:58:30.245025    2586 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v1-predictor-serving-cert: secret "isvc-autogluon-v1-predictor-serving-cert" not found
Sep 16 18:58:30.245203 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:58:30.245097    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a0d46bc-616b-40ea-8e1a-686bdb35d568-proxy-tls podName:7a0d46bc-616b-40ea-8e1a-686bdb35d568 nodeName:}" failed. No retries permitted until 2026-09-16 18:58:30.745074831 +0000 UTC m=+1086.188936409 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7a0d46bc-616b-40ea-8e1a-686bdb35d568-proxy-tls") pod "isvc-autogluon-v1-predictor-574b5b7495-s2xwx" (UID: "7a0d46bc-616b-40ea-8e1a-686bdb35d568") : secret "isvc-autogluon-v1-predictor-serving-cert" not found
Sep 16 18:58:30.245421 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:30.245399    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a0d46bc-616b-40ea-8e1a-686bdb35d568-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-s2xwx\" (UID: \"7a0d46bc-616b-40ea-8e1a-686bdb35d568\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s2xwx"
Sep 16 18:58:30.245598 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:30.245577    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a0d46bc-616b-40ea-8e1a-686bdb35d568-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-s2xwx\" (UID: \"7a0d46bc-616b-40ea-8e1a-686bdb35d568\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s2xwx"
Sep 16 18:58:30.245655 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:30.245594    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a0d46bc-616b-40ea-8e1a-686bdb35d568-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-s2xwx\" (UID: \"7a0d46bc-616b-40ea-8e1a-686bdb35d568\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s2xwx"
Sep 16 18:58:30.256987 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:30.256961    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr8n9\" (UniqueName: \"kubernetes.io/projected/7a0d46bc-616b-40ea-8e1a-686bdb35d568-kube-api-access-qr8n9\") pod \"isvc-autogluon-v1-predictor-574b5b7495-s2xwx\" (UID: \"7a0d46bc-616b-40ea-8e1a-686bdb35d568\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s2xwx"
Sep 16 18:58:30.748163 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:30.748128    2586 generic.go:356] "Generic (PLEG): container finished" podID="99c64e29-e71f-4f90-ac1a-43e016a420bd" containerID="6deda5eef23f44796f663de63f3850101554efdc691d4b349a4df3a615ddda36" exitCode=2
Sep 16 18:58:30.748163 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:30.748155    2586 generic.go:356] "Generic (PLEG): container finished" podID="99c64e29-e71f-4f90-ac1a-43e016a420bd" containerID="ecf5ac2433b8d290cf8cc1ae872fbcaa986ad0a9c38d54c870d0b88d21ac96f2" exitCode=2
Sep 16 18:58:30.748695 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:30.748167    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kmhd5" event={"ID":"99c64e29-e71f-4f90-ac1a-43e016a420bd","Type":"ContainerDied","Data":"6deda5eef23f44796f663de63f3850101554efdc691d4b349a4df3a615ddda36"}
Sep 16 18:58:30.748695 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:30.748189    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kmhd5"
Sep 16 18:58:30.748695 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:30.748258    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kmhd5" event={"ID":"99c64e29-e71f-4f90-ac1a-43e016a420bd","Type":"ContainerDied","Data":"ecf5ac2433b8d290cf8cc1ae872fbcaa986ad0a9c38d54c870d0b88d21ac96f2"}
Sep 16 18:58:30.748695 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:30.748296    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kmhd5" event={"ID":"99c64e29-e71f-4f90-ac1a-43e016a420bd","Type":"ContainerDied","Data":"d68ad0305296aa14474ea33184d6ee45d09eac7e1d5758db11803011bbddd0e3"}
Sep 16 18:58:30.748695 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:30.748322    2586 scope.go:117] "RemoveContainer" containerID="6deda5eef23f44796f663de63f3850101554efdc691d4b349a4df3a615ddda36"
Sep 16 18:58:30.748695 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:30.748615    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a0d46bc-616b-40ea-8e1a-686bdb35d568-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-s2xwx\" (UID: \"7a0d46bc-616b-40ea-8e1a-686bdb35d568\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s2xwx"
Sep 16 18:58:30.751159 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:30.751133    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a0d46bc-616b-40ea-8e1a-686bdb35d568-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-s2xwx\" (UID: \"7a0d46bc-616b-40ea-8e1a-686bdb35d568\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s2xwx"
Sep 16 18:58:30.756814 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:30.756794    2586 scope.go:117] "RemoveContainer" containerID="ecf5ac2433b8d290cf8cc1ae872fbcaa986ad0a9c38d54c870d0b88d21ac96f2"
Sep 16 18:58:30.765434 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:30.765420    2586 scope.go:117] "RemoveContainer" containerID="6deda5eef23f44796f663de63f3850101554efdc691d4b349a4df3a615ddda36"
Sep 16 18:58:30.765686 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:58:30.765668    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6deda5eef23f44796f663de63f3850101554efdc691d4b349a4df3a615ddda36\": container with ID starting with 6deda5eef23f44796f663de63f3850101554efdc691d4b349a4df3a615ddda36 not found: ID does not exist" containerID="6deda5eef23f44796f663de63f3850101554efdc691d4b349a4df3a615ddda36"
Sep 16 18:58:30.765732 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:30.765694    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6deda5eef23f44796f663de63f3850101554efdc691d4b349a4df3a615ddda36"} err="failed to get container status \"6deda5eef23f44796f663de63f3850101554efdc691d4b349a4df3a615ddda36\": rpc error: code = NotFound desc = could not find container \"6deda5eef23f44796f663de63f3850101554efdc691d4b349a4df3a615ddda36\": container with ID starting with 6deda5eef23f44796f663de63f3850101554efdc691d4b349a4df3a615ddda36 not found: ID does not exist"
Sep 16 18:58:30.765732 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:30.765710    2586 scope.go:117] "RemoveContainer" containerID="ecf5ac2433b8d290cf8cc1ae872fbcaa986ad0a9c38d54c870d0b88d21ac96f2"
Sep 16 18:58:30.765966 ip-10-0-129-190 kubenswrapper[2586]: E0916 18:58:30.765949    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecf5ac2433b8d290cf8cc1ae872fbcaa986ad0a9c38d54c870d0b88d21ac96f2\": container with ID starting with ecf5ac2433b8d290cf8cc1ae872fbcaa986ad0a9c38d54c870d0b88d21ac96f2 not found: ID does not exist" containerID="ecf5ac2433b8d290cf8cc1ae872fbcaa986ad0a9c38d54c870d0b88d21ac96f2"
Sep 16 18:58:30.766007 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:30.765972    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecf5ac2433b8d290cf8cc1ae872fbcaa986ad0a9c38d54c870d0b88d21ac96f2"} err="failed to get container status \"ecf5ac2433b8d290cf8cc1ae872fbcaa986ad0a9c38d54c870d0b88d21ac96f2\": rpc error: code = NotFound desc = could not find container \"ecf5ac2433b8d290cf8cc1ae872fbcaa986ad0a9c38d54c870d0b88d21ac96f2\": container with ID starting with ecf5ac2433b8d290cf8cc1ae872fbcaa986ad0a9c38d54c870d0b88d21ac96f2 not found: ID does not exist"
Sep 16 18:58:30.766007 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:30.765987    2586 scope.go:117] "RemoveContainer" containerID="6deda5eef23f44796f663de63f3850101554efdc691d4b349a4df3a615ddda36"
Sep 16 18:58:30.766203 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:30.766186    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6deda5eef23f44796f663de63f3850101554efdc691d4b349a4df3a615ddda36"} err="failed to get container status \"6deda5eef23f44796f663de63f3850101554efdc691d4b349a4df3a615ddda36\": rpc error: code = NotFound desc = could not find container \"6deda5eef23f44796f663de63f3850101554efdc691d4b349a4df3a615ddda36\": container with ID starting with 6deda5eef23f44796f663de63f3850101554efdc691d4b349a4df3a615ddda36 not found: ID does not exist"
Sep 16 18:58:30.766261 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:30.766203    2586 scope.go:117] "RemoveContainer" containerID="ecf5ac2433b8d290cf8cc1ae872fbcaa986ad0a9c38d54c870d0b88d21ac96f2"
Sep 16 18:58:30.766467 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:30.766448    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecf5ac2433b8d290cf8cc1ae872fbcaa986ad0a9c38d54c870d0b88d21ac96f2"} err="failed to get container status \"ecf5ac2433b8d290cf8cc1ae872fbcaa986ad0a9c38d54c870d0b88d21ac96f2\": rpc error: code = NotFound desc = could not find container \"ecf5ac2433b8d290cf8cc1ae872fbcaa986ad0a9c38d54c870d0b88d21ac96f2\": container with ID starting with ecf5ac2433b8d290cf8cc1ae872fbcaa986ad0a9c38d54c870d0b88d21ac96f2 not found: ID does not exist"
Sep 16 18:58:30.781925 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:30.781900    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kmhd5"]
Sep 16 18:58:30.785549 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:30.785525    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kmhd5"]
Sep 16 18:58:31.050071 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:31.049978    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s2xwx"
Sep 16 18:58:31.092887 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:31.092852    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99c64e29-e71f-4f90-ac1a-43e016a420bd" path="/var/lib/kubelet/pods/99c64e29-e71f-4f90-ac1a-43e016a420bd/volumes"
Sep 16 18:58:31.336941 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:31.336916    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s2xwx"]
Sep 16 18:58:31.339531 ip-10-0-129-190 kubenswrapper[2586]: W0916 18:58:31.339507    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a0d46bc_616b_40ea_8e1a_686bdb35d568.slice/crio-3098c0a9ea387c17ad5f8e00f41a4443c4d8621f0e59ffa62909e63bc3e5e675 WatchSource:0}: Error finding container 3098c0a9ea387c17ad5f8e00f41a4443c4d8621f0e59ffa62909e63bc3e5e675: Status 404 returned error can't find the container with id 3098c0a9ea387c17ad5f8e00f41a4443c4d8621f0e59ffa62909e63bc3e5e675
Sep 16 18:58:31.755852 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:31.755818    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s2xwx" event={"ID":"7a0d46bc-616b-40ea-8e1a-686bdb35d568","Type":"ContainerStarted","Data":"75975e3d77d585d4791372d139416965a682ac538989fdad0eca948ebeac9141"}
Sep 16 18:58:31.756264 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:31.755870    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s2xwx" event={"ID":"7a0d46bc-616b-40ea-8e1a-686bdb35d568","Type":"ContainerStarted","Data":"3098c0a9ea387c17ad5f8e00f41a4443c4d8621f0e59ffa62909e63bc3e5e675"}
Sep 16 18:58:38.440390 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:38.440359    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p"
Sep 16 18:58:44.920778 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:44.920743    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p"]
Sep 16 18:58:44.921170 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:44.920991    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p" podUID="7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007" containerName="kserve-container" containerID="cri-o://8f4aee9980bd759bb497f8a597573bb5bc4b4c1fd5f45ee29769476c5fb4ba6e" gracePeriod=30
Sep 16 18:58:44.921170 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:44.921063    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p" podUID="7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007" containerName="kube-rbac-proxy" containerID="cri-o://940bd5505b1d1ded9040429b6601644a4c02ce82347b0eb329db2437f4367a58" gracePeriod=30
Sep 16 18:58:45.822992 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:45.822959    2586 generic.go:356] "Generic (PLEG): container finished" podID="7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007" containerID="940bd5505b1d1ded9040429b6601644a4c02ce82347b0eb329db2437f4367a58" exitCode=2
Sep 16 18:58:45.823167 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:45.823039    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p" event={"ID":"7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007","Type":"ContainerDied","Data":"940bd5505b1d1ded9040429b6601644a4c02ce82347b0eb329db2437f4367a58"}
Sep 16 18:58:48.435502 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:48.435449    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p" podUID="7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Sep 16 18:58:48.440606 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:48.440577    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p" podUID="7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 16 18:58:49.843301 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:49.843187    2586 generic.go:356] "Generic (PLEG): container finished" podID="7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007" containerID="8f4aee9980bd759bb497f8a597573bb5bc4b4c1fd5f45ee29769476c5fb4ba6e" exitCode=0
Sep 16 18:58:49.843301 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:49.843263    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p" event={"ID":"7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007","Type":"ContainerDied","Data":"8f4aee9980bd759bb497f8a597573bb5bc4b4c1fd5f45ee29769476c5fb4ba6e"}
Sep 16 18:58:50.076166 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:50.076141    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p"
Sep 16 18:58:50.109434 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:50.109406    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007-proxy-tls\") pod \"7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007\" (UID: \"7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007\") "
Sep 16 18:58:50.109623 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:50.109482    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007-kserve-provision-location\") pod \"7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007\" (UID: \"7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007\") "
Sep 16 18:58:50.109869 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:50.109819    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007" (UID: "7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:58:50.110282 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:50.109977    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007-cabundle-cert\") pod \"7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007\" (UID: \"7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007\") "
Sep 16 18:58:50.110282 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:50.110129    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gq99x\" (UniqueName: \"kubernetes.io/projected/7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007-kube-api-access-gq99x\") pod \"7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007\" (UID: \"7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007\") "
Sep 16 18:58:50.110282 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:50.110171    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007-kube-rbac-proxy-sar-config\") pod \"7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007\" (UID: \"7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007\") "
Sep 16 18:58:50.110491 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:50.110461    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007" (UID: "7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:58:50.110491 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:50.110462    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 18:58:50.110975 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:50.110914    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007" (UID: "7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:58:50.112860 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:50.112833    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007-kube-api-access-gq99x" (OuterVolumeSpecName: "kube-api-access-gq99x") pod "7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007" (UID: "7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007"). InnerVolumeSpecName "kube-api-access-gq99x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:58:50.113008 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:50.112986    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007" (UID: "7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:58:50.211788 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:50.211747    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 18:58:50.211788 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:50.211785    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq99x\" (UniqueName: \"kubernetes.io/projected/7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007-kube-api-access-gq99x\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 18:58:50.211976 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:50.211801    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 18:58:50.211976 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:50.211816    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 18:58:50.850031 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:50.849996    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p" event={"ID":"7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007","Type":"ContainerDied","Data":"a5de0170914fc528bc724888288f57981bb2cfdae24bc4bb678ae101ddc8c197"}
Sep 16 18:58:50.850440 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:50.850061    2586 scope.go:117] "RemoveContainer" containerID="940bd5505b1d1ded9040429b6601644a4c02ce82347b0eb329db2437f4367a58"
Sep 16 18:58:50.850440 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:50.850011    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p"
Sep 16 18:58:50.859084 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:50.859064    2586 scope.go:117] "RemoveContainer" containerID="8f4aee9980bd759bb497f8a597573bb5bc4b4c1fd5f45ee29769476c5fb4ba6e"
Sep 16 18:58:50.866482 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:50.866465    2586 scope.go:117] "RemoveContainer" containerID="6bd078c2b8e57c28b0f804d954dc475330780518b499e9f50af08222ff372c93"
Sep 16 18:58:50.880173 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:50.880151    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p"]
Sep 16 18:58:50.889967 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:50.889926    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-jln7p"]
Sep 16 18:58:51.092593 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:58:51.092555    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007" path="/var/lib/kubelet/pods/7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007/volumes"
Sep 16 18:59:00.906524 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:59:00.906491    2586 generic.go:356] "Generic (PLEG): container finished" podID="7a0d46bc-616b-40ea-8e1a-686bdb35d568" containerID="75975e3d77d585d4791372d139416965a682ac538989fdad0eca948ebeac9141" exitCode=0
Sep 16 18:59:00.906884 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:59:00.906564    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s2xwx" event={"ID":"7a0d46bc-616b-40ea-8e1a-686bdb35d568","Type":"ContainerDied","Data":"75975e3d77d585d4791372d139416965a682ac538989fdad0eca948ebeac9141"}
Sep 16 18:59:00.907955 ip-10-0-129-190 kubenswrapper[2586]: I0916 18:59:00.907937    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:01:03.546107 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:03.546073    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s2xwx" event={"ID":"7a0d46bc-616b-40ea-8e1a-686bdb35d568","Type":"ContainerStarted","Data":"0b50cb744a83c2b264a39d88f71f8eb47b2066463058e710f343ea7ea49b53a9"}
Sep 16 19:01:03.546625 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:03.546168    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s2xwx" event={"ID":"7a0d46bc-616b-40ea-8e1a-686bdb35d568","Type":"ContainerStarted","Data":"5bce5bda4cf40b339a0a42a6a9bd65350bd818f637f4d7e8267c681306a439b1"}
Sep 16 19:01:03.546625 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:03.546345    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s2xwx"
Sep 16 19:01:04.552332 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:04.552297    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s2xwx"
Sep 16 19:01:04.553464 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:04.553438    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s2xwx" podUID="7a0d46bc-616b-40ea-8e1a-686bdb35d568" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 16 19:01:05.556782 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:05.556740    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s2xwx" podUID="7a0d46bc-616b-40ea-8e1a-686bdb35d568" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 16 19:01:10.561645 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:10.561610    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s2xwx"
Sep 16 19:01:10.570422 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:10.562168    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s2xwx" podUID="7a0d46bc-616b-40ea-8e1a-686bdb35d568" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 16 19:01:10.614964 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:10.614913    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s2xwx" podStartSLOduration=9.168078693 podStartE2EDuration="2m40.614897475s" totalImagesPullingTime="2m2.182212954s" totalInitContainerRuntime="29.264605828s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:58:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:01:03.618469686 +0000 UTC m=+1239.062331262" watchObservedRunningTime="2026-09-16 19:01:10.614897475 +0000 UTC m=+1246.058759050"
Sep 16 19:01:20.563063 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:20.562983    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s2xwx"
Sep 16 19:01:36.517196 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:36.517163    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s2xwx"]
Sep 16 19:01:36.517676 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:36.517513    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s2xwx" podUID="7a0d46bc-616b-40ea-8e1a-686bdb35d568" containerName="kserve-container" containerID="cri-o://5bce5bda4cf40b339a0a42a6a9bd65350bd818f637f4d7e8267c681306a439b1" gracePeriod=30
Sep 16 19:01:36.517676 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:36.517530    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s2xwx" podUID="7a0d46bc-616b-40ea-8e1a-686bdb35d568" containerName="kube-rbac-proxy" containerID="cri-o://0b50cb744a83c2b264a39d88f71f8eb47b2066463058e710f343ea7ea49b53a9" gracePeriod=30
Sep 16 19:01:36.700248 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:36.700183    2586 generic.go:356] "Generic (PLEG): container finished" podID="7a0d46bc-616b-40ea-8e1a-686bdb35d568" containerID="0b50cb744a83c2b264a39d88f71f8eb47b2066463058e710f343ea7ea49b53a9" exitCode=2
Sep 16 19:01:36.700402 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:36.700276    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s2xwx" event={"ID":"7a0d46bc-616b-40ea-8e1a-686bdb35d568","Type":"ContainerDied","Data":"0b50cb744a83c2b264a39d88f71f8eb47b2066463058e710f343ea7ea49b53a9"}
Sep 16 19:01:37.963600 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:37.963576    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s2xwx"
Sep 16 19:01:38.118718 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:38.118675    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a0d46bc-616b-40ea-8e1a-686bdb35d568-kube-rbac-proxy-sar-config\") pod \"7a0d46bc-616b-40ea-8e1a-686bdb35d568\" (UID: \"7a0d46bc-616b-40ea-8e1a-686bdb35d568\") "
Sep 16 19:01:38.118878 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:38.118733    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a0d46bc-616b-40ea-8e1a-686bdb35d568-cabundle-cert\") pod \"7a0d46bc-616b-40ea-8e1a-686bdb35d568\" (UID: \"7a0d46bc-616b-40ea-8e1a-686bdb35d568\") "
Sep 16 19:01:38.118878 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:38.118763    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a0d46bc-616b-40ea-8e1a-686bdb35d568-proxy-tls\") pod \"7a0d46bc-616b-40ea-8e1a-686bdb35d568\" (UID: \"7a0d46bc-616b-40ea-8e1a-686bdb35d568\") "
Sep 16 19:01:38.118963 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:38.118883    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a0d46bc-616b-40ea-8e1a-686bdb35d568-kserve-provision-location\") pod \"7a0d46bc-616b-40ea-8e1a-686bdb35d568\" (UID: \"7a0d46bc-616b-40ea-8e1a-686bdb35d568\") "
Sep 16 19:01:38.118963 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:38.118940    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qr8n9\" (UniqueName: \"kubernetes.io/projected/7a0d46bc-616b-40ea-8e1a-686bdb35d568-kube-api-access-qr8n9\") pod \"7a0d46bc-616b-40ea-8e1a-686bdb35d568\" (UID: \"7a0d46bc-616b-40ea-8e1a-686bdb35d568\") "
Sep 16 19:01:38.119163 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:38.119061    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a0d46bc-616b-40ea-8e1a-686bdb35d568-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7a0d46bc-616b-40ea-8e1a-686bdb35d568" (UID: "7a0d46bc-616b-40ea-8e1a-686bdb35d568"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:01:38.119322 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:38.119174    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a0d46bc-616b-40ea-8e1a-686bdb35d568-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7a0d46bc-616b-40ea-8e1a-686bdb35d568" (UID: "7a0d46bc-616b-40ea-8e1a-686bdb35d568"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:01:38.119322 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:38.119290    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a0d46bc-616b-40ea-8e1a-686bdb35d568-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:01:38.119322 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:38.119311    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a0d46bc-616b-40ea-8e1a-686bdb35d568-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:01:38.120860 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:38.120835    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a0d46bc-616b-40ea-8e1a-686bdb35d568-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7a0d46bc-616b-40ea-8e1a-686bdb35d568" (UID: "7a0d46bc-616b-40ea-8e1a-686bdb35d568"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:01:38.121046 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:38.120950    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a0d46bc-616b-40ea-8e1a-686bdb35d568-kube-api-access-qr8n9" (OuterVolumeSpecName: "kube-api-access-qr8n9") pod "7a0d46bc-616b-40ea-8e1a-686bdb35d568" (UID: "7a0d46bc-616b-40ea-8e1a-686bdb35d568"). InnerVolumeSpecName "kube-api-access-qr8n9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:01:38.123212 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:38.123183    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a0d46bc-616b-40ea-8e1a-686bdb35d568-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7a0d46bc-616b-40ea-8e1a-686bdb35d568" (UID: "7a0d46bc-616b-40ea-8e1a-686bdb35d568"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:38.220548 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:38.220497    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a0d46bc-616b-40ea-8e1a-686bdb35d568-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:01:38.220548 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:38.220543    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a0d46bc-616b-40ea-8e1a-686bdb35d568-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:01:38.220548 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:38.220555    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qr8n9\" (UniqueName: \"kubernetes.io/projected/7a0d46bc-616b-40ea-8e1a-686bdb35d568-kube-api-access-qr8n9\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:01:38.711838 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:38.711803    2586 generic.go:356] "Generic (PLEG): container finished" podID="7a0d46bc-616b-40ea-8e1a-686bdb35d568" containerID="5bce5bda4cf40b339a0a42a6a9bd65350bd818f637f4d7e8267c681306a439b1" exitCode=0
Sep 16 19:01:38.711997 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:38.711884    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s2xwx"
Sep 16 19:01:38.711997 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:38.711886    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s2xwx" event={"ID":"7a0d46bc-616b-40ea-8e1a-686bdb35d568","Type":"ContainerDied","Data":"5bce5bda4cf40b339a0a42a6a9bd65350bd818f637f4d7e8267c681306a439b1"}
Sep 16 19:01:38.711997 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:38.711955    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s2xwx" event={"ID":"7a0d46bc-616b-40ea-8e1a-686bdb35d568","Type":"ContainerDied","Data":"3098c0a9ea387c17ad5f8e00f41a4443c4d8621f0e59ffa62909e63bc3e5e675"}
Sep 16 19:01:38.711997 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:38.711993    2586 scope.go:117] "RemoveContainer" containerID="0b50cb744a83c2b264a39d88f71f8eb47b2066463058e710f343ea7ea49b53a9"
Sep 16 19:01:38.720657 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:38.720639    2586 scope.go:117] "RemoveContainer" containerID="5bce5bda4cf40b339a0a42a6a9bd65350bd818f637f4d7e8267c681306a439b1"
Sep 16 19:01:38.728076 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:38.728059    2586 scope.go:117] "RemoveContainer" containerID="75975e3d77d585d4791372d139416965a682ac538989fdad0eca948ebeac9141"
Sep 16 19:01:38.735023 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:38.735006    2586 scope.go:117] "RemoveContainer" containerID="0b50cb744a83c2b264a39d88f71f8eb47b2066463058e710f343ea7ea49b53a9"
Sep 16 19:01:38.735332 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:01:38.735312    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b50cb744a83c2b264a39d88f71f8eb47b2066463058e710f343ea7ea49b53a9\": container with ID starting with 0b50cb744a83c2b264a39d88f71f8eb47b2066463058e710f343ea7ea49b53a9 not found: ID does not exist" containerID="0b50cb744a83c2b264a39d88f71f8eb47b2066463058e710f343ea7ea49b53a9"
Sep 16 19:01:38.735396 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:38.735341    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b50cb744a83c2b264a39d88f71f8eb47b2066463058e710f343ea7ea49b53a9"} err="failed to get container status \"0b50cb744a83c2b264a39d88f71f8eb47b2066463058e710f343ea7ea49b53a9\": rpc error: code = NotFound desc = could not find container \"0b50cb744a83c2b264a39d88f71f8eb47b2066463058e710f343ea7ea49b53a9\": container with ID starting with 0b50cb744a83c2b264a39d88f71f8eb47b2066463058e710f343ea7ea49b53a9 not found: ID does not exist"
Sep 16 19:01:38.735396 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:38.735357    2586 scope.go:117] "RemoveContainer" containerID="5bce5bda4cf40b339a0a42a6a9bd65350bd818f637f4d7e8267c681306a439b1"
Sep 16 19:01:38.735606 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:01:38.735586    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bce5bda4cf40b339a0a42a6a9bd65350bd818f637f4d7e8267c681306a439b1\": container with ID starting with 5bce5bda4cf40b339a0a42a6a9bd65350bd818f637f4d7e8267c681306a439b1 not found: ID does not exist" containerID="5bce5bda4cf40b339a0a42a6a9bd65350bd818f637f4d7e8267c681306a439b1"
Sep 16 19:01:38.735657 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:38.735614    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bce5bda4cf40b339a0a42a6a9bd65350bd818f637f4d7e8267c681306a439b1"} err="failed to get container status \"5bce5bda4cf40b339a0a42a6a9bd65350bd818f637f4d7e8267c681306a439b1\": rpc error: code = NotFound desc = could not find container \"5bce5bda4cf40b339a0a42a6a9bd65350bd818f637f4d7e8267c681306a439b1\": container with ID starting with 5bce5bda4cf40b339a0a42a6a9bd65350bd818f637f4d7e8267c681306a439b1 not found: ID does not exist"
Sep 16 19:01:38.735657 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:38.735630    2586 scope.go:117] "RemoveContainer" containerID="75975e3d77d585d4791372d139416965a682ac538989fdad0eca948ebeac9141"
Sep 16 19:01:38.735852 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:01:38.735838    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75975e3d77d585d4791372d139416965a682ac538989fdad0eca948ebeac9141\": container with ID starting with 75975e3d77d585d4791372d139416965a682ac538989fdad0eca948ebeac9141 not found: ID does not exist" containerID="75975e3d77d585d4791372d139416965a682ac538989fdad0eca948ebeac9141"
Sep 16 19:01:38.735898 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:38.735855    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75975e3d77d585d4791372d139416965a682ac538989fdad0eca948ebeac9141"} err="failed to get container status \"75975e3d77d585d4791372d139416965a682ac538989fdad0eca948ebeac9141\": rpc error: code = NotFound desc = could not find container \"75975e3d77d585d4791372d139416965a682ac538989fdad0eca948ebeac9141\": container with ID starting with 75975e3d77d585d4791372d139416965a682ac538989fdad0eca948ebeac9141 not found: ID does not exist"
Sep 16 19:01:38.741687 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:38.741655    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s2xwx"]
Sep 16 19:01:38.744038 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:38.744019    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-s2xwx"]
Sep 16 19:01:39.091626 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:39.091549    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a0d46bc-616b-40ea-8e1a-686bdb35d568" path="/var/lib/kubelet/pods/7a0d46bc-616b-40ea-8e1a-686bdb35d568/volumes"
Sep 16 19:01:41.840565 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:41.840528    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ztppk"]
Sep 16 19:01:41.841048 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:41.840854    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a0d46bc-616b-40ea-8e1a-686bdb35d568" containerName="storage-initializer"
Sep 16 19:01:41.841048 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:41.840865    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a0d46bc-616b-40ea-8e1a-686bdb35d568" containerName="storage-initializer"
Sep 16 19:01:41.841048 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:41.840874    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a0d46bc-616b-40ea-8e1a-686bdb35d568" containerName="kserve-container"
Sep 16 19:01:41.841048 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:41.840879    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a0d46bc-616b-40ea-8e1a-686bdb35d568" containerName="kserve-container"
Sep 16 19:01:41.841048 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:41.840888    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a0d46bc-616b-40ea-8e1a-686bdb35d568" containerName="kube-rbac-proxy"
Sep 16 19:01:41.841048 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:41.840894    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a0d46bc-616b-40ea-8e1a-686bdb35d568" containerName="kube-rbac-proxy"
Sep 16 19:01:41.841048 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:41.840901    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007" containerName="kserve-container"
Sep 16 19:01:41.841048 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:41.840906    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007" containerName="kserve-container"
Sep 16 19:01:41.841048 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:41.840918    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007" containerName="storage-initializer"
Sep 16 19:01:41.841048 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:41.840924    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007" containerName="storage-initializer"
Sep 16 19:01:41.841048 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:41.840937    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007" containerName="kube-rbac-proxy"
Sep 16 19:01:41.841048 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:41.840942    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007" containerName="kube-rbac-proxy"
Sep 16 19:01:41.841048 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:41.840993    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a0d46bc-616b-40ea-8e1a-686bdb35d568" containerName="kserve-container"
Sep 16 19:01:41.841048 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:41.841001    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007" containerName="kube-rbac-proxy"
Sep 16 19:01:41.841048 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:41.841007    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a0d46bc-616b-40ea-8e1a-686bdb35d568" containerName="kube-rbac-proxy"
Sep 16 19:01:41.841048 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:41.841016    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ec1fa1e-f8da-45f3-b4d1-4f7877e1e007" containerName="kserve-container"
Sep 16 19:01:41.844469 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:41.844447    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ztppk"
Sep 16 19:01:41.848414 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:41.848392    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:01:41.848529 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:41.848392    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:01:41.848529 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:41.848392    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Sep 16 19:01:41.848650 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:41.848556    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:01:41.848748 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:41.848733    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:01:41.848820 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:41.848745    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Sep 16 19:01:41.849352 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:41.849335    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-pv9ct\""
Sep 16 19:01:41.864266 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:41.864209    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ztppk"]
Sep 16 19:01:41.951368 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:41.951335    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8snrb\" (UniqueName: \"kubernetes.io/projected/886c934e-3162-4331-af40-b52cd04091ad-kube-api-access-8snrb\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-ztppk\" (UID: \"886c934e-3162-4331-af40-b52cd04091ad\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ztppk"
Sep 16 19:01:41.951561 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:41.951382    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/886c934e-3162-4331-af40-b52cd04091ad-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-ztppk\" (UID: \"886c934e-3162-4331-af40-b52cd04091ad\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ztppk"
Sep 16 19:01:41.951561 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:41.951417    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/886c934e-3162-4331-af40-b52cd04091ad-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-ztppk\" (UID: \"886c934e-3162-4331-af40-b52cd04091ad\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ztppk"
Sep 16 19:01:41.951561 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:41.951459    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/886c934e-3162-4331-af40-b52cd04091ad-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-ztppk\" (UID: \"886c934e-3162-4331-af40-b52cd04091ad\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ztppk"
Sep 16 19:01:41.951561 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:41.951479    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/886c934e-3162-4331-af40-b52cd04091ad-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-ztppk\" (UID: \"886c934e-3162-4331-af40-b52cd04091ad\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ztppk"
Sep 16 19:01:42.052514 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:42.052473    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/886c934e-3162-4331-af40-b52cd04091ad-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-ztppk\" (UID: \"886c934e-3162-4331-af40-b52cd04091ad\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ztppk"
Sep 16 19:01:42.052708 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:42.052523    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/886c934e-3162-4331-af40-b52cd04091ad-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-ztppk\" (UID: \"886c934e-3162-4331-af40-b52cd04091ad\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ztppk"
Sep 16 19:01:42.052708 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:42.052554    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/886c934e-3162-4331-af40-b52cd04091ad-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-ztppk\" (UID: \"886c934e-3162-4331-af40-b52cd04091ad\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ztppk"
Sep 16 19:01:42.052708 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:42.052597    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8snrb\" (UniqueName: \"kubernetes.io/projected/886c934e-3162-4331-af40-b52cd04091ad-kube-api-access-8snrb\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-ztppk\" (UID: \"886c934e-3162-4331-af40-b52cd04091ad\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ztppk"
Sep 16 19:01:42.052708 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:42.052632    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/886c934e-3162-4331-af40-b52cd04091ad-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-ztppk\" (UID: \"886c934e-3162-4331-af40-b52cd04091ad\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ztppk"
Sep 16 19:01:42.053069 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:42.053040    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/886c934e-3162-4331-af40-b52cd04091ad-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-ztppk\" (UID: \"886c934e-3162-4331-af40-b52cd04091ad\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ztppk"
Sep 16 19:01:42.053299 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:42.053277    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/886c934e-3162-4331-af40-b52cd04091ad-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-ztppk\" (UID: \"886c934e-3162-4331-af40-b52cd04091ad\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ztppk"
Sep 16 19:01:42.053370 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:42.053311    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/886c934e-3162-4331-af40-b52cd04091ad-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-ztppk\" (UID: \"886c934e-3162-4331-af40-b52cd04091ad\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ztppk"
Sep 16 19:01:42.055220 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:42.055198    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/886c934e-3162-4331-af40-b52cd04091ad-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-ztppk\" (UID: \"886c934e-3162-4331-af40-b52cd04091ad\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ztppk"
Sep 16 19:01:42.063334 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:42.063313    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8snrb\" (UniqueName: \"kubernetes.io/projected/886c934e-3162-4331-af40-b52cd04091ad-kube-api-access-8snrb\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-ztppk\" (UID: \"886c934e-3162-4331-af40-b52cd04091ad\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ztppk"
Sep 16 19:01:42.155510 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:42.155468    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ztppk"
Sep 16 19:01:42.441496 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:42.441471    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ztppk"]
Sep 16 19:01:42.443914 ip-10-0-129-190 kubenswrapper[2586]: W0916 19:01:42.443887    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod886c934e_3162_4331_af40_b52cd04091ad.slice/crio-c5f46e5b941e38dad0c6e799a9ee12d9df294c2f22fe36198d93ada984b69481 WatchSource:0}: Error finding container c5f46e5b941e38dad0c6e799a9ee12d9df294c2f22fe36198d93ada984b69481: Status 404 returned error can't find the container with id c5f46e5b941e38dad0c6e799a9ee12d9df294c2f22fe36198d93ada984b69481
Sep 16 19:01:42.734325 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:42.734212    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ztppk" event={"ID":"886c934e-3162-4331-af40-b52cd04091ad","Type":"ContainerStarted","Data":"e4a84038fa60fa54cce4300b93a151096dc337e70dad497d433d18e68eae2ddf"}
Sep 16 19:01:42.734325 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:42.734310    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ztppk" event={"ID":"886c934e-3162-4331-af40-b52cd04091ad","Type":"ContainerStarted","Data":"c5f46e5b941e38dad0c6e799a9ee12d9df294c2f22fe36198d93ada984b69481"}
Sep 16 19:01:47.757792 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:47.757760    2586 generic.go:356] "Generic (PLEG): container finished" podID="886c934e-3162-4331-af40-b52cd04091ad" containerID="e4a84038fa60fa54cce4300b93a151096dc337e70dad497d433d18e68eae2ddf" exitCode=0
Sep 16 19:01:47.758195 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:47.757834    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ztppk" event={"ID":"886c934e-3162-4331-af40-b52cd04091ad","Type":"ContainerDied","Data":"e4a84038fa60fa54cce4300b93a151096dc337e70dad497d433d18e68eae2ddf"}
Sep 16 19:01:48.764648 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:48.764606    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ztppk" event={"ID":"886c934e-3162-4331-af40-b52cd04091ad","Type":"ContainerStarted","Data":"4a27a66c014a2a8690cfba2d351410e1d1d49af1741a00474351bcca625521fa"}
Sep 16 19:01:48.765024 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:48.764729    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ztppk" event={"ID":"886c934e-3162-4331-af40-b52cd04091ad","Type":"ContainerStarted","Data":"b11e2213924ff546bb009aee2195ae5d820357527b4a780efe01f2fd9f35bd41"}
Sep 16 19:01:48.765024 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:48.764954    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ztppk"
Sep 16 19:01:48.790045 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:48.789992    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ztppk" podStartSLOduration=2.741851897 podStartE2EDuration="7.789976864s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.048124967s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:01:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:01:48.787635101 +0000 UTC m=+1284.231496700" watchObservedRunningTime="2026-09-16 19:01:48.789976864 +0000 UTC m=+1284.233838440"
Sep 16 19:01:49.769883 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:49.769842    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ztppk"
Sep 16 19:01:55.779572 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:01:55.779542    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ztppk"
Sep 16 19:03:20.090698 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:20.090664    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ztppk"
Sep 16 19:03:27.927379 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:27.927348    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ztppk"]
Sep 16 19:03:27.927811 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:27.927586    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ztppk" podUID="886c934e-3162-4331-af40-b52cd04091ad" containerName="kserve-container" containerID="cri-o://b11e2213924ff546bb009aee2195ae5d820357527b4a780efe01f2fd9f35bd41" gracePeriod=30
Sep 16 19:03:27.927811 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:27.927637    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ztppk" podUID="886c934e-3162-4331-af40-b52cd04091ad" containerName="kube-rbac-proxy" containerID="cri-o://4a27a66c014a2a8690cfba2d351410e1d1d49af1741a00474351bcca625521fa" gracePeriod=30
Sep 16 19:03:28.216047 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:28.215960    2586 generic.go:356] "Generic (PLEG): container finished" podID="886c934e-3162-4331-af40-b52cd04091ad" containerID="4a27a66c014a2a8690cfba2d351410e1d1d49af1741a00474351bcca625521fa" exitCode=2
Sep 16 19:03:28.216047 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:28.216035    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ztppk" event={"ID":"886c934e-3162-4331-af40-b52cd04091ad","Type":"ContainerDied","Data":"4a27a66c014a2a8690cfba2d351410e1d1d49af1741a00474351bcca625521fa"}
Sep 16 19:03:29.223798 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:29.223756    2586 generic.go:356] "Generic (PLEG): container finished" podID="886c934e-3162-4331-af40-b52cd04091ad" containerID="b11e2213924ff546bb009aee2195ae5d820357527b4a780efe01f2fd9f35bd41" exitCode=0
Sep 16 19:03:29.224296 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:29.223828    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ztppk" event={"ID":"886c934e-3162-4331-af40-b52cd04091ad","Type":"ContainerDied","Data":"b11e2213924ff546bb009aee2195ae5d820357527b4a780efe01f2fd9f35bd41"}
Sep 16 19:03:29.276354 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:29.276329    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ztppk"
Sep 16 19:03:29.458576 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:29.458471    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/886c934e-3162-4331-af40-b52cd04091ad-kserve-provision-location\") pod \"886c934e-3162-4331-af40-b52cd04091ad\" (UID: \"886c934e-3162-4331-af40-b52cd04091ad\") "
Sep 16 19:03:29.458576 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:29.458526    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8snrb\" (UniqueName: \"kubernetes.io/projected/886c934e-3162-4331-af40-b52cd04091ad-kube-api-access-8snrb\") pod \"886c934e-3162-4331-af40-b52cd04091ad\" (UID: \"886c934e-3162-4331-af40-b52cd04091ad\") "
Sep 16 19:03:29.458576 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:29.458577    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/886c934e-3162-4331-af40-b52cd04091ad-cabundle-cert\") pod \"886c934e-3162-4331-af40-b52cd04091ad\" (UID: \"886c934e-3162-4331-af40-b52cd04091ad\") "
Sep 16 19:03:29.458851 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:29.458614    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/886c934e-3162-4331-af40-b52cd04091ad-proxy-tls\") pod \"886c934e-3162-4331-af40-b52cd04091ad\" (UID: \"886c934e-3162-4331-af40-b52cd04091ad\") "
Sep 16 19:03:29.458851 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:29.458662    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/886c934e-3162-4331-af40-b52cd04091ad-kube-rbac-proxy-sar-config\") pod \"886c934e-3162-4331-af40-b52cd04091ad\" (UID: \"886c934e-3162-4331-af40-b52cd04091ad\") "
Sep 16 19:03:29.459081 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:29.459048    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/886c934e-3162-4331-af40-b52cd04091ad-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "886c934e-3162-4331-af40-b52cd04091ad" (UID: "886c934e-3162-4331-af40-b52cd04091ad"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:03:29.459186 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:29.459081    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/886c934e-3162-4331-af40-b52cd04091ad-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "886c934e-3162-4331-af40-b52cd04091ad" (UID: "886c934e-3162-4331-af40-b52cd04091ad"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:03:29.460734 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:29.460712    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/886c934e-3162-4331-af40-b52cd04091ad-kube-api-access-8snrb" (OuterVolumeSpecName: "kube-api-access-8snrb") pod "886c934e-3162-4331-af40-b52cd04091ad" (UID: "886c934e-3162-4331-af40-b52cd04091ad"). InnerVolumeSpecName "kube-api-access-8snrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:03:29.460833 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:29.460746    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/886c934e-3162-4331-af40-b52cd04091ad-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "886c934e-3162-4331-af40-b52cd04091ad" (UID: "886c934e-3162-4331-af40-b52cd04091ad"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:03:29.463413 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:29.463389    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/886c934e-3162-4331-af40-b52cd04091ad-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "886c934e-3162-4331-af40-b52cd04091ad" (UID: "886c934e-3162-4331-af40-b52cd04091ad"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:03:29.559458 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:29.559426    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/886c934e-3162-4331-af40-b52cd04091ad-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:03:29.559458 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:29.559452    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8snrb\" (UniqueName: \"kubernetes.io/projected/886c934e-3162-4331-af40-b52cd04091ad-kube-api-access-8snrb\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:03:29.559458 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:29.559463    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/886c934e-3162-4331-af40-b52cd04091ad-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:03:29.559688 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:29.559472    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/886c934e-3162-4331-af40-b52cd04091ad-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:03:29.559688 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:29.559481    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/886c934e-3162-4331-af40-b52cd04091ad-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:03:30.230874 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:30.230833    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ztppk" event={"ID":"886c934e-3162-4331-af40-b52cd04091ad","Type":"ContainerDied","Data":"c5f46e5b941e38dad0c6e799a9ee12d9df294c2f22fe36198d93ada984b69481"}
Sep 16 19:03:30.230874 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:30.230874    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ztppk"
Sep 16 19:03:30.231352 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:30.230912    2586 scope.go:117] "RemoveContainer" containerID="4a27a66c014a2a8690cfba2d351410e1d1d49af1741a00474351bcca625521fa"
Sep 16 19:03:30.239647 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:30.239621    2586 scope.go:117] "RemoveContainer" containerID="b11e2213924ff546bb009aee2195ae5d820357527b4a780efe01f2fd9f35bd41"
Sep 16 19:03:30.247473 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:30.247455    2586 scope.go:117] "RemoveContainer" containerID="e4a84038fa60fa54cce4300b93a151096dc337e70dad497d433d18e68eae2ddf"
Sep 16 19:03:30.262878 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:30.262853    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ztppk"]
Sep 16 19:03:30.266940 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:30.266917    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-ztppk"]
Sep 16 19:03:31.092390 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:31.092356    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="886c934e-3162-4331-af40-b52cd04091ad" path="/var/lib/kubelet/pods/886c934e-3162-4331-af40-b52cd04091ad/volumes"
Sep 16 19:03:33.148023 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:33.147989    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8"]
Sep 16 19:03:33.148417 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:33.148320    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="886c934e-3162-4331-af40-b52cd04091ad" containerName="kube-rbac-proxy"
Sep 16 19:03:33.148417 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:33.148333    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="886c934e-3162-4331-af40-b52cd04091ad" containerName="kube-rbac-proxy"
Sep 16 19:03:33.148417 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:33.148344    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="886c934e-3162-4331-af40-b52cd04091ad" containerName="kserve-container"
Sep 16 19:03:33.148417 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:33.148350    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="886c934e-3162-4331-af40-b52cd04091ad" containerName="kserve-container"
Sep 16 19:03:33.148417 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:33.148363    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="886c934e-3162-4331-af40-b52cd04091ad" containerName="storage-initializer"
Sep 16 19:03:33.148417 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:33.148369    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="886c934e-3162-4331-af40-b52cd04091ad" containerName="storage-initializer"
Sep 16 19:03:33.148417 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:33.148418    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="886c934e-3162-4331-af40-b52cd04091ad" containerName="kserve-container"
Sep 16 19:03:33.148639 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:33.148429    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="886c934e-3162-4331-af40-b52cd04091ad" containerName="kube-rbac-proxy"
Sep 16 19:03:33.152590 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:33.152563    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8"
Sep 16 19:03:33.156626 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:33.156603    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:03:33.156769 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:33.156702    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-pv9ct\""
Sep 16 19:03:33.157766 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:33.157744    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Sep 16 19:03:33.157882 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:33.157811    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:03:33.157882 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:33.157862    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:03:33.158088 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:33.157876    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:03:33.158271 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:33.158253    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Sep 16 19:03:33.168268 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:33.168242    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8"]
Sep 16 19:03:33.192316 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:33.192285    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56169800-0e24-4136-9f71-56338b384979-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8\" (UID: \"56169800-0e24-4136-9f71-56338b384979\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8"
Sep 16 19:03:33.192494 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:33.192333    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56169800-0e24-4136-9f71-56338b384979-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8\" (UID: \"56169800-0e24-4136-9f71-56338b384979\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8"
Sep 16 19:03:33.192494 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:33.192367    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56169800-0e24-4136-9f71-56338b384979-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8\" (UID: \"56169800-0e24-4136-9f71-56338b384979\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8"
Sep 16 19:03:33.192494 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:33.192439    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdb4n\" (UniqueName: \"kubernetes.io/projected/56169800-0e24-4136-9f71-56338b384979-kube-api-access-gdb4n\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8\" (UID: \"56169800-0e24-4136-9f71-56338b384979\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8"
Sep 16 19:03:33.192652 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:33.192562    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56169800-0e24-4136-9f71-56338b384979-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8\" (UID: \"56169800-0e24-4136-9f71-56338b384979\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8"
Sep 16 19:03:33.292985 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:33.292951    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56169800-0e24-4136-9f71-56338b384979-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8\" (UID: \"56169800-0e24-4136-9f71-56338b384979\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8"
Sep 16 19:03:33.293168 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:33.292995    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56169800-0e24-4136-9f71-56338b384979-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8\" (UID: \"56169800-0e24-4136-9f71-56338b384979\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8"
Sep 16 19:03:33.293168 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:33.293027    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56169800-0e24-4136-9f71-56338b384979-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8\" (UID: \"56169800-0e24-4136-9f71-56338b384979\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8"
Sep 16 19:03:33.293168 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:33.293074    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdb4n\" (UniqueName: \"kubernetes.io/projected/56169800-0e24-4136-9f71-56338b384979-kube-api-access-gdb4n\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8\" (UID: \"56169800-0e24-4136-9f71-56338b384979\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8"
Sep 16 19:03:33.293168 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:33.293120    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56169800-0e24-4136-9f71-56338b384979-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8\" (UID: \"56169800-0e24-4136-9f71-56338b384979\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8"
Sep 16 19:03:33.293477 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:33.293447    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56169800-0e24-4136-9f71-56338b384979-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8\" (UID: \"56169800-0e24-4136-9f71-56338b384979\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8"
Sep 16 19:03:33.293750 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:33.293730    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56169800-0e24-4136-9f71-56338b384979-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8\" (UID: \"56169800-0e24-4136-9f71-56338b384979\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8"
Sep 16 19:03:33.293795 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:33.293745    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56169800-0e24-4136-9f71-56338b384979-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8\" (UID: \"56169800-0e24-4136-9f71-56338b384979\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8"
Sep 16 19:03:33.295731 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:33.295705    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56169800-0e24-4136-9f71-56338b384979-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8\" (UID: \"56169800-0e24-4136-9f71-56338b384979\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8"
Sep 16 19:03:33.303574 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:33.303551    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdb4n\" (UniqueName: \"kubernetes.io/projected/56169800-0e24-4136-9f71-56338b384979-kube-api-access-gdb4n\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8\" (UID: \"56169800-0e24-4136-9f71-56338b384979\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8"
Sep 16 19:03:33.462960 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:33.462873    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8"
Sep 16 19:03:33.735752 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:33.735726    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8"]
Sep 16 19:03:33.737796 ip-10-0-129-190 kubenswrapper[2586]: W0916 19:03:33.737771    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56169800_0e24_4136_9f71_56338b384979.slice/crio-19d76442891bc0bf586a97fe1283874dc76a13266c81d683f37d7dddb93334ed WatchSource:0}: Error finding container 19d76442891bc0bf586a97fe1283874dc76a13266c81d683f37d7dddb93334ed: Status 404 returned error can't find the container with id 19d76442891bc0bf586a97fe1283874dc76a13266c81d683f37d7dddb93334ed
Sep 16 19:03:34.259189 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:34.259151    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8" event={"ID":"56169800-0e24-4136-9f71-56338b384979","Type":"ContainerStarted","Data":"5a31891b13f0bf77970074487435e5d3490d13f043a4193e51ef9a013b86f99f"}
Sep 16 19:03:34.259651 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:03:34.259211    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8" event={"ID":"56169800-0e24-4136-9f71-56338b384979","Type":"ContainerStarted","Data":"19d76442891bc0bf586a97fe1283874dc76a13266c81d683f37d7dddb93334ed"}
Sep 16 19:04:06.404726 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:04:06.404694    2586 generic.go:356] "Generic (PLEG): container finished" podID="56169800-0e24-4136-9f71-56338b384979" containerID="5a31891b13f0bf77970074487435e5d3490d13f043a4193e51ef9a013b86f99f" exitCode=0
Sep 16 19:04:06.405134 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:04:06.404750    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8" event={"ID":"56169800-0e24-4136-9f71-56338b384979","Type":"ContainerDied","Data":"5a31891b13f0bf77970074487435e5d3490d13f043a4193e51ef9a013b86f99f"}
Sep 16 19:04:07.411765 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:04:07.411728    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8" event={"ID":"56169800-0e24-4136-9f71-56338b384979","Type":"ContainerStarted","Data":"1ffdb95b52f3dadf337a346c1ca1c3671f156cd2f827ac7d1d13b48dcd66e763"}
Sep 16 19:04:07.412192 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:04:07.411805    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8" event={"ID":"56169800-0e24-4136-9f71-56338b384979","Type":"ContainerStarted","Data":"6d3919f6a222423f8082a5be75c6a3ccba96f64e513970e653d7a59a177167ee"}
Sep 16 19:04:07.412192 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:04:07.412120    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8"
Sep 16 19:04:07.438863 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:04:07.438804    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8" podStartSLOduration=2.769829586 podStartE2EDuration="34.438786204s" totalImagesPullingTime="0s" totalInitContainerRuntime="31.668956618s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:03:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:04:07.435929947 +0000 UTC m=+1422.879791523" watchObservedRunningTime="2026-09-16 19:04:07.438786204 +0000 UTC m=+1422.882647780"
Sep 16 19:04:08.416747 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:04:08.416716    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8"
Sep 16 19:04:14.425924 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:04:14.425886    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8"
Sep 16 19:05:36.091424 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:36.091394    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8"
Sep 16 19:05:47.129414 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:47.129365    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8" podUID="56169800-0e24-4136-9f71-56338b384979" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.41:8080/v2/models/isvc-autogluon-v2-variants/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 19:05:49.851112 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:49.851074    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8"]
Sep 16 19:05:49.851532 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:49.851361    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8" podUID="56169800-0e24-4136-9f71-56338b384979" containerName="kserve-container" containerID="cri-o://6d3919f6a222423f8082a5be75c6a3ccba96f64e513970e653d7a59a177167ee" gracePeriod=30
Sep 16 19:05:49.851532 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:49.851382    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8" podUID="56169800-0e24-4136-9f71-56338b384979" containerName="kube-rbac-proxy" containerID="cri-o://1ffdb95b52f3dadf337a346c1ca1c3671f156cd2f827ac7d1d13b48dcd66e763" gracePeriod=30
Sep 16 19:05:50.905556 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:50.905525    2586 generic.go:356] "Generic (PLEG): container finished" podID="56169800-0e24-4136-9f71-56338b384979" containerID="1ffdb95b52f3dadf337a346c1ca1c3671f156cd2f827ac7d1d13b48dcd66e763" exitCode=2
Sep 16 19:05:50.905940 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:50.905563    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8" event={"ID":"56169800-0e24-4136-9f71-56338b384979","Type":"ContainerDied","Data":"1ffdb95b52f3dadf337a346c1ca1c3671f156cd2f827ac7d1d13b48dcd66e763"}
Sep 16 19:05:51.206372 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:51.206346    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8"
Sep 16 19:05:51.291472 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:51.291438    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56169800-0e24-4136-9f71-56338b384979-cabundle-cert\") pod \"56169800-0e24-4136-9f71-56338b384979\" (UID: \"56169800-0e24-4136-9f71-56338b384979\") "
Sep 16 19:05:51.291657 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:51.291520    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56169800-0e24-4136-9f71-56338b384979-kserve-provision-location\") pod \"56169800-0e24-4136-9f71-56338b384979\" (UID: \"56169800-0e24-4136-9f71-56338b384979\") "
Sep 16 19:05:51.291657 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:51.291560    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56169800-0e24-4136-9f71-56338b384979-proxy-tls\") pod \"56169800-0e24-4136-9f71-56338b384979\" (UID: \"56169800-0e24-4136-9f71-56338b384979\") "
Sep 16 19:05:51.291657 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:51.291590    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdb4n\" (UniqueName: \"kubernetes.io/projected/56169800-0e24-4136-9f71-56338b384979-kube-api-access-gdb4n\") pod \"56169800-0e24-4136-9f71-56338b384979\" (UID: \"56169800-0e24-4136-9f71-56338b384979\") "
Sep 16 19:05:51.291822 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:51.291658    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56169800-0e24-4136-9f71-56338b384979-kube-rbac-proxy-sar-config\") pod \"56169800-0e24-4136-9f71-56338b384979\" (UID: \"56169800-0e24-4136-9f71-56338b384979\") "
Sep 16 19:05:51.291873 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:51.291843    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56169800-0e24-4136-9f71-56338b384979-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "56169800-0e24-4136-9f71-56338b384979" (UID: "56169800-0e24-4136-9f71-56338b384979"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:05:51.292046 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:51.292027    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56169800-0e24-4136-9f71-56338b384979-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:05:51.292118 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:51.292066    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56169800-0e24-4136-9f71-56338b384979-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "56169800-0e24-4136-9f71-56338b384979" (UID: "56169800-0e24-4136-9f71-56338b384979"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:05:51.293641 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:51.293617    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56169800-0e24-4136-9f71-56338b384979-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "56169800-0e24-4136-9f71-56338b384979" (UID: "56169800-0e24-4136-9f71-56338b384979"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:05:51.293734 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:51.293702    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56169800-0e24-4136-9f71-56338b384979-kube-api-access-gdb4n" (OuterVolumeSpecName: "kube-api-access-gdb4n") pod "56169800-0e24-4136-9f71-56338b384979" (UID: "56169800-0e24-4136-9f71-56338b384979"). InnerVolumeSpecName "kube-api-access-gdb4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:05:51.295873 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:51.295849    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56169800-0e24-4136-9f71-56338b384979-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "56169800-0e24-4136-9f71-56338b384979" (UID: "56169800-0e24-4136-9f71-56338b384979"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:05:51.392546 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:51.392511    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56169800-0e24-4136-9f71-56338b384979-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:05:51.392546 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:51.392540    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56169800-0e24-4136-9f71-56338b384979-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:05:51.392546 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:51.392551    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56169800-0e24-4136-9f71-56338b384979-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:05:51.392767 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:51.392560    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdb4n\" (UniqueName: \"kubernetes.io/projected/56169800-0e24-4136-9f71-56338b384979-kube-api-access-gdb4n\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:05:51.912384 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:51.912347    2586 generic.go:356] "Generic (PLEG): container finished" podID="56169800-0e24-4136-9f71-56338b384979" containerID="6d3919f6a222423f8082a5be75c6a3ccba96f64e513970e653d7a59a177167ee" exitCode=0
Sep 16 19:05:51.912853 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:51.912405    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8"
Sep 16 19:05:51.912853 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:51.912444    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8" event={"ID":"56169800-0e24-4136-9f71-56338b384979","Type":"ContainerDied","Data":"6d3919f6a222423f8082a5be75c6a3ccba96f64e513970e653d7a59a177167ee"}
Sep 16 19:05:51.912853 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:51.912505    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8" event={"ID":"56169800-0e24-4136-9f71-56338b384979","Type":"ContainerDied","Data":"19d76442891bc0bf586a97fe1283874dc76a13266c81d683f37d7dddb93334ed"}
Sep 16 19:05:51.912853 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:51.912534    2586 scope.go:117] "RemoveContainer" containerID="1ffdb95b52f3dadf337a346c1ca1c3671f156cd2f827ac7d1d13b48dcd66e763"
Sep 16 19:05:51.922050 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:51.922035    2586 scope.go:117] "RemoveContainer" containerID="6d3919f6a222423f8082a5be75c6a3ccba96f64e513970e653d7a59a177167ee"
Sep 16 19:05:51.929579 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:51.929565    2586 scope.go:117] "RemoveContainer" containerID="5a31891b13f0bf77970074487435e5d3490d13f043a4193e51ef9a013b86f99f"
Sep 16 19:05:51.939339 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:51.939141    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8"]
Sep 16 19:05:51.940304 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:51.940267    2586 scope.go:117] "RemoveContainer" containerID="1ffdb95b52f3dadf337a346c1ca1c3671f156cd2f827ac7d1d13b48dcd66e763"
Sep 16 19:05:51.940593 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:05:51.940566    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ffdb95b52f3dadf337a346c1ca1c3671f156cd2f827ac7d1d13b48dcd66e763\": container with ID starting with 1ffdb95b52f3dadf337a346c1ca1c3671f156cd2f827ac7d1d13b48dcd66e763 not found: ID does not exist" containerID="1ffdb95b52f3dadf337a346c1ca1c3671f156cd2f827ac7d1d13b48dcd66e763"
Sep 16 19:05:51.940682 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:51.940602    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ffdb95b52f3dadf337a346c1ca1c3671f156cd2f827ac7d1d13b48dcd66e763"} err="failed to get container status \"1ffdb95b52f3dadf337a346c1ca1c3671f156cd2f827ac7d1d13b48dcd66e763\": rpc error: code = NotFound desc = could not find container \"1ffdb95b52f3dadf337a346c1ca1c3671f156cd2f827ac7d1d13b48dcd66e763\": container with ID starting with 1ffdb95b52f3dadf337a346c1ca1c3671f156cd2f827ac7d1d13b48dcd66e763 not found: ID does not exist"
Sep 16 19:05:51.940682 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:51.940623    2586 scope.go:117] "RemoveContainer" containerID="6d3919f6a222423f8082a5be75c6a3ccba96f64e513970e653d7a59a177167ee"
Sep 16 19:05:51.940944 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:05:51.940910    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d3919f6a222423f8082a5be75c6a3ccba96f64e513970e653d7a59a177167ee\": container with ID starting with 6d3919f6a222423f8082a5be75c6a3ccba96f64e513970e653d7a59a177167ee not found: ID does not exist" containerID="6d3919f6a222423f8082a5be75c6a3ccba96f64e513970e653d7a59a177167ee"
Sep 16 19:05:51.941049 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:51.940950    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d3919f6a222423f8082a5be75c6a3ccba96f64e513970e653d7a59a177167ee"} err="failed to get container status \"6d3919f6a222423f8082a5be75c6a3ccba96f64e513970e653d7a59a177167ee\": rpc error: code = NotFound desc = could not find container \"6d3919f6a222423f8082a5be75c6a3ccba96f64e513970e653d7a59a177167ee\": container with ID starting with 6d3919f6a222423f8082a5be75c6a3ccba96f64e513970e653d7a59a177167ee not found: ID does not exist"
Sep 16 19:05:51.941049 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:51.940975    2586 scope.go:117] "RemoveContainer" containerID="5a31891b13f0bf77970074487435e5d3490d13f043a4193e51ef9a013b86f99f"
Sep 16 19:05:51.941424 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:05:51.941276    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a31891b13f0bf77970074487435e5d3490d13f043a4193e51ef9a013b86f99f\": container with ID starting with 5a31891b13f0bf77970074487435e5d3490d13f043a4193e51ef9a013b86f99f not found: ID does not exist" containerID="5a31891b13f0bf77970074487435e5d3490d13f043a4193e51ef9a013b86f99f"
Sep 16 19:05:51.941424 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:51.941309    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a31891b13f0bf77970074487435e5d3490d13f043a4193e51ef9a013b86f99f"} err="failed to get container status \"5a31891b13f0bf77970074487435e5d3490d13f043a4193e51ef9a013b86f99f\": rpc error: code = NotFound desc = could not find container \"5a31891b13f0bf77970074487435e5d3490d13f043a4193e51ef9a013b86f99f\": container with ID starting with 5a31891b13f0bf77970074487435e5d3490d13f043a4193e51ef9a013b86f99f not found: ID does not exist"
Sep 16 19:05:51.941789 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:51.941768    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-d2hs8"]
Sep 16 19:05:53.097948 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:53.097910    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56169800-0e24-4136-9f71-56338b384979" path="/var/lib/kubelet/pods/56169800-0e24-4136-9f71-56338b384979/volumes"
Sep 16 19:05:54.972586 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:54.972551    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55"]
Sep 16 19:05:54.972981 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:54.972861    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56169800-0e24-4136-9f71-56338b384979" containerName="kserve-container"
Sep 16 19:05:54.972981 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:54.972872    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="56169800-0e24-4136-9f71-56338b384979" containerName="kserve-container"
Sep 16 19:05:54.972981 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:54.972891    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56169800-0e24-4136-9f71-56338b384979" containerName="kube-rbac-proxy"
Sep 16 19:05:54.972981 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:54.972898    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="56169800-0e24-4136-9f71-56338b384979" containerName="kube-rbac-proxy"
Sep 16 19:05:54.972981 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:54.972905    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56169800-0e24-4136-9f71-56338b384979" containerName="storage-initializer"
Sep 16 19:05:54.972981 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:54.972911    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="56169800-0e24-4136-9f71-56338b384979" containerName="storage-initializer"
Sep 16 19:05:54.972981 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:54.972961    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="56169800-0e24-4136-9f71-56338b384979" containerName="kserve-container"
Sep 16 19:05:54.972981 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:54.972971    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="56169800-0e24-4136-9f71-56338b384979" containerName="kube-rbac-proxy"
Sep 16 19:05:54.977319 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:54.977297    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55"
Sep 16 19:05:54.980044 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:54.980021    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:05:54.980168 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:54.980054    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:05:54.980168 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:54.980091    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Sep 16 19:05:54.980958 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:54.980940    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:05:54.981057 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:54.980946    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:05:54.981057 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:54.980980    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-pv9ct\""
Sep 16 19:05:54.981171 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:54.981086    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Sep 16 19:05:54.989942 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:54.989923    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55"]
Sep 16 19:05:55.127006 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:55.126973    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjbfr\" (UniqueName: \"kubernetes.io/projected/b98671e4-5458-4cbe-b5dd-657202607f15-kube-api-access-wjbfr\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55\" (UID: \"b98671e4-5458-4cbe-b5dd-657202607f15\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55"
Sep 16 19:05:55.127006 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:55.127010    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b98671e4-5458-4cbe-b5dd-657202607f15-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55\" (UID: \"b98671e4-5458-4cbe-b5dd-657202607f15\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55"
Sep 16 19:05:55.127201 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:55.127031    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b98671e4-5458-4cbe-b5dd-657202607f15-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55\" (UID: \"b98671e4-5458-4cbe-b5dd-657202607f15\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55"
Sep 16 19:05:55.127201 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:55.127102    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b98671e4-5458-4cbe-b5dd-657202607f15-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55\" (UID: \"b98671e4-5458-4cbe-b5dd-657202607f15\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55"
Sep 16 19:05:55.127201 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:55.127143    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b98671e4-5458-4cbe-b5dd-657202607f15-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55\" (UID: \"b98671e4-5458-4cbe-b5dd-657202607f15\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55"
Sep 16 19:05:55.228112 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:55.228023    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjbfr\" (UniqueName: \"kubernetes.io/projected/b98671e4-5458-4cbe-b5dd-657202607f15-kube-api-access-wjbfr\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55\" (UID: \"b98671e4-5458-4cbe-b5dd-657202607f15\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55"
Sep 16 19:05:55.228112 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:55.228065    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b98671e4-5458-4cbe-b5dd-657202607f15-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55\" (UID: \"b98671e4-5458-4cbe-b5dd-657202607f15\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55"
Sep 16 19:05:55.228112 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:55.228087    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b98671e4-5458-4cbe-b5dd-657202607f15-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55\" (UID: \"b98671e4-5458-4cbe-b5dd-657202607f15\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55"
Sep 16 19:05:55.228424 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:55.228117    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b98671e4-5458-4cbe-b5dd-657202607f15-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55\" (UID: \"b98671e4-5458-4cbe-b5dd-657202607f15\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55"
Sep 16 19:05:55.228424 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:55.228143    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b98671e4-5458-4cbe-b5dd-657202607f15-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55\" (UID: \"b98671e4-5458-4cbe-b5dd-657202607f15\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55"
Sep 16 19:05:55.228637 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:55.228614    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b98671e4-5458-4cbe-b5dd-657202607f15-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55\" (UID: \"b98671e4-5458-4cbe-b5dd-657202607f15\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55"
Sep 16 19:05:55.228783 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:55.228762    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b98671e4-5458-4cbe-b5dd-657202607f15-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55\" (UID: \"b98671e4-5458-4cbe-b5dd-657202607f15\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55"
Sep 16 19:05:55.228829 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:55.228787    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b98671e4-5458-4cbe-b5dd-657202607f15-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55\" (UID: \"b98671e4-5458-4cbe-b5dd-657202607f15\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55"
Sep 16 19:05:55.230522 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:55.230503    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b98671e4-5458-4cbe-b5dd-657202607f15-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55\" (UID: \"b98671e4-5458-4cbe-b5dd-657202607f15\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55"
Sep 16 19:05:55.237658 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:55.237639    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjbfr\" (UniqueName: \"kubernetes.io/projected/b98671e4-5458-4cbe-b5dd-657202607f15-kube-api-access-wjbfr\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55\" (UID: \"b98671e4-5458-4cbe-b5dd-657202607f15\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55"
Sep 16 19:05:55.287313 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:55.287288    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55"
Sep 16 19:05:55.562895 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:55.562868    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55"]
Sep 16 19:05:55.565370 ip-10-0-129-190 kubenswrapper[2586]: W0916 19:05:55.565340    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb98671e4_5458_4cbe_b5dd_657202607f15.slice/crio-d6d9b35719c31ced7f2c9ae23fadaa27af813de6aa4ddfa00c203e875ddbdcb4 WatchSource:0}: Error finding container d6d9b35719c31ced7f2c9ae23fadaa27af813de6aa4ddfa00c203e875ddbdcb4: Status 404 returned error can't find the container with id d6d9b35719c31ced7f2c9ae23fadaa27af813de6aa4ddfa00c203e875ddbdcb4
Sep 16 19:05:55.933526 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:55.933490    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55" event={"ID":"b98671e4-5458-4cbe-b5dd-657202607f15","Type":"ContainerStarted","Data":"5d6d2858315fd94bc1794d2ba4363ceb09539fbb45c3e5201f890ed1dc0ae80b"}
Sep 16 19:05:55.933735 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:05:55.933566    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55" event={"ID":"b98671e4-5458-4cbe-b5dd-657202607f15","Type":"ContainerStarted","Data":"d6d9b35719c31ced7f2c9ae23fadaa27af813de6aa4ddfa00c203e875ddbdcb4"}
Sep 16 19:06:28.078461 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:06:28.078427    2586 generic.go:356] "Generic (PLEG): container finished" podID="b98671e4-5458-4cbe-b5dd-657202607f15" containerID="5d6d2858315fd94bc1794d2ba4363ceb09539fbb45c3e5201f890ed1dc0ae80b" exitCode=0
Sep 16 19:06:28.078809 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:06:28.078501    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55" event={"ID":"b98671e4-5458-4cbe-b5dd-657202607f15","Type":"ContainerDied","Data":"5d6d2858315fd94bc1794d2ba4363ceb09539fbb45c3e5201f890ed1dc0ae80b"}
Sep 16 19:06:29.085692 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:06:29.085656    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55" event={"ID":"b98671e4-5458-4cbe-b5dd-657202607f15","Type":"ContainerStarted","Data":"e4217cdba214463d0455b39f8f664a582dc0b09cc9d29ccbf73f6bcb95de1b2e"}
Sep 16 19:06:29.086058 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:06:29.085743    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55" event={"ID":"b98671e4-5458-4cbe-b5dd-657202607f15","Type":"ContainerStarted","Data":"a9824e7b0bc3c8343eb83af04f343135cbfca8938e5dbccc3197bf8143b7fbef"}
Sep 16 19:06:29.086101 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:06:29.086066    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55"
Sep 16 19:06:29.093068 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:06:29.093042    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55"
Sep 16 19:06:29.114431 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:06:29.114374    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55" podStartSLOduration=2.676069703 podStartE2EDuration="35.114355998s" totalImagesPullingTime="0s" totalInitContainerRuntime="32.438286295s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:05:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:06:29.111448913 +0000 UTC m=+1564.555310502" watchObservedRunningTime="2026-09-16 19:06:29.114355998 +0000 UTC m=+1564.558217575"
Sep 16 19:06:36.101985 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:06:36.101950    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55"
Sep 16 19:08:06.091224 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:06.091193    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55"
Sep 16 19:08:17.128440 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:17.128397    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55" podUID="b98671e4-5458-4cbe-b5dd-657202607f15" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.42:8080/v2/models/isvc-autogluon-v2-noslash/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 19:08:21.290850 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:21.290808    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55"]
Sep 16 19:08:21.291327 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:21.291093    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55" podUID="b98671e4-5458-4cbe-b5dd-657202607f15" containerName="kserve-container" containerID="cri-o://a9824e7b0bc3c8343eb83af04f343135cbfca8938e5dbccc3197bf8143b7fbef" gracePeriod=30
Sep 16 19:08:21.291327 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:21.291243    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55" podUID="b98671e4-5458-4cbe-b5dd-657202607f15" containerName="kube-rbac-proxy" containerID="cri-o://e4217cdba214463d0455b39f8f664a582dc0b09cc9d29ccbf73f6bcb95de1b2e" gracePeriod=30
Sep 16 19:08:21.592551 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:21.592454    2586 generic.go:356] "Generic (PLEG): container finished" podID="b98671e4-5458-4cbe-b5dd-657202607f15" containerID="e4217cdba214463d0455b39f8f664a582dc0b09cc9d29ccbf73f6bcb95de1b2e" exitCode=2
Sep 16 19:08:21.592551 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:21.592532    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55" event={"ID":"b98671e4-5458-4cbe-b5dd-657202607f15","Type":"ContainerDied","Data":"e4217cdba214463d0455b39f8f664a582dc0b09cc9d29ccbf73f6bcb95de1b2e"}
Sep 16 19:08:22.599779 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:22.599740    2586 generic.go:356] "Generic (PLEG): container finished" podID="b98671e4-5458-4cbe-b5dd-657202607f15" containerID="a9824e7b0bc3c8343eb83af04f343135cbfca8938e5dbccc3197bf8143b7fbef" exitCode=0
Sep 16 19:08:22.600131 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:22.599808    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55" event={"ID":"b98671e4-5458-4cbe-b5dd-657202607f15","Type":"ContainerDied","Data":"a9824e7b0bc3c8343eb83af04f343135cbfca8938e5dbccc3197bf8143b7fbef"}
Sep 16 19:08:22.649348 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:22.649325    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55"
Sep 16 19:08:22.776500 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:22.776414    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b98671e4-5458-4cbe-b5dd-657202607f15-kube-rbac-proxy-sar-config\") pod \"b98671e4-5458-4cbe-b5dd-657202607f15\" (UID: \"b98671e4-5458-4cbe-b5dd-657202607f15\") "
Sep 16 19:08:22.776500 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:22.776460    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wjbfr\" (UniqueName: \"kubernetes.io/projected/b98671e4-5458-4cbe-b5dd-657202607f15-kube-api-access-wjbfr\") pod \"b98671e4-5458-4cbe-b5dd-657202607f15\" (UID: \"b98671e4-5458-4cbe-b5dd-657202607f15\") "
Sep 16 19:08:22.776741 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:22.776534    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b98671e4-5458-4cbe-b5dd-657202607f15-cabundle-cert\") pod \"b98671e4-5458-4cbe-b5dd-657202607f15\" (UID: \"b98671e4-5458-4cbe-b5dd-657202607f15\") "
Sep 16 19:08:22.776741 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:22.776552    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b98671e4-5458-4cbe-b5dd-657202607f15-proxy-tls\") pod \"b98671e4-5458-4cbe-b5dd-657202607f15\" (UID: \"b98671e4-5458-4cbe-b5dd-657202607f15\") "
Sep 16 19:08:22.776741 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:22.776576    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b98671e4-5458-4cbe-b5dd-657202607f15-kserve-provision-location\") pod \"b98671e4-5458-4cbe-b5dd-657202607f15\" (UID: \"b98671e4-5458-4cbe-b5dd-657202607f15\") "
Sep 16 19:08:22.776914 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:22.776771    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b98671e4-5458-4cbe-b5dd-657202607f15-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b98671e4-5458-4cbe-b5dd-657202607f15" (UID: "b98671e4-5458-4cbe-b5dd-657202607f15"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:08:22.776914 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:22.776896    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b98671e4-5458-4cbe-b5dd-657202607f15-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b98671e4-5458-4cbe-b5dd-657202607f15" (UID: "b98671e4-5458-4cbe-b5dd-657202607f15"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:08:22.778618 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:22.778587    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b98671e4-5458-4cbe-b5dd-657202607f15-kube-api-access-wjbfr" (OuterVolumeSpecName: "kube-api-access-wjbfr") pod "b98671e4-5458-4cbe-b5dd-657202607f15" (UID: "b98671e4-5458-4cbe-b5dd-657202607f15"). InnerVolumeSpecName "kube-api-access-wjbfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:08:22.778731 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:22.778625    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b98671e4-5458-4cbe-b5dd-657202607f15-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b98671e4-5458-4cbe-b5dd-657202607f15" (UID: "b98671e4-5458-4cbe-b5dd-657202607f15"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:08:22.781068 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:22.781042    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b98671e4-5458-4cbe-b5dd-657202607f15-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b98671e4-5458-4cbe-b5dd-657202607f15" (UID: "b98671e4-5458-4cbe-b5dd-657202607f15"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:08:22.877591 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:22.877554    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b98671e4-5458-4cbe-b5dd-657202607f15-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:08:22.877591 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:22.877585    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b98671e4-5458-4cbe-b5dd-657202607f15-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:08:22.877591 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:22.877594    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b98671e4-5458-4cbe-b5dd-657202607f15-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:08:22.877823 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:22.877604    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b98671e4-5458-4cbe-b5dd-657202607f15-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:08:22.877823 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:22.877613    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjbfr\" (UniqueName: \"kubernetes.io/projected/b98671e4-5458-4cbe-b5dd-657202607f15-kube-api-access-wjbfr\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:08:23.606275 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:23.606250    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55"
Sep 16 19:08:23.606275 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:23.606254    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55" event={"ID":"b98671e4-5458-4cbe-b5dd-657202607f15","Type":"ContainerDied","Data":"d6d9b35719c31ced7f2c9ae23fadaa27af813de6aa4ddfa00c203e875ddbdcb4"}
Sep 16 19:08:23.606815 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:23.606331    2586 scope.go:117] "RemoveContainer" containerID="e4217cdba214463d0455b39f8f664a582dc0b09cc9d29ccbf73f6bcb95de1b2e"
Sep 16 19:08:23.614768 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:23.614743    2586 scope.go:117] "RemoveContainer" containerID="a9824e7b0bc3c8343eb83af04f343135cbfca8938e5dbccc3197bf8143b7fbef"
Sep 16 19:08:23.622165 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:23.622143    2586 scope.go:117] "RemoveContainer" containerID="5d6d2858315fd94bc1794d2ba4363ceb09539fbb45c3e5201f890ed1dc0ae80b"
Sep 16 19:08:23.635583 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:23.635560    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55"]
Sep 16 19:08:23.645108 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:23.645083    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-4hh55"]
Sep 16 19:08:25.092691 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:25.092658    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b98671e4-5458-4cbe-b5dd-657202607f15" path="/var/lib/kubelet/pods/b98671e4-5458-4cbe-b5dd-657202607f15/volumes"
Sep 16 19:08:26.509923 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:26.509885    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg"]
Sep 16 19:08:26.510341 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:26.510218    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b98671e4-5458-4cbe-b5dd-657202607f15" containerName="storage-initializer"
Sep 16 19:08:26.510341 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:26.510252    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="b98671e4-5458-4cbe-b5dd-657202607f15" containerName="storage-initializer"
Sep 16 19:08:26.510341 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:26.510264    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b98671e4-5458-4cbe-b5dd-657202607f15" containerName="kserve-container"
Sep 16 19:08:26.510341 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:26.510270    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="b98671e4-5458-4cbe-b5dd-657202607f15" containerName="kserve-container"
Sep 16 19:08:26.510341 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:26.510283    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b98671e4-5458-4cbe-b5dd-657202607f15" containerName="kube-rbac-proxy"
Sep 16 19:08:26.510341 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:26.510288    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="b98671e4-5458-4cbe-b5dd-657202607f15" containerName="kube-rbac-proxy"
Sep 16 19:08:26.510539 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:26.510346    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="b98671e4-5458-4cbe-b5dd-657202607f15" containerName="kube-rbac-proxy"
Sep 16 19:08:26.510539 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:26.510354    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="b98671e4-5458-4cbe-b5dd-657202607f15" containerName="kserve-container"
Sep 16 19:08:26.515420 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:26.515396    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg"
Sep 16 19:08:26.517969 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:26.517945    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Sep 16 19:08:26.518090 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:26.517947    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:08:26.518177 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:26.518099    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:08:26.518415 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:26.518399    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:08:26.518493 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:26.518414    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-pv9ct\""
Sep 16 19:08:26.518905 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:26.518889    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Sep 16 19:08:26.518976 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:26.518915    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:08:26.534574 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:26.534550    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg"]
Sep 16 19:08:26.613943 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:26.613906    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f317c7c-a201-43fd-a16e-3b498d7d74d2-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg\" (UID: \"6f317c7c-a201-43fd-a16e-3b498d7d74d2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg"
Sep 16 19:08:26.614138 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:26.613969    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f317c7c-a201-43fd-a16e-3b498d7d74d2-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg\" (UID: \"6f317c7c-a201-43fd-a16e-3b498d7d74d2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg"
Sep 16 19:08:26.614138 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:26.614015    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4lj4\" (UniqueName: \"kubernetes.io/projected/6f317c7c-a201-43fd-a16e-3b498d7d74d2-kube-api-access-t4lj4\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg\" (UID: \"6f317c7c-a201-43fd-a16e-3b498d7d74d2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg"
Sep 16 19:08:26.614138 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:26.614034    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f317c7c-a201-43fd-a16e-3b498d7d74d2-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg\" (UID: \"6f317c7c-a201-43fd-a16e-3b498d7d74d2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg"
Sep 16 19:08:26.614138 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:26.614064    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f317c7c-a201-43fd-a16e-3b498d7d74d2-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg\" (UID: \"6f317c7c-a201-43fd-a16e-3b498d7d74d2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg"
Sep 16 19:08:26.715386 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:26.715351    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f317c7c-a201-43fd-a16e-3b498d7d74d2-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg\" (UID: \"6f317c7c-a201-43fd-a16e-3b498d7d74d2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg"
Sep 16 19:08:26.715592 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:26.715419    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f317c7c-a201-43fd-a16e-3b498d7d74d2-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg\" (UID: \"6f317c7c-a201-43fd-a16e-3b498d7d74d2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg"
Sep 16 19:08:26.715592 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:26.715466    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4lj4\" (UniqueName: \"kubernetes.io/projected/6f317c7c-a201-43fd-a16e-3b498d7d74d2-kube-api-access-t4lj4\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg\" (UID: \"6f317c7c-a201-43fd-a16e-3b498d7d74d2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg"
Sep 16 19:08:26.715592 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:26.715492    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f317c7c-a201-43fd-a16e-3b498d7d74d2-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg\" (UID: \"6f317c7c-a201-43fd-a16e-3b498d7d74d2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg"
Sep 16 19:08:26.715592 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:08:26.715509    2586 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-serving-cert: secret "isvc-autogluon-ts-v1-predictor-serving-cert" not found
Sep 16 19:08:26.715592 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:08:26.715588    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f317c7c-a201-43fd-a16e-3b498d7d74d2-proxy-tls podName:6f317c7c-a201-43fd-a16e-3b498d7d74d2 nodeName:}" failed. No retries permitted until 2026-09-16 19:08:27.215558303 +0000 UTC m=+1682.659419870 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6f317c7c-a201-43fd-a16e-3b498d7d74d2-proxy-tls") pod "isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg" (UID: "6f317c7c-a201-43fd-a16e-3b498d7d74d2") : secret "isvc-autogluon-ts-v1-predictor-serving-cert" not found
Sep 16 19:08:26.715592 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:26.715518    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f317c7c-a201-43fd-a16e-3b498d7d74d2-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg\" (UID: \"6f317c7c-a201-43fd-a16e-3b498d7d74d2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg"
Sep 16 19:08:26.715944 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:26.715915    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f317c7c-a201-43fd-a16e-3b498d7d74d2-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg\" (UID: \"6f317c7c-a201-43fd-a16e-3b498d7d74d2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg"
Sep 16 19:08:26.716332 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:26.716309    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f317c7c-a201-43fd-a16e-3b498d7d74d2-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg\" (UID: \"6f317c7c-a201-43fd-a16e-3b498d7d74d2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg"
Sep 16 19:08:26.716421 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:26.716350    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f317c7c-a201-43fd-a16e-3b498d7d74d2-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg\" (UID: \"6f317c7c-a201-43fd-a16e-3b498d7d74d2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg"
Sep 16 19:08:26.731414 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:26.731383    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4lj4\" (UniqueName: \"kubernetes.io/projected/6f317c7c-a201-43fd-a16e-3b498d7d74d2-kube-api-access-t4lj4\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg\" (UID: \"6f317c7c-a201-43fd-a16e-3b498d7d74d2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg"
Sep 16 19:08:27.219564 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:27.219509    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f317c7c-a201-43fd-a16e-3b498d7d74d2-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg\" (UID: \"6f317c7c-a201-43fd-a16e-3b498d7d74d2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg"
Sep 16 19:08:27.222024 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:27.221996    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f317c7c-a201-43fd-a16e-3b498d7d74d2-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg\" (UID: \"6f317c7c-a201-43fd-a16e-3b498d7d74d2\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg"
Sep 16 19:08:27.426389 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:27.426350    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg"
Sep 16 19:08:27.708067 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:27.708033    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg"]
Sep 16 19:08:27.711833 ip-10-0-129-190 kubenswrapper[2586]: W0916 19:08:27.711807    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f317c7c_a201_43fd_a16e_3b498d7d74d2.slice/crio-8c0302eb76f8eb3a912e3e591e3083974b0c909f07b286c939ef077dedeefd85 WatchSource:0}: Error finding container 8c0302eb76f8eb3a912e3e591e3083974b0c909f07b286c939ef077dedeefd85: Status 404 returned error can't find the container with id 8c0302eb76f8eb3a912e3e591e3083974b0c909f07b286c939ef077dedeefd85
Sep 16 19:08:28.634857 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:28.634817    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg" event={"ID":"6f317c7c-a201-43fd-a16e-3b498d7d74d2","Type":"ContainerStarted","Data":"48d82dc6d6386272e9da88d02ac9cf710e3b5125ac93ee16b580c952a2335359"}
Sep 16 19:08:28.635038 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:28.634877    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg" event={"ID":"6f317c7c-a201-43fd-a16e-3b498d7d74d2","Type":"ContainerStarted","Data":"8c0302eb76f8eb3a912e3e591e3083974b0c909f07b286c939ef077dedeefd85"}
Sep 16 19:08:32.654557 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:32.654467    2586 generic.go:356] "Generic (PLEG): container finished" podID="6f317c7c-a201-43fd-a16e-3b498d7d74d2" containerID="48d82dc6d6386272e9da88d02ac9cf710e3b5125ac93ee16b580c952a2335359" exitCode=0
Sep 16 19:08:32.654557 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:32.654540    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg" event={"ID":"6f317c7c-a201-43fd-a16e-3b498d7d74d2","Type":"ContainerDied","Data":"48d82dc6d6386272e9da88d02ac9cf710e3b5125ac93ee16b580c952a2335359"}
Sep 16 19:08:33.661754 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:33.661713    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg" event={"ID":"6f317c7c-a201-43fd-a16e-3b498d7d74d2","Type":"ContainerStarted","Data":"74c3b5580feea4e2d7101f96345cc4a7168c2d406a1db364809b7c0fdb1554f3"}
Sep 16 19:08:33.662165 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:33.661840    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg" event={"ID":"6f317c7c-a201-43fd-a16e-3b498d7d74d2","Type":"ContainerStarted","Data":"9a3ea5a9ac3de97955d562be69821f934195c846eec801902b84e165f2ce2aec"}
Sep 16 19:08:33.662165 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:33.662148    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg"
Sep 16 19:08:33.662271 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:33.662257    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg"
Sep 16 19:08:33.663585 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:33.663560    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg" podUID="6f317c7c-a201-43fd-a16e-3b498d7d74d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 16 19:08:34.667403 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:34.667362    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg" podUID="6f317c7c-a201-43fd-a16e-3b498d7d74d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 16 19:08:39.673073 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:39.673039    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg"
Sep 16 19:08:39.673702 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:39.673673    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg" podUID="6f317c7c-a201-43fd-a16e-3b498d7d74d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 16 19:08:39.699286 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:39.699203    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg" podStartSLOduration=9.084155083 podStartE2EDuration="13.69918362s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.615028537s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:08:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:08:33.701138155 +0000 UTC m=+1689.144999731" watchObservedRunningTime="2026-09-16 19:08:39.69918362 +0000 UTC m=+1695.143045197"
Sep 16 19:08:49.675340 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:08:49.675265    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg"
Sep 16 19:09:08.786153 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:08.786121    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg"]
Sep 16 19:09:08.786597 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:08.786375    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg" podUID="6f317c7c-a201-43fd-a16e-3b498d7d74d2" containerName="kserve-container" containerID="cri-o://9a3ea5a9ac3de97955d562be69821f934195c846eec801902b84e165f2ce2aec" gracePeriod=30
Sep 16 19:09:08.786597 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:08.786419    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg" podUID="6f317c7c-a201-43fd-a16e-3b498d7d74d2" containerName="kube-rbac-proxy" containerID="cri-o://74c3b5580feea4e2d7101f96345cc4a7168c2d406a1db364809b7c0fdb1554f3" gracePeriod=30
Sep 16 19:09:09.668427 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:09.668376    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg" podUID="6f317c7c-a201-43fd-a16e-3b498d7d74d2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Sep 16 19:09:09.673804 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:09.673771    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg" podUID="6f317c7c-a201-43fd-a16e-3b498d7d74d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 16 19:09:09.831046 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:09.831012    2586 generic.go:356] "Generic (PLEG): container finished" podID="6f317c7c-a201-43fd-a16e-3b498d7d74d2" containerID="74c3b5580feea4e2d7101f96345cc4a7168c2d406a1db364809b7c0fdb1554f3" exitCode=2
Sep 16 19:09:09.831433 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:09.831074    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg" event={"ID":"6f317c7c-a201-43fd-a16e-3b498d7d74d2","Type":"ContainerDied","Data":"74c3b5580feea4e2d7101f96345cc4a7168c2d406a1db364809b7c0fdb1554f3"}
Sep 16 19:09:10.528941 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:10.528916    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg"
Sep 16 19:09:10.589154 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:10.589067    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f317c7c-a201-43fd-a16e-3b498d7d74d2-proxy-tls\") pod \"6f317c7c-a201-43fd-a16e-3b498d7d74d2\" (UID: \"6f317c7c-a201-43fd-a16e-3b498d7d74d2\") "
Sep 16 19:09:10.589154 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:10.589127    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f317c7c-a201-43fd-a16e-3b498d7d74d2-cabundle-cert\") pod \"6f317c7c-a201-43fd-a16e-3b498d7d74d2\" (UID: \"6f317c7c-a201-43fd-a16e-3b498d7d74d2\") "
Sep 16 19:09:10.589154 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:10.589148    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t4lj4\" (UniqueName: \"kubernetes.io/projected/6f317c7c-a201-43fd-a16e-3b498d7d74d2-kube-api-access-t4lj4\") pod \"6f317c7c-a201-43fd-a16e-3b498d7d74d2\" (UID: \"6f317c7c-a201-43fd-a16e-3b498d7d74d2\") "
Sep 16 19:09:10.589454 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:10.589168    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f317c7c-a201-43fd-a16e-3b498d7d74d2-kserve-provision-location\") pod \"6f317c7c-a201-43fd-a16e-3b498d7d74d2\" (UID: \"6f317c7c-a201-43fd-a16e-3b498d7d74d2\") "
Sep 16 19:09:10.589454 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:10.589209    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f317c7c-a201-43fd-a16e-3b498d7d74d2-kube-rbac-proxy-sar-config\") pod \"6f317c7c-a201-43fd-a16e-3b498d7d74d2\" (UID: \"6f317c7c-a201-43fd-a16e-3b498d7d74d2\") "
Sep 16 19:09:10.589593 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:10.589569    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f317c7c-a201-43fd-a16e-3b498d7d74d2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6f317c7c-a201-43fd-a16e-3b498d7d74d2" (UID: "6f317c7c-a201-43fd-a16e-3b498d7d74d2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:09:10.589671 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:10.589646    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f317c7c-a201-43fd-a16e-3b498d7d74d2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6f317c7c-a201-43fd-a16e-3b498d7d74d2" (UID: "6f317c7c-a201-43fd-a16e-3b498d7d74d2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:09:10.590262 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:10.590214    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f317c7c-a201-43fd-a16e-3b498d7d74d2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6f317c7c-a201-43fd-a16e-3b498d7d74d2" (UID: "6f317c7c-a201-43fd-a16e-3b498d7d74d2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:09:10.591427 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:10.591405    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f317c7c-a201-43fd-a16e-3b498d7d74d2-kube-api-access-t4lj4" (OuterVolumeSpecName: "kube-api-access-t4lj4") pod "6f317c7c-a201-43fd-a16e-3b498d7d74d2" (UID: "6f317c7c-a201-43fd-a16e-3b498d7d74d2"). InnerVolumeSpecName "kube-api-access-t4lj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:09:10.591514 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:10.591404    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f317c7c-a201-43fd-a16e-3b498d7d74d2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6f317c7c-a201-43fd-a16e-3b498d7d74d2" (UID: "6f317c7c-a201-43fd-a16e-3b498d7d74d2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:09:10.690308 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:10.690266    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f317c7c-a201-43fd-a16e-3b498d7d74d2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:09:10.690308 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:10.690297    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f317c7c-a201-43fd-a16e-3b498d7d74d2-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:09:10.690308 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:10.690306    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f317c7c-a201-43fd-a16e-3b498d7d74d2-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:09:10.690308 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:10.690317    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4lj4\" (UniqueName: \"kubernetes.io/projected/6f317c7c-a201-43fd-a16e-3b498d7d74d2-kube-api-access-t4lj4\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:09:10.690566 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:10.690326    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f317c7c-a201-43fd-a16e-3b498d7d74d2-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:09:10.836968 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:10.836930    2586 generic.go:356] "Generic (PLEG): container finished" podID="6f317c7c-a201-43fd-a16e-3b498d7d74d2" containerID="9a3ea5a9ac3de97955d562be69821f934195c846eec801902b84e165f2ce2aec" exitCode=0
Sep 16 19:09:10.837387 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:10.836995    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg"
Sep 16 19:09:10.837387 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:10.837014    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg" event={"ID":"6f317c7c-a201-43fd-a16e-3b498d7d74d2","Type":"ContainerDied","Data":"9a3ea5a9ac3de97955d562be69821f934195c846eec801902b84e165f2ce2aec"}
Sep 16 19:09:10.837387 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:10.837077    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg" event={"ID":"6f317c7c-a201-43fd-a16e-3b498d7d74d2","Type":"ContainerDied","Data":"8c0302eb76f8eb3a912e3e591e3083974b0c909f07b286c939ef077dedeefd85"}
Sep 16 19:09:10.837387 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:10.837104    2586 scope.go:117] "RemoveContainer" containerID="74c3b5580feea4e2d7101f96345cc4a7168c2d406a1db364809b7c0fdb1554f3"
Sep 16 19:09:10.851398 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:10.851376    2586 scope.go:117] "RemoveContainer" containerID="9a3ea5a9ac3de97955d562be69821f934195c846eec801902b84e165f2ce2aec"
Sep 16 19:09:10.859119 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:10.859098    2586 scope.go:117] "RemoveContainer" containerID="48d82dc6d6386272e9da88d02ac9cf710e3b5125ac93ee16b580c952a2335359"
Sep 16 19:09:10.866653 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:10.866630    2586 scope.go:117] "RemoveContainer" containerID="74c3b5580feea4e2d7101f96345cc4a7168c2d406a1db364809b7c0fdb1554f3"
Sep 16 19:09:10.867110 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:09:10.867078    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74c3b5580feea4e2d7101f96345cc4a7168c2d406a1db364809b7c0fdb1554f3\": container with ID starting with 74c3b5580feea4e2d7101f96345cc4a7168c2d406a1db364809b7c0fdb1554f3 not found: ID does not exist" containerID="74c3b5580feea4e2d7101f96345cc4a7168c2d406a1db364809b7c0fdb1554f3"
Sep 16 19:09:10.867198 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:10.867120    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74c3b5580feea4e2d7101f96345cc4a7168c2d406a1db364809b7c0fdb1554f3"} err="failed to get container status \"74c3b5580feea4e2d7101f96345cc4a7168c2d406a1db364809b7c0fdb1554f3\": rpc error: code = NotFound desc = could not find container \"74c3b5580feea4e2d7101f96345cc4a7168c2d406a1db364809b7c0fdb1554f3\": container with ID starting with 74c3b5580feea4e2d7101f96345cc4a7168c2d406a1db364809b7c0fdb1554f3 not found: ID does not exist"
Sep 16 19:09:10.867198 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:10.867140    2586 scope.go:117] "RemoveContainer" containerID="9a3ea5a9ac3de97955d562be69821f934195c846eec801902b84e165f2ce2aec"
Sep 16 19:09:10.867481 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:09:10.867456    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a3ea5a9ac3de97955d562be69821f934195c846eec801902b84e165f2ce2aec\": container with ID starting with 9a3ea5a9ac3de97955d562be69821f934195c846eec801902b84e165f2ce2aec not found: ID does not exist" containerID="9a3ea5a9ac3de97955d562be69821f934195c846eec801902b84e165f2ce2aec"
Sep 16 19:09:10.867526 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:10.867495    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a3ea5a9ac3de97955d562be69821f934195c846eec801902b84e165f2ce2aec"} err="failed to get container status \"9a3ea5a9ac3de97955d562be69821f934195c846eec801902b84e165f2ce2aec\": rpc error: code = NotFound desc = could not find container \"9a3ea5a9ac3de97955d562be69821f934195c846eec801902b84e165f2ce2aec\": container with ID starting with 9a3ea5a9ac3de97955d562be69821f934195c846eec801902b84e165f2ce2aec not found: ID does not exist"
Sep 16 19:09:10.867526 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:10.867518    2586 scope.go:117] "RemoveContainer" containerID="48d82dc6d6386272e9da88d02ac9cf710e3b5125ac93ee16b580c952a2335359"
Sep 16 19:09:10.867797 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:09:10.867779    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48d82dc6d6386272e9da88d02ac9cf710e3b5125ac93ee16b580c952a2335359\": container with ID starting with 48d82dc6d6386272e9da88d02ac9cf710e3b5125ac93ee16b580c952a2335359 not found: ID does not exist" containerID="48d82dc6d6386272e9da88d02ac9cf710e3b5125ac93ee16b580c952a2335359"
Sep 16 19:09:10.867841 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:10.867802    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48d82dc6d6386272e9da88d02ac9cf710e3b5125ac93ee16b580c952a2335359"} err="failed to get container status \"48d82dc6d6386272e9da88d02ac9cf710e3b5125ac93ee16b580c952a2335359\": rpc error: code = NotFound desc = could not find container \"48d82dc6d6386272e9da88d02ac9cf710e3b5125ac93ee16b580c952a2335359\": container with ID starting with 48d82dc6d6386272e9da88d02ac9cf710e3b5125ac93ee16b580c952a2335359 not found: ID does not exist"
Sep 16 19:09:10.869223 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:10.869205    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg"]
Sep 16 19:09:10.872675 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:10.872653    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-4h6sg"]
Sep 16 19:09:11.092569 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:11.092536    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f317c7c-a201-43fd-a16e-3b498d7d74d2" path="/var/lib/kubelet/pods/6f317c7c-a201-43fd-a16e-3b498d7d74d2/volumes"
Sep 16 19:09:14.057415 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:14.057378    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt"]
Sep 16 19:09:14.057774 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:14.057706    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f317c7c-a201-43fd-a16e-3b498d7d74d2" containerName="kube-rbac-proxy"
Sep 16 19:09:14.057774 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:14.057718    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f317c7c-a201-43fd-a16e-3b498d7d74d2" containerName="kube-rbac-proxy"
Sep 16 19:09:14.057774 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:14.057731    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f317c7c-a201-43fd-a16e-3b498d7d74d2" containerName="storage-initializer"
Sep 16 19:09:14.057774 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:14.057738    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f317c7c-a201-43fd-a16e-3b498d7d74d2" containerName="storage-initializer"
Sep 16 19:09:14.057774 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:14.057752    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f317c7c-a201-43fd-a16e-3b498d7d74d2" containerName="kserve-container"
Sep 16 19:09:14.057774 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:14.057758    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f317c7c-a201-43fd-a16e-3b498d7d74d2" containerName="kserve-container"
Sep 16 19:09:14.057956 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:14.057807    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f317c7c-a201-43fd-a16e-3b498d7d74d2" containerName="kube-rbac-proxy"
Sep 16 19:09:14.057956 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:14.057815    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f317c7c-a201-43fd-a16e-3b498d7d74d2" containerName="kserve-container"
Sep 16 19:09:14.062858 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:14.062835    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt"
Sep 16 19:09:14.066815 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:14.066783    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-predictor-serving-cert\""
Sep 16 19:09:14.067561 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:14.067544    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:09:14.067622 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:14.067604    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:09:14.067866 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:14.067846    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-kube-rbac-proxy-sar-config\""
Sep 16 19:09:14.067866 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:14.067850    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:09:14.068003 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:14.067852    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-pv9ct\""
Sep 16 19:09:14.076273 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:14.076254    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:09:14.090147 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:14.090123    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt"]
Sep 16 19:09:14.122244 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:14.122197    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd09894b-9106-484b-b2b5-cb1e1924d7fb-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt\" (UID: \"fd09894b-9106-484b-b2b5-cb1e1924d7fb\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt"
Sep 16 19:09:14.122405 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:14.122255    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd09894b-9106-484b-b2b5-cb1e1924d7fb-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt\" (UID: \"fd09894b-9106-484b-b2b5-cb1e1924d7fb\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt"
Sep 16 19:09:14.122405 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:14.122278    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxzxd\" (UniqueName: \"kubernetes.io/projected/fd09894b-9106-484b-b2b5-cb1e1924d7fb-kube-api-access-pxzxd\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt\" (UID: \"fd09894b-9106-484b-b2b5-cb1e1924d7fb\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt"
Sep 16 19:09:14.122405 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:14.122300    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd09894b-9106-484b-b2b5-cb1e1924d7fb-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt\" (UID: \"fd09894b-9106-484b-b2b5-cb1e1924d7fb\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt"
Sep 16 19:09:14.122405 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:14.122318    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd09894b-9106-484b-b2b5-cb1e1924d7fb-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt\" (UID: \"fd09894b-9106-484b-b2b5-cb1e1924d7fb\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt"
Sep 16 19:09:14.223382 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:14.223342    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd09894b-9106-484b-b2b5-cb1e1924d7fb-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt\" (UID: \"fd09894b-9106-484b-b2b5-cb1e1924d7fb\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt"
Sep 16 19:09:14.223382 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:14.223390    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd09894b-9106-484b-b2b5-cb1e1924d7fb-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt\" (UID: \"fd09894b-9106-484b-b2b5-cb1e1924d7fb\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt"
Sep 16 19:09:14.223661 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:14.223503    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxzxd\" (UniqueName: \"kubernetes.io/projected/fd09894b-9106-484b-b2b5-cb1e1924d7fb-kube-api-access-pxzxd\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt\" (UID: \"fd09894b-9106-484b-b2b5-cb1e1924d7fb\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt"
Sep 16 19:09:14.223661 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:14.223552    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd09894b-9106-484b-b2b5-cb1e1924d7fb-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt\" (UID: \"fd09894b-9106-484b-b2b5-cb1e1924d7fb\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt"
Sep 16 19:09:14.223661 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:14.223574    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd09894b-9106-484b-b2b5-cb1e1924d7fb-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt\" (UID: \"fd09894b-9106-484b-b2b5-cb1e1924d7fb\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt"
Sep 16 19:09:14.223933 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:14.223905    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd09894b-9106-484b-b2b5-cb1e1924d7fb-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt\" (UID: \"fd09894b-9106-484b-b2b5-cb1e1924d7fb\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt"
Sep 16 19:09:14.224129 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:14.224110    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd09894b-9106-484b-b2b5-cb1e1924d7fb-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt\" (UID: \"fd09894b-9106-484b-b2b5-cb1e1924d7fb\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt"
Sep 16 19:09:14.224249 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:14.224211    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd09894b-9106-484b-b2b5-cb1e1924d7fb-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt\" (UID: \"fd09894b-9106-484b-b2b5-cb1e1924d7fb\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt"
Sep 16 19:09:14.226179 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:14.226160    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd09894b-9106-484b-b2b5-cb1e1924d7fb-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt\" (UID: \"fd09894b-9106-484b-b2b5-cb1e1924d7fb\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt"
Sep 16 19:09:14.236879 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:14.236852    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxzxd\" (UniqueName: \"kubernetes.io/projected/fd09894b-9106-484b-b2b5-cb1e1924d7fb-kube-api-access-pxzxd\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt\" (UID: \"fd09894b-9106-484b-b2b5-cb1e1924d7fb\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt"
Sep 16 19:09:14.373360 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:14.373326    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt"
Sep 16 19:09:14.659985 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:14.659960    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt"]
Sep 16 19:09:14.662483 ip-10-0-129-190 kubenswrapper[2586]: W0916 19:09:14.662446    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd09894b_9106_484b_b2b5_cb1e1924d7fb.slice/crio-17966aa8894bb86455c27433aae30d89f3f4aec28cdc63996552d53b2651986d WatchSource:0}: Error finding container 17966aa8894bb86455c27433aae30d89f3f4aec28cdc63996552d53b2651986d: Status 404 returned error can't find the container with id 17966aa8894bb86455c27433aae30d89f3f4aec28cdc63996552d53b2651986d
Sep 16 19:09:14.861271 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:14.861207    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt" event={"ID":"fd09894b-9106-484b-b2b5-cb1e1924d7fb","Type":"ContainerStarted","Data":"3c167526cf8ebbe31f0a0f462a9bae69e2a84748ea773b63aa67b2a970d3efd6"}
Sep 16 19:09:14.861467 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:14.861289    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt" event={"ID":"fd09894b-9106-484b-b2b5-cb1e1924d7fb","Type":"ContainerStarted","Data":"17966aa8894bb86455c27433aae30d89f3f4aec28cdc63996552d53b2651986d"}
Sep 16 19:09:19.886897 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:19.886864    2586 generic.go:356] "Generic (PLEG): container finished" podID="fd09894b-9106-484b-b2b5-cb1e1924d7fb" containerID="3c167526cf8ebbe31f0a0f462a9bae69e2a84748ea773b63aa67b2a970d3efd6" exitCode=0
Sep 16 19:09:19.887281 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:19.886936    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt" event={"ID":"fd09894b-9106-484b-b2b5-cb1e1924d7fb","Type":"ContainerDied","Data":"3c167526cf8ebbe31f0a0f462a9bae69e2a84748ea773b63aa67b2a970d3efd6"}
Sep 16 19:09:20.894393 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:20.894353    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt" event={"ID":"fd09894b-9106-484b-b2b5-cb1e1924d7fb","Type":"ContainerStarted","Data":"fe24bf3053d37bb69b23f2e858ef405bf13cb909a108f27c29c0870fb270ed09"}
Sep 16 19:09:20.894967 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:20.894460    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt" event={"ID":"fd09894b-9106-484b-b2b5-cb1e1924d7fb","Type":"ContainerStarted","Data":"4f86ecda65b129f9a37a520ba4b83295dec68aecce75b5246446b5ac43c81cd3"}
Sep 16 19:09:20.894967 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:20.894789    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt"
Sep 16 19:09:20.894967 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:20.894958    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt"
Sep 16 19:09:20.896245 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:20.896207    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt" podUID="fd09894b-9106-484b-b2b5-cb1e1924d7fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 16 19:09:20.932662 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:20.932606    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt" podStartSLOduration=2.420457308 podStartE2EDuration="6.932589605s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.512132297s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:09:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:09:20.930495381 +0000 UTC m=+1736.374356957" watchObservedRunningTime="2026-09-16 19:09:20.932589605 +0000 UTC m=+1736.376451182"
Sep 16 19:09:21.898724 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:21.898678    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt" podUID="fd09894b-9106-484b-b2b5-cb1e1924d7fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 16 19:09:26.903401 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:26.903373    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt"
Sep 16 19:09:26.903938 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:26.903910    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt" podUID="fd09894b-9106-484b-b2b5-cb1e1924d7fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 16 19:09:36.904511 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:36.904477    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt"
Sep 16 19:09:49.112438 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:49.112402    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt"]
Sep 16 19:09:49.112823 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:49.112666    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt" podUID="fd09894b-9106-484b-b2b5-cb1e1924d7fb" containerName="kube-rbac-proxy" containerID="cri-o://fe24bf3053d37bb69b23f2e858ef405bf13cb909a108f27c29c0870fb270ed09" gracePeriod=30
Sep 16 19:09:49.112823 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:49.112778    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt" podUID="fd09894b-9106-484b-b2b5-cb1e1924d7fb" containerName="kserve-container" containerID="cri-o://4f86ecda65b129f9a37a520ba4b83295dec68aecce75b5246446b5ac43c81cd3" gracePeriod=30
Sep 16 19:09:50.034565 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:50.034527    2586 generic.go:356] "Generic (PLEG): container finished" podID="fd09894b-9106-484b-b2b5-cb1e1924d7fb" containerID="fe24bf3053d37bb69b23f2e858ef405bf13cb909a108f27c29c0870fb270ed09" exitCode=2
Sep 16 19:09:50.034735 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:50.034588    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt" event={"ID":"fd09894b-9106-484b-b2b5-cb1e1924d7fb","Type":"ContainerDied","Data":"fe24bf3053d37bb69b23f2e858ef405bf13cb909a108f27c29c0870fb270ed09"}
Sep 16 19:09:50.465989 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:50.465965    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt"
Sep 16 19:09:50.554330 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:50.554285    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd09894b-9106-484b-b2b5-cb1e1924d7fb-kube-rbac-proxy-sar-config\") pod \"fd09894b-9106-484b-b2b5-cb1e1924d7fb\" (UID: \"fd09894b-9106-484b-b2b5-cb1e1924d7fb\") "
Sep 16 19:09:50.554520 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:50.554382    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd09894b-9106-484b-b2b5-cb1e1924d7fb-kserve-provision-location\") pod \"fd09894b-9106-484b-b2b5-cb1e1924d7fb\" (UID: \"fd09894b-9106-484b-b2b5-cb1e1924d7fb\") "
Sep 16 19:09:50.554520 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:50.554424    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxzxd\" (UniqueName: \"kubernetes.io/projected/fd09894b-9106-484b-b2b5-cb1e1924d7fb-kube-api-access-pxzxd\") pod \"fd09894b-9106-484b-b2b5-cb1e1924d7fb\" (UID: \"fd09894b-9106-484b-b2b5-cb1e1924d7fb\") "
Sep 16 19:09:50.554520 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:50.554478    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd09894b-9106-484b-b2b5-cb1e1924d7fb-proxy-tls\") pod \"fd09894b-9106-484b-b2b5-cb1e1924d7fb\" (UID: \"fd09894b-9106-484b-b2b5-cb1e1924d7fb\") "
Sep 16 19:09:50.554741 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:50.554526    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd09894b-9106-484b-b2b5-cb1e1924d7fb-cabundle-cert\") pod \"fd09894b-9106-484b-b2b5-cb1e1924d7fb\" (UID: \"fd09894b-9106-484b-b2b5-cb1e1924d7fb\") "
Sep 16 19:09:50.554741 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:50.554701    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd09894b-9106-484b-b2b5-cb1e1924d7fb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fd09894b-9106-484b-b2b5-cb1e1924d7fb" (UID: "fd09894b-9106-484b-b2b5-cb1e1924d7fb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:09:50.554850 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:50.554774    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd09894b-9106-484b-b2b5-cb1e1924d7fb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:09:50.554961 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:50.554929    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd09894b-9106-484b-b2b5-cb1e1924d7fb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fd09894b-9106-484b-b2b5-cb1e1924d7fb" (UID: "fd09894b-9106-484b-b2b5-cb1e1924d7fb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:09:50.555582 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:50.555560    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd09894b-9106-484b-b2b5-cb1e1924d7fb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fd09894b-9106-484b-b2b5-cb1e1924d7fb" (UID: "fd09894b-9106-484b-b2b5-cb1e1924d7fb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:09:50.556620 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:50.556600    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd09894b-9106-484b-b2b5-cb1e1924d7fb-kube-api-access-pxzxd" (OuterVolumeSpecName: "kube-api-access-pxzxd") pod "fd09894b-9106-484b-b2b5-cb1e1924d7fb" (UID: "fd09894b-9106-484b-b2b5-cb1e1924d7fb"). InnerVolumeSpecName "kube-api-access-pxzxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:09:50.556733 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:50.556711    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd09894b-9106-484b-b2b5-cb1e1924d7fb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fd09894b-9106-484b-b2b5-cb1e1924d7fb" (UID: "fd09894b-9106-484b-b2b5-cb1e1924d7fb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:09:50.655296 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:50.655262    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd09894b-9106-484b-b2b5-cb1e1924d7fb-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:09:50.655296 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:50.655293    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxzxd\" (UniqueName: \"kubernetes.io/projected/fd09894b-9106-484b-b2b5-cb1e1924d7fb-kube-api-access-pxzxd\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:09:50.655459 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:50.655307    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd09894b-9106-484b-b2b5-cb1e1924d7fb-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:09:50.655459 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:50.655320    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd09894b-9106-484b-b2b5-cb1e1924d7fb-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:09:51.041806 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:51.041720    2586 generic.go:356] "Generic (PLEG): container finished" podID="fd09894b-9106-484b-b2b5-cb1e1924d7fb" containerID="4f86ecda65b129f9a37a520ba4b83295dec68aecce75b5246446b5ac43c81cd3" exitCode=0
Sep 16 19:09:51.041806 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:51.041787    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt"
Sep 16 19:09:51.041806 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:51.041795    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt" event={"ID":"fd09894b-9106-484b-b2b5-cb1e1924d7fb","Type":"ContainerDied","Data":"4f86ecda65b129f9a37a520ba4b83295dec68aecce75b5246446b5ac43c81cd3"}
Sep 16 19:09:51.042042 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:51.041856    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt" event={"ID":"fd09894b-9106-484b-b2b5-cb1e1924d7fb","Type":"ContainerDied","Data":"17966aa8894bb86455c27433aae30d89f3f4aec28cdc63996552d53b2651986d"}
Sep 16 19:09:51.042042 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:51.041883    2586 scope.go:117] "RemoveContainer" containerID="fe24bf3053d37bb69b23f2e858ef405bf13cb909a108f27c29c0870fb270ed09"
Sep 16 19:09:51.051301 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:51.051212    2586 scope.go:117] "RemoveContainer" containerID="4f86ecda65b129f9a37a520ba4b83295dec68aecce75b5246446b5ac43c81cd3"
Sep 16 19:09:51.059161 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:51.059142    2586 scope.go:117] "RemoveContainer" containerID="3c167526cf8ebbe31f0a0f462a9bae69e2a84748ea773b63aa67b2a970d3efd6"
Sep 16 19:09:51.067019 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:51.067000    2586 scope.go:117] "RemoveContainer" containerID="fe24bf3053d37bb69b23f2e858ef405bf13cb909a108f27c29c0870fb270ed09"
Sep 16 19:09:51.067488 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:09:51.067425    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe24bf3053d37bb69b23f2e858ef405bf13cb909a108f27c29c0870fb270ed09\": container with ID starting with fe24bf3053d37bb69b23f2e858ef405bf13cb909a108f27c29c0870fb270ed09 not found: ID does not exist" containerID="fe24bf3053d37bb69b23f2e858ef405bf13cb909a108f27c29c0870fb270ed09"
Sep 16 19:09:51.067488 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:51.067459    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe24bf3053d37bb69b23f2e858ef405bf13cb909a108f27c29c0870fb270ed09"} err="failed to get container status \"fe24bf3053d37bb69b23f2e858ef405bf13cb909a108f27c29c0870fb270ed09\": rpc error: code = NotFound desc = could not find container \"fe24bf3053d37bb69b23f2e858ef405bf13cb909a108f27c29c0870fb270ed09\": container with ID starting with fe24bf3053d37bb69b23f2e858ef405bf13cb909a108f27c29c0870fb270ed09 not found: ID does not exist"
Sep 16 19:09:51.067488 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:51.067484    2586 scope.go:117] "RemoveContainer" containerID="4f86ecda65b129f9a37a520ba4b83295dec68aecce75b5246446b5ac43c81cd3"
Sep 16 19:09:51.067819 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:09:51.067797    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f86ecda65b129f9a37a520ba4b83295dec68aecce75b5246446b5ac43c81cd3\": container with ID starting with 4f86ecda65b129f9a37a520ba4b83295dec68aecce75b5246446b5ac43c81cd3 not found: ID does not exist" containerID="4f86ecda65b129f9a37a520ba4b83295dec68aecce75b5246446b5ac43c81cd3"
Sep 16 19:09:51.067953 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:51.067929    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f86ecda65b129f9a37a520ba4b83295dec68aecce75b5246446b5ac43c81cd3"} err="failed to get container status \"4f86ecda65b129f9a37a520ba4b83295dec68aecce75b5246446b5ac43c81cd3\": rpc error: code = NotFound desc = could not find container \"4f86ecda65b129f9a37a520ba4b83295dec68aecce75b5246446b5ac43c81cd3\": container with ID starting with 4f86ecda65b129f9a37a520ba4b83295dec68aecce75b5246446b5ac43c81cd3 not found: ID does not exist"
Sep 16 19:09:51.068025 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:51.067968    2586 scope.go:117] "RemoveContainer" containerID="3c167526cf8ebbe31f0a0f462a9bae69e2a84748ea773b63aa67b2a970d3efd6"
Sep 16 19:09:51.068702 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:09:51.068684    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c167526cf8ebbe31f0a0f462a9bae69e2a84748ea773b63aa67b2a970d3efd6\": container with ID starting with 3c167526cf8ebbe31f0a0f462a9bae69e2a84748ea773b63aa67b2a970d3efd6 not found: ID does not exist" containerID="3c167526cf8ebbe31f0a0f462a9bae69e2a84748ea773b63aa67b2a970d3efd6"
Sep 16 19:09:51.068790 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:51.068711    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c167526cf8ebbe31f0a0f462a9bae69e2a84748ea773b63aa67b2a970d3efd6"} err="failed to get container status \"3c167526cf8ebbe31f0a0f462a9bae69e2a84748ea773b63aa67b2a970d3efd6\": rpc error: code = NotFound desc = could not find container \"3c167526cf8ebbe31f0a0f462a9bae69e2a84748ea773b63aa67b2a970d3efd6\": container with ID starting with 3c167526cf8ebbe31f0a0f462a9bae69e2a84748ea773b63aa67b2a970d3efd6 not found: ID does not exist"
Sep 16 19:09:51.069369 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:51.069349    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt"]
Sep 16 19:09:51.074939 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:51.074918    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-4zljt"]
Sep 16 19:09:51.096790 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:51.096766    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd09894b-9106-484b-b2b5-cb1e1924d7fb" path="/var/lib/kubelet/pods/fd09894b-9106-484b-b2b5-cb1e1924d7fb/volumes"
Sep 16 19:09:54.487204 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:54.487167    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-rktzp"]
Sep 16 19:09:54.487675 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:54.487515    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd09894b-9106-484b-b2b5-cb1e1924d7fb" containerName="kube-rbac-proxy"
Sep 16 19:09:54.487675 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:54.487528    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd09894b-9106-484b-b2b5-cb1e1924d7fb" containerName="kube-rbac-proxy"
Sep 16 19:09:54.487675 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:54.487548    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd09894b-9106-484b-b2b5-cb1e1924d7fb" containerName="storage-initializer"
Sep 16 19:09:54.487675 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:54.487554    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd09894b-9106-484b-b2b5-cb1e1924d7fb" containerName="storage-initializer"
Sep 16 19:09:54.487675 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:54.487566    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd09894b-9106-484b-b2b5-cb1e1924d7fb" containerName="kserve-container"
Sep 16 19:09:54.487675 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:54.487572    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd09894b-9106-484b-b2b5-cb1e1924d7fb" containerName="kserve-container"
Sep 16 19:09:54.487675 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:54.487619    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd09894b-9106-484b-b2b5-cb1e1924d7fb" containerName="kserve-container"
Sep 16 19:09:54.487675 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:54.487628    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd09894b-9106-484b-b2b5-cb1e1924d7fb" containerName="kube-rbac-proxy"
Sep 16 19:09:54.492665 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:54.492634    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-rktzp"
Sep 16 19:09:54.495116 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:54.495091    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 16 19:09:54.495269 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:54.495182    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-pv9ct\""
Sep 16 19:09:54.495269 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:54.495241    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:09:54.495408 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:54.495386    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:09:54.495986 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:54.495969    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:09:54.496064 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:54.495987    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:09:54.496064 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:54.495975    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 16 19:09:54.506137 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:54.506113    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-rktzp"]
Sep 16 19:09:54.594092 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:54.594047    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61efacc4-37a7-4825-9edd-c3737a03217f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-76cc455874-rktzp\" (UID: \"61efacc4-37a7-4825-9edd-c3737a03217f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-rktzp"
Sep 16 19:09:54.594278 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:54.594119    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61efacc4-37a7-4825-9edd-c3737a03217f-kserve-provision-location\") pod \"isvc-canary-create-predictor-76cc455874-rktzp\" (UID: \"61efacc4-37a7-4825-9edd-c3737a03217f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-rktzp"
Sep 16 19:09:54.594278 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:54.594145    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61efacc4-37a7-4825-9edd-c3737a03217f-proxy-tls\") pod \"isvc-canary-create-predictor-76cc455874-rktzp\" (UID: \"61efacc4-37a7-4825-9edd-c3737a03217f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-rktzp"
Sep 16 19:09:54.594278 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:54.594213    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/61efacc4-37a7-4825-9edd-c3737a03217f-cabundle-cert\") pod \"isvc-canary-create-predictor-76cc455874-rktzp\" (UID: \"61efacc4-37a7-4825-9edd-c3737a03217f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-rktzp"
Sep 16 19:09:54.594404 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:54.594303    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9sz7\" (UniqueName: \"kubernetes.io/projected/61efacc4-37a7-4825-9edd-c3737a03217f-kube-api-access-x9sz7\") pod \"isvc-canary-create-predictor-76cc455874-rktzp\" (UID: \"61efacc4-37a7-4825-9edd-c3737a03217f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-rktzp"
Sep 16 19:09:54.694982 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:54.694949    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61efacc4-37a7-4825-9edd-c3737a03217f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-76cc455874-rktzp\" (UID: \"61efacc4-37a7-4825-9edd-c3737a03217f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-rktzp"
Sep 16 19:09:54.695161 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:54.694995    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61efacc4-37a7-4825-9edd-c3737a03217f-kserve-provision-location\") pod \"isvc-canary-create-predictor-76cc455874-rktzp\" (UID: \"61efacc4-37a7-4825-9edd-c3737a03217f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-rktzp"
Sep 16 19:09:54.695161 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:54.695016    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61efacc4-37a7-4825-9edd-c3737a03217f-proxy-tls\") pod \"isvc-canary-create-predictor-76cc455874-rktzp\" (UID: \"61efacc4-37a7-4825-9edd-c3737a03217f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-rktzp"
Sep 16 19:09:54.695161 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:54.695041    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/61efacc4-37a7-4825-9edd-c3737a03217f-cabundle-cert\") pod \"isvc-canary-create-predictor-76cc455874-rktzp\" (UID: \"61efacc4-37a7-4825-9edd-c3737a03217f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-rktzp"
Sep 16 19:09:54.695161 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:54.695078    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9sz7\" (UniqueName: \"kubernetes.io/projected/61efacc4-37a7-4825-9edd-c3737a03217f-kube-api-access-x9sz7\") pod \"isvc-canary-create-predictor-76cc455874-rktzp\" (UID: \"61efacc4-37a7-4825-9edd-c3737a03217f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-rktzp"
Sep 16 19:09:54.695484 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:54.695463    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61efacc4-37a7-4825-9edd-c3737a03217f-kserve-provision-location\") pod \"isvc-canary-create-predictor-76cc455874-rktzp\" (UID: \"61efacc4-37a7-4825-9edd-c3737a03217f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-rktzp"
Sep 16 19:09:54.695810 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:54.695784    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61efacc4-37a7-4825-9edd-c3737a03217f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-76cc455874-rktzp\" (UID: \"61efacc4-37a7-4825-9edd-c3737a03217f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-rktzp"
Sep 16 19:09:54.695810 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:54.695801    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/61efacc4-37a7-4825-9edd-c3737a03217f-cabundle-cert\") pod \"isvc-canary-create-predictor-76cc455874-rktzp\" (UID: \"61efacc4-37a7-4825-9edd-c3737a03217f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-rktzp"
Sep 16 19:09:54.697619 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:54.697598    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61efacc4-37a7-4825-9edd-c3737a03217f-proxy-tls\") pod \"isvc-canary-create-predictor-76cc455874-rktzp\" (UID: \"61efacc4-37a7-4825-9edd-c3737a03217f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-rktzp"
Sep 16 19:09:54.708055 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:54.708029    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9sz7\" (UniqueName: \"kubernetes.io/projected/61efacc4-37a7-4825-9edd-c3737a03217f-kube-api-access-x9sz7\") pod \"isvc-canary-create-predictor-76cc455874-rktzp\" (UID: \"61efacc4-37a7-4825-9edd-c3737a03217f\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-rktzp"
Sep 16 19:09:54.804157 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:54.804068    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-rktzp"
Sep 16 19:09:55.082003 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:55.081976    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-rktzp"]
Sep 16 19:09:55.084445 ip-10-0-129-190 kubenswrapper[2586]: W0916 19:09:55.084418    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61efacc4_37a7_4825_9edd_c3737a03217f.slice/crio-8adb8111464299456e6d78a8c31a503f8f4c6597fa0aa9a71f84fad20e119143 WatchSource:0}: Error finding container 8adb8111464299456e6d78a8c31a503f8f4c6597fa0aa9a71f84fad20e119143: Status 404 returned error can't find the container with id 8adb8111464299456e6d78a8c31a503f8f4c6597fa0aa9a71f84fad20e119143
Sep 16 19:09:56.066577 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:56.066544    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-rktzp" event={"ID":"61efacc4-37a7-4825-9edd-c3737a03217f","Type":"ContainerStarted","Data":"55f9d3016da973368b9208e07e43115593844dbfad5c230fec24fe706a5953a2"}
Sep 16 19:09:56.066993 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:56.066600    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-rktzp" event={"ID":"61efacc4-37a7-4825-9edd-c3737a03217f","Type":"ContainerStarted","Data":"8adb8111464299456e6d78a8c31a503f8f4c6597fa0aa9a71f84fad20e119143"}
Sep 16 19:09:59.081310 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:59.081274    2586 generic.go:356] "Generic (PLEG): container finished" podID="61efacc4-37a7-4825-9edd-c3737a03217f" containerID="55f9d3016da973368b9208e07e43115593844dbfad5c230fec24fe706a5953a2" exitCode=0
Sep 16 19:09:59.081688 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:59.081346    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-rktzp" event={"ID":"61efacc4-37a7-4825-9edd-c3737a03217f","Type":"ContainerDied","Data":"55f9d3016da973368b9208e07e43115593844dbfad5c230fec24fe706a5953a2"}
Sep 16 19:09:59.082671 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:09:59.082656    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:10:13.165088 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:10:13.165051    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-rktzp" event={"ID":"61efacc4-37a7-4825-9edd-c3737a03217f","Type":"ContainerStarted","Data":"d7c97af5a1430a0f69e8acaf4f109867afb154e68ee77dc3fc837e3b4b2ad8a5"}
Sep 16 19:10:13.165554 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:10:13.165110    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-rktzp" event={"ID":"61efacc4-37a7-4825-9edd-c3737a03217f","Type":"ContainerStarted","Data":"d6eac8b79103b16b73e92bb4517c3bf39c49da622d50e46dc9119d024b911e3b"}
Sep 16 19:10:13.165554 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:10:13.165381    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-rktzp"
Sep 16 19:10:13.165554 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:10:13.165525    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-rktzp"
Sep 16 19:10:13.166909 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:10:13.166880    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-rktzp" podUID="61efacc4-37a7-4825-9edd-c3737a03217f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 16 19:10:14.169440 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:10:14.169397    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-rktzp" podUID="61efacc4-37a7-4825-9edd-c3737a03217f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 16 19:10:19.175179 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:10:19.175094    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-rktzp"
Sep 16 19:10:19.175759 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:10:19.175725    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-rktzp" podUID="61efacc4-37a7-4825-9edd-c3737a03217f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 16 19:10:19.202531 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:10:19.202475    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-rktzp" podStartSLOduration=7.816855681 podStartE2EDuration="25.202459096s" totalImagesPullingTime="13.59384376s" totalInitContainerRuntime="3.791759655s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:09:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:10:13.199773186 +0000 UTC m=+1788.643634764" watchObservedRunningTime="2026-09-16 19:10:19.202459096 +0000 UTC m=+1794.646320672"
Sep 16 19:10:29.175909 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:10:29.175869    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-rktzp" podUID="61efacc4-37a7-4825-9edd-c3737a03217f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 16 19:10:39.175614 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:10:39.175573    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-rktzp" podUID="61efacc4-37a7-4825-9edd-c3737a03217f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 16 19:10:47.960833 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:10:47.960793    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-866dcdbf79-5gnxs"]
Sep 16 19:10:48.019787 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:10:48.019754    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-866dcdbf79-5gnxs"]
Sep 16 19:10:48.019992 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:10:48.019879    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-866dcdbf79-5gnxs"
Sep 16 19:10:48.023867 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:10:48.023835    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
Sep 16 19:10:48.024008 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:10:48.023931    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
Sep 16 19:10:48.065243 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:10:48.065205    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6bf274d5-34ca-40b1-a1d9-3625e7cd7791-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-866dcdbf79-5gnxs\" (UID: \"6bf274d5-34ca-40b1-a1d9-3625e7cd7791\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-866dcdbf79-5gnxs"
Sep 16 19:10:48.065382 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:10:48.065253    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6bf274d5-34ca-40b1-a1d9-3625e7cd7791-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-866dcdbf79-5gnxs\" (UID: \"6bf274d5-34ca-40b1-a1d9-3625e7cd7791\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-866dcdbf79-5gnxs"
Sep 16 19:10:48.065382 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:10:48.065347    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6bf274d5-34ca-40b1-a1d9-3625e7cd7791-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-866dcdbf79-5gnxs\" (UID: \"6bf274d5-34ca-40b1-a1d9-3625e7cd7791\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-866dcdbf79-5gnxs"
Sep 16 19:10:48.065470 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:10:48.065383    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cwrf\" (UniqueName: \"kubernetes.io/projected/6bf274d5-34ca-40b1-a1d9-3625e7cd7791-kube-api-access-4cwrf\") pod \"isvc-sklearn-v2-predictor-866dcdbf79-5gnxs\" (UID: \"6bf274d5-34ca-40b1-a1d9-3625e7cd7791\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-866dcdbf79-5gnxs"
Sep 16 19:10:48.065470 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:10:48.065414    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6bf274d5-34ca-40b1-a1d9-3625e7cd7791-proxy-tls\") pod \"isvc-sklearn-v2-predictor-866dcdbf79-5gnxs\" (UID: \"6bf274d5-34ca-40b1-a1d9-3625e7cd7791\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-866dcdbf79-5gnxs"
Sep 16 19:10:48.166698 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:10:48.166665    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6bf274d5-34ca-40b1-a1d9-3625e7cd7791-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-866dcdbf79-5gnxs\" (UID: \"6bf274d5-34ca-40b1-a1d9-3625e7cd7791\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-866dcdbf79-5gnxs"
Sep 16 19:10:48.166698 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:10:48.166700    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6bf274d5-34ca-40b1-a1d9-3625e7cd7791-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-866dcdbf79-5gnxs\" (UID: \"6bf274d5-34ca-40b1-a1d9-3625e7cd7791\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-866dcdbf79-5gnxs"
Sep 16 19:10:48.166927 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:10:48.166754    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6bf274d5-34ca-40b1-a1d9-3625e7cd7791-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-866dcdbf79-5gnxs\" (UID: \"6bf274d5-34ca-40b1-a1d9-3625e7cd7791\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-866dcdbf79-5gnxs"
Sep 16 19:10:48.166927 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:10:48.166773    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cwrf\" (UniqueName: \"kubernetes.io/projected/6bf274d5-34ca-40b1-a1d9-3625e7cd7791-kube-api-access-4cwrf\") pod \"isvc-sklearn-v2-predictor-866dcdbf79-5gnxs\" (UID: \"6bf274d5-34ca-40b1-a1d9-3625e7cd7791\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-866dcdbf79-5gnxs"
Sep 16 19:10:48.166927 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:10:48.166810    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6bf274d5-34ca-40b1-a1d9-3625e7cd7791-proxy-tls\") pod \"isvc-sklearn-v2-predictor-866dcdbf79-5gnxs\" (UID: \"6bf274d5-34ca-40b1-a1d9-3625e7cd7791\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-866dcdbf79-5gnxs"
Sep 16 19:10:48.167114 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:10:48.167057    2586 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-predictor-serving-cert: secret "isvc-sklearn-v2-predictor-serving-cert" not found
Sep 16 19:10:48.167114 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:10:48.167077    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6bf274d5-34ca-40b1-a1d9-3625e7cd7791-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-866dcdbf79-5gnxs\" (UID: \"6bf274d5-34ca-40b1-a1d9-3625e7cd7791\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-866dcdbf79-5gnxs"
Sep 16 19:10:48.167213 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:10:48.167136    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bf274d5-34ca-40b1-a1d9-3625e7cd7791-proxy-tls podName:6bf274d5-34ca-40b1-a1d9-3625e7cd7791 nodeName:}" failed. No retries permitted until 2026-09-16 19:10:48.667100302 +0000 UTC m=+1824.110961858 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6bf274d5-34ca-40b1-a1d9-3625e7cd7791-proxy-tls") pod "isvc-sklearn-v2-predictor-866dcdbf79-5gnxs" (UID: "6bf274d5-34ca-40b1-a1d9-3625e7cd7791") : secret "isvc-sklearn-v2-predictor-serving-cert" not found
Sep 16 19:10:48.167524 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:10:48.167508    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6bf274d5-34ca-40b1-a1d9-3625e7cd7791-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-866dcdbf79-5gnxs\" (UID: \"6bf274d5-34ca-40b1-a1d9-3625e7cd7791\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-866dcdbf79-5gnxs"
Sep 16 19:10:48.167564 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:10:48.167519    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6bf274d5-34ca-40b1-a1d9-3625e7cd7791-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-866dcdbf79-5gnxs\" (UID: \"6bf274d5-34ca-40b1-a1d9-3625e7cd7791\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-866dcdbf79-5gnxs"
Sep 16 19:10:48.179962 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:10:48.179939    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cwrf\" (UniqueName: \"kubernetes.io/projected/6bf274d5-34ca-40b1-a1d9-3625e7cd7791-kube-api-access-4cwrf\") pod \"isvc-sklearn-v2-predictor-866dcdbf79-5gnxs\" (UID: \"6bf274d5-34ca-40b1-a1d9-3625e7cd7791\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-866dcdbf79-5gnxs"
Sep 16 19:10:48.670284 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:10:48.670243    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6bf274d5-34ca-40b1-a1d9-3625e7cd7791-proxy-tls\") pod \"isvc-sklearn-v2-predictor-866dcdbf79-5gnxs\" (UID: \"6bf274d5-34ca-40b1-a1d9-3625e7cd7791\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-866dcdbf79-5gnxs"
Sep 16 19:10:48.672700 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:10:48.672676    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6bf274d5-34ca-40b1-a1d9-3625e7cd7791-proxy-tls\") pod \"isvc-sklearn-v2-predictor-866dcdbf79-5gnxs\" (UID: \"6bf274d5-34ca-40b1-a1d9-3625e7cd7791\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-866dcdbf79-5gnxs"
Sep 16 19:10:48.930031 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:10:48.929933    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-866dcdbf79-5gnxs"
Sep 16 19:10:49.175696 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:10:49.175628    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-rktzp" podUID="61efacc4-37a7-4825-9edd-c3737a03217f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 16 19:10:49.210935 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:10:49.210820    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-866dcdbf79-5gnxs"]
Sep 16 19:10:49.213563 ip-10-0-129-190 kubenswrapper[2586]: W0916 19:10:49.213535    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bf274d5_34ca_40b1_a1d9_3625e7cd7791.slice/crio-cc86e480e586fe50fcfa968555792ff63efb11dfba055a47ace086b1f2f5b05b WatchSource:0}: Error finding container cc86e480e586fe50fcfa968555792ff63efb11dfba055a47ace086b1f2f5b05b: Status 404 returned error can't find the container with id cc86e480e586fe50fcfa968555792ff63efb11dfba055a47ace086b1f2f5b05b
Sep 16 19:10:49.339904 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:10:49.339861    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-866dcdbf79-5gnxs" event={"ID":"6bf274d5-34ca-40b1-a1d9-3625e7cd7791","Type":"ContainerStarted","Data":"cb250998980025b51f17392d415f6a3d442f8b3697a06a0b1797532ec7ac105c"}
Sep 16 19:10:49.340074 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:10:49.339944    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-866dcdbf79-5gnxs" event={"ID":"6bf274d5-34ca-40b1-a1d9-3625e7cd7791","Type":"ContainerStarted","Data":"cc86e480e586fe50fcfa968555792ff63efb11dfba055a47ace086b1f2f5b05b"}
Sep 16 19:10:53.359161 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:10:53.359127    2586 generic.go:356] "Generic (PLEG): container finished" podID="6bf274d5-34ca-40b1-a1d9-3625e7cd7791" containerID="cb250998980025b51f17392d415f6a3d442f8b3697a06a0b1797532ec7ac105c" exitCode=0
Sep 16 19:10:53.359606 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:10:53.359198    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-866dcdbf79-5gnxs" event={"ID":"6bf274d5-34ca-40b1-a1d9-3625e7cd7791","Type":"ContainerDied","Data":"cb250998980025b51f17392d415f6a3d442f8b3697a06a0b1797532ec7ac105c"}
Sep 16 19:10:54.366577 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:10:54.366542    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-866dcdbf79-5gnxs" event={"ID":"6bf274d5-34ca-40b1-a1d9-3625e7cd7791","Type":"ContainerStarted","Data":"4d81e5e7f69af18866d97cc053cdefa75494e0907704c57c31d6eee258dc0998"}
Sep 16 19:10:54.366946 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:10:54.366629    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-866dcdbf79-5gnxs" event={"ID":"6bf274d5-34ca-40b1-a1d9-3625e7cd7791","Type":"ContainerStarted","Data":"4a66cc9bbde4f9528015a8f46864d3a0678b48f188a9704b632780bde3e92d85"}
Sep 16 19:10:54.366946 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:10:54.366894    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-866dcdbf79-5gnxs"
Sep 16 19:10:54.367065 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:10:54.367041    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-866dcdbf79-5gnxs"
Sep 16 19:10:54.368616 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:10:54.368590    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-866dcdbf79-5gnxs" podUID="6bf274d5-34ca-40b1-a1d9-3625e7cd7791" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 16 19:10:54.413081 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:10:54.413035    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-866dcdbf79-5gnxs" podStartSLOduration=3.831404684 podStartE2EDuration="7.413022152s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.581617468s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:10:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:10:54.412123719 +0000 UTC m=+1829.855985288" watchObservedRunningTime="2026-09-16 19:10:54.413022152 +0000 UTC m=+1829.856883727"
Sep 16 19:10:55.371077 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:10:55.371040    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-866dcdbf79-5gnxs" podUID="6bf274d5-34ca-40b1-a1d9-3625e7cd7791" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 16 19:10:59.175731 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:10:59.175691    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-rktzp" podUID="61efacc4-37a7-4825-9edd-c3737a03217f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 16 19:11:00.376249 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:00.376207    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-866dcdbf79-5gnxs"
Sep 16 19:11:00.376730 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:00.376707    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-866dcdbf79-5gnxs" podUID="6bf274d5-34ca-40b1-a1d9-3625e7cd7791" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 16 19:11:09.176055 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:09.176010    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-rktzp" podUID="61efacc4-37a7-4825-9edd-c3737a03217f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 16 19:11:10.377648 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:10.377604    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-866dcdbf79-5gnxs" podUID="6bf274d5-34ca-40b1-a1d9-3625e7cd7791" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 16 19:11:18.940835 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:18.940790    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-rktzp"]
Sep 16 19:11:18.941350 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:18.941141    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-rktzp" podUID="61efacc4-37a7-4825-9edd-c3737a03217f" containerName="kserve-container" containerID="cri-o://d6eac8b79103b16b73e92bb4517c3bf39c49da622d50e46dc9119d024b911e3b" gracePeriod=30
Sep 16 19:11:18.941350 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:18.941221    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-rktzp" podUID="61efacc4-37a7-4825-9edd-c3737a03217f" containerName="kube-rbac-proxy" containerID="cri-o://d7c97af5a1430a0f69e8acaf4f109867afb154e68ee77dc3fc837e3b4b2ad8a5" gracePeriod=30
Sep 16 19:11:18.941484 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:18.941402    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-rktzp" podUID="61efacc4-37a7-4825-9edd-c3737a03217f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 16 19:11:19.170626 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:19.170585    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-rktzp" podUID="61efacc4-37a7-4825-9edd-c3737a03217f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused"
Sep 16 19:11:19.488054 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:19.488020    2586 generic.go:356] "Generic (PLEG): container finished" podID="61efacc4-37a7-4825-9edd-c3737a03217f" containerID="d7c97af5a1430a0f69e8acaf4f109867afb154e68ee77dc3fc837e3b4b2ad8a5" exitCode=2
Sep 16 19:11:19.488271 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:19.488069    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-rktzp" event={"ID":"61efacc4-37a7-4825-9edd-c3737a03217f","Type":"ContainerDied","Data":"d7c97af5a1430a0f69e8acaf4f109867afb154e68ee77dc3fc837e3b4b2ad8a5"}
Sep 16 19:11:20.377457 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:20.377415    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-866dcdbf79-5gnxs" podUID="6bf274d5-34ca-40b1-a1d9-3625e7cd7791" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 16 19:11:24.169943 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:24.169898    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-rktzp" podUID="61efacc4-37a7-4825-9edd-c3737a03217f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused"
Sep 16 19:11:28.943157 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:28.943126    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-rktzp"
Sep 16 19:11:29.170433 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:29.170387    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-rktzp" podUID="61efacc4-37a7-4825-9edd-c3737a03217f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused"
Sep 16 19:11:29.170616 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:29.170528    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-rktzp"
Sep 16 19:11:29.699449 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:29.699402    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-prp88"]
Sep 16 19:11:29.703659 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:29.703627    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-prp88"
Sep 16 19:11:29.706114 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:29.706079    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 16 19:11:29.706935 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:29.706911    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 16 19:11:29.723070 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:29.723043    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-prp88"]
Sep 16 19:11:29.738032 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:29.738009    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99d3c63d-b16e-4a7f-8929-cbd6357e5a80-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-prp88\" (UID: \"99d3c63d-b16e-4a7f-8929-cbd6357e5a80\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-prp88"
Sep 16 19:11:29.738155 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:29.738039    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fv5k\" (UniqueName: \"kubernetes.io/projected/99d3c63d-b16e-4a7f-8929-cbd6357e5a80-kube-api-access-7fv5k\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-prp88\" (UID: \"99d3c63d-b16e-4a7f-8929-cbd6357e5a80\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-prp88"
Sep 16 19:11:29.738155 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:29.738063    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99d3c63d-b16e-4a7f-8929-cbd6357e5a80-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-prp88\" (UID: \"99d3c63d-b16e-4a7f-8929-cbd6357e5a80\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-prp88"
Sep 16 19:11:29.738292 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:29.738167    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99d3c63d-b16e-4a7f-8929-cbd6357e5a80-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-prp88\" (UID: \"99d3c63d-b16e-4a7f-8929-cbd6357e5a80\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-prp88"
Sep 16 19:11:29.738292 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:29.738275    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99d3c63d-b16e-4a7f-8929-cbd6357e5a80-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-prp88\" (UID: \"99d3c63d-b16e-4a7f-8929-cbd6357e5a80\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-prp88"
Sep 16 19:11:29.839305 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:29.839266    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99d3c63d-b16e-4a7f-8929-cbd6357e5a80-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-prp88\" (UID: \"99d3c63d-b16e-4a7f-8929-cbd6357e5a80\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-prp88"
Sep 16 19:11:29.839305 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:29.839307    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fv5k\" (UniqueName: \"kubernetes.io/projected/99d3c63d-b16e-4a7f-8929-cbd6357e5a80-kube-api-access-7fv5k\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-prp88\" (UID: \"99d3c63d-b16e-4a7f-8929-cbd6357e5a80\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-prp88"
Sep 16 19:11:29.839557 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:29.839327    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99d3c63d-b16e-4a7f-8929-cbd6357e5a80-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-prp88\" (UID: \"99d3c63d-b16e-4a7f-8929-cbd6357e5a80\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-prp88"
Sep 16 19:11:29.839557 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:29.839356    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99d3c63d-b16e-4a7f-8929-cbd6357e5a80-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-prp88\" (UID: \"99d3c63d-b16e-4a7f-8929-cbd6357e5a80\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-prp88"
Sep 16 19:11:29.839557 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:29.839421    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99d3c63d-b16e-4a7f-8929-cbd6357e5a80-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-prp88\" (UID: \"99d3c63d-b16e-4a7f-8929-cbd6357e5a80\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-prp88"
Sep 16 19:11:29.839557 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:11:29.839554    2586 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-serving-cert: secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Sep 16 19:11:29.839784 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:11:29.839619    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99d3c63d-b16e-4a7f-8929-cbd6357e5a80-proxy-tls podName:99d3c63d-b16e-4a7f-8929-cbd6357e5a80 nodeName:}" failed. No retries permitted until 2026-09-16 19:11:30.339602189 +0000 UTC m=+1865.783463742 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/99d3c63d-b16e-4a7f-8929-cbd6357e5a80-proxy-tls") pod "isvc-canary-promote-v2-predictor-769cdc7dcc-prp88" (UID: "99d3c63d-b16e-4a7f-8929-cbd6357e5a80") : secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Sep 16 19:11:29.839858 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:29.839820    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99d3c63d-b16e-4a7f-8929-cbd6357e5a80-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-prp88\" (UID: \"99d3c63d-b16e-4a7f-8929-cbd6357e5a80\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-prp88"
Sep 16 19:11:29.840047 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:29.840023    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99d3c63d-b16e-4a7f-8929-cbd6357e5a80-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-prp88\" (UID: \"99d3c63d-b16e-4a7f-8929-cbd6357e5a80\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-prp88"
Sep 16 19:11:29.840114 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:29.840041    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99d3c63d-b16e-4a7f-8929-cbd6357e5a80-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-prp88\" (UID: \"99d3c63d-b16e-4a7f-8929-cbd6357e5a80\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-prp88"
Sep 16 19:11:29.852611 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:29.852586    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-kthkb"]
Sep 16 19:11:29.856119 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:29.856085    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fv5k\" (UniqueName: \"kubernetes.io/projected/99d3c63d-b16e-4a7f-8929-cbd6357e5a80-kube-api-access-7fv5k\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-prp88\" (UID: \"99d3c63d-b16e-4a7f-8929-cbd6357e5a80\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-prp88"
Sep 16 19:11:29.856478 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:29.856461    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-kthkb"
Sep 16 19:11:29.858776 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:29.858752    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 16 19:11:29.875813 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:29.875787    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-kthkb"]
Sep 16 19:11:29.940604 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:29.940574    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7cc11d70-7c6c-4ae6-85d1-c27b73152ba5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6b6f66ff94-kthkb\" (UID: \"7cc11d70-7c6c-4ae6-85d1-c27b73152ba5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-kthkb"
Sep 16 19:11:29.940792 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:29.940646    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7cc11d70-7c6c-4ae6-85d1-c27b73152ba5-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6b6f66ff94-kthkb\" (UID: \"7cc11d70-7c6c-4ae6-85d1-c27b73152ba5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-kthkb"
Sep 16 19:11:29.940792 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:29.940669    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cc11d70-7c6c-4ae6-85d1-c27b73152ba5-proxy-tls\") pod \"isvc-canary-promote-predictor-6b6f66ff94-kthkb\" (UID: \"7cc11d70-7c6c-4ae6-85d1-c27b73152ba5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-kthkb"
Sep 16 19:11:29.940792 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:29.940688    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6hh9\" (UniqueName: \"kubernetes.io/projected/7cc11d70-7c6c-4ae6-85d1-c27b73152ba5-kube-api-access-j6hh9\") pod \"isvc-canary-promote-predictor-6b6f66ff94-kthkb\" (UID: \"7cc11d70-7c6c-4ae6-85d1-c27b73152ba5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-kthkb"
Sep 16 19:11:29.940792 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:29.940708    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7cc11d70-7c6c-4ae6-85d1-c27b73152ba5-cabundle-cert\") pod \"isvc-canary-promote-predictor-6b6f66ff94-kthkb\" (UID: \"7cc11d70-7c6c-4ae6-85d1-c27b73152ba5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-kthkb"
Sep 16 19:11:30.041878 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:30.041789    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7cc11d70-7c6c-4ae6-85d1-c27b73152ba5-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6b6f66ff94-kthkb\" (UID: \"7cc11d70-7c6c-4ae6-85d1-c27b73152ba5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-kthkb"
Sep 16 19:11:30.041878 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:30.041827    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cc11d70-7c6c-4ae6-85d1-c27b73152ba5-proxy-tls\") pod \"isvc-canary-promote-predictor-6b6f66ff94-kthkb\" (UID: \"7cc11d70-7c6c-4ae6-85d1-c27b73152ba5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-kthkb"
Sep 16 19:11:30.041878 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:30.041849    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6hh9\" (UniqueName: \"kubernetes.io/projected/7cc11d70-7c6c-4ae6-85d1-c27b73152ba5-kube-api-access-j6hh9\") pod \"isvc-canary-promote-predictor-6b6f66ff94-kthkb\" (UID: \"7cc11d70-7c6c-4ae6-85d1-c27b73152ba5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-kthkb"
Sep 16 19:11:30.042342 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:30.041901    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7cc11d70-7c6c-4ae6-85d1-c27b73152ba5-cabundle-cert\") pod \"isvc-canary-promote-predictor-6b6f66ff94-kthkb\" (UID: \"7cc11d70-7c6c-4ae6-85d1-c27b73152ba5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-kthkb"
Sep 16 19:11:30.042342 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:30.042007    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7cc11d70-7c6c-4ae6-85d1-c27b73152ba5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6b6f66ff94-kthkb\" (UID: \"7cc11d70-7c6c-4ae6-85d1-c27b73152ba5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-kthkb"
Sep 16 19:11:30.042342 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:30.042314    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7cc11d70-7c6c-4ae6-85d1-c27b73152ba5-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6b6f66ff94-kthkb\" (UID: \"7cc11d70-7c6c-4ae6-85d1-c27b73152ba5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-kthkb"
Sep 16 19:11:30.042731 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:30.042711    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7cc11d70-7c6c-4ae6-85d1-c27b73152ba5-cabundle-cert\") pod \"isvc-canary-promote-predictor-6b6f66ff94-kthkb\" (UID: \"7cc11d70-7c6c-4ae6-85d1-c27b73152ba5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-kthkb"
Sep 16 19:11:30.042773 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:30.042711    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7cc11d70-7c6c-4ae6-85d1-c27b73152ba5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6b6f66ff94-kthkb\" (UID: \"7cc11d70-7c6c-4ae6-85d1-c27b73152ba5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-kthkb"
Sep 16 19:11:30.044529 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:30.044508    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cc11d70-7c6c-4ae6-85d1-c27b73152ba5-proxy-tls\") pod \"isvc-canary-promote-predictor-6b6f66ff94-kthkb\" (UID: \"7cc11d70-7c6c-4ae6-85d1-c27b73152ba5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-kthkb"
Sep 16 19:11:30.054320 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:30.054299    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6hh9\" (UniqueName: \"kubernetes.io/projected/7cc11d70-7c6c-4ae6-85d1-c27b73152ba5-kube-api-access-j6hh9\") pod \"isvc-canary-promote-predictor-6b6f66ff94-kthkb\" (UID: \"7cc11d70-7c6c-4ae6-85d1-c27b73152ba5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-kthkb"
Sep 16 19:11:30.171840 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:30.171797    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-kthkb"
Sep 16 19:11:30.345038 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:30.344940    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99d3c63d-b16e-4a7f-8929-cbd6357e5a80-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-prp88\" (UID: \"99d3c63d-b16e-4a7f-8929-cbd6357e5a80\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-prp88"
Sep 16 19:11:30.347341 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:30.347322    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99d3c63d-b16e-4a7f-8929-cbd6357e5a80-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-prp88\" (UID: \"99d3c63d-b16e-4a7f-8929-cbd6357e5a80\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-prp88"
Sep 16 19:11:30.376871 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:30.376838    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-866dcdbf79-5gnxs" podUID="6bf274d5-34ca-40b1-a1d9-3625e7cd7791" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 16 19:11:30.461499 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:30.461471    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-kthkb"]
Sep 16 19:11:30.463108 ip-10-0-129-190 kubenswrapper[2586]: W0916 19:11:30.463079    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cc11d70_7c6c_4ae6_85d1_c27b73152ba5.slice/crio-228188b50918ab4e46cbf20cc558b1727649eaa728d279e34a4c9cd78eab8ae9 WatchSource:0}: Error finding container 228188b50918ab4e46cbf20cc558b1727649eaa728d279e34a4c9cd78eab8ae9: Status 404 returned error can't find the container with id 228188b50918ab4e46cbf20cc558b1727649eaa728d279e34a4c9cd78eab8ae9
Sep 16 19:11:30.540124 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:30.540094    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-kthkb" event={"ID":"7cc11d70-7c6c-4ae6-85d1-c27b73152ba5","Type":"ContainerStarted","Data":"07fa13b16fda310eebec247ffd895a2a71b7d4c527a18c0320cc44a64c0a1ee2"}
Sep 16 19:11:30.540293 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:30.540160    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-kthkb" event={"ID":"7cc11d70-7c6c-4ae6-85d1-c27b73152ba5","Type":"ContainerStarted","Data":"228188b50918ab4e46cbf20cc558b1727649eaa728d279e34a4c9cd78eab8ae9"}
Sep 16 19:11:30.613742 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:30.613702    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-prp88"
Sep 16 19:11:30.913833 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:30.913707    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-prp88"]
Sep 16 19:11:30.916595 ip-10-0-129-190 kubenswrapper[2586]: W0916 19:11:30.916568    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99d3c63d_b16e_4a7f_8929_cbd6357e5a80.slice/crio-bc87cf99a6ec7319821424ec3416eba9b046fa443531e2ca1b79ad6d1c921982 WatchSource:0}: Error finding container bc87cf99a6ec7319821424ec3416eba9b046fa443531e2ca1b79ad6d1c921982: Status 404 returned error can't find the container with id bc87cf99a6ec7319821424ec3416eba9b046fa443531e2ca1b79ad6d1c921982
Sep 16 19:11:31.546341 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:31.546305    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-prp88" event={"ID":"99d3c63d-b16e-4a7f-8929-cbd6357e5a80","Type":"ContainerStarted","Data":"464409f1825fc60beacd220b013332d0a4a51dde765c55d4b8dfb4c1c10e9889"}
Sep 16 19:11:31.546942 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:31.546379    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-prp88" event={"ID":"99d3c63d-b16e-4a7f-8929-cbd6357e5a80","Type":"ContainerStarted","Data":"bc87cf99a6ec7319821424ec3416eba9b046fa443531e2ca1b79ad6d1c921982"}
Sep 16 19:11:34.170440 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:34.170396    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-rktzp" podUID="61efacc4-37a7-4825-9edd-c3737a03217f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused"
Sep 16 19:11:34.562975 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:34.562885    2586 generic.go:356] "Generic (PLEG): container finished" podID="7cc11d70-7c6c-4ae6-85d1-c27b73152ba5" containerID="07fa13b16fda310eebec247ffd895a2a71b7d4c527a18c0320cc44a64c0a1ee2" exitCode=0
Sep 16 19:11:34.562975 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:34.562923    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-kthkb" event={"ID":"7cc11d70-7c6c-4ae6-85d1-c27b73152ba5","Type":"ContainerDied","Data":"07fa13b16fda310eebec247ffd895a2a71b7d4c527a18c0320cc44a64c0a1ee2"}
Sep 16 19:11:35.570428 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:35.570392    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-kthkb" event={"ID":"7cc11d70-7c6c-4ae6-85d1-c27b73152ba5","Type":"ContainerStarted","Data":"2f828450d83c3b3868e54197008d9d3a9fb0d8c1a5c73ca6d62a10f557a6f03b"}
Sep 16 19:11:35.570922 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:35.570473    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-kthkb" event={"ID":"7cc11d70-7c6c-4ae6-85d1-c27b73152ba5","Type":"ContainerStarted","Data":"caf634bde1172f2e5f49ab93d0267eaa2f2877a0837cf3691fdc43f4d534346e"}
Sep 16 19:11:35.570922 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:35.570722    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-kthkb"
Sep 16 19:11:35.571972 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:35.571947    2586 generic.go:356] "Generic (PLEG): container finished" podID="99d3c63d-b16e-4a7f-8929-cbd6357e5a80" containerID="464409f1825fc60beacd220b013332d0a4a51dde765c55d4b8dfb4c1c10e9889" exitCode=0
Sep 16 19:11:35.572102 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:35.572021    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-prp88" event={"ID":"99d3c63d-b16e-4a7f-8929-cbd6357e5a80","Type":"ContainerDied","Data":"464409f1825fc60beacd220b013332d0a4a51dde765c55d4b8dfb4c1c10e9889"}
Sep 16 19:11:35.608442 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:35.608393    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-kthkb" podStartSLOduration=2.877476573 podStartE2EDuration="6.608377377s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.730900804s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:11:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:11:35.606366042 +0000 UTC m=+1871.050227619" watchObservedRunningTime="2026-09-16 19:11:35.608377377 +0000 UTC m=+1871.052238952"
Sep 16 19:11:36.578493 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:36.578452    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-prp88" event={"ID":"99d3c63d-b16e-4a7f-8929-cbd6357e5a80","Type":"ContainerStarted","Data":"c8d1642fd0f078cbde64a2bec99a0d2a8b85b2a32b272b8e58e36994f46fa39c"}
Sep 16 19:11:36.578909 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:36.578524    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-prp88" event={"ID":"99d3c63d-b16e-4a7f-8929-cbd6357e5a80","Type":"ContainerStarted","Data":"a759c24c00893e78b0f127f14b2cf9c27bad700037bec5472ca15ecb4b1c1709"}
Sep 16 19:11:36.578909 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:36.578752    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-kthkb"
Sep 16 19:11:36.579047 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:36.579030    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-prp88"
Sep 16 19:11:36.579196 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:36.579175    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-prp88"
Sep 16 19:11:36.580036 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:36.580010    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-kthkb" podUID="7cc11d70-7c6c-4ae6-85d1-c27b73152ba5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 16 19:11:36.580123 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:36.580044    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-prp88" podUID="99d3c63d-b16e-4a7f-8929-cbd6357e5a80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 16 19:11:36.623826 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:36.623780    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-prp88" podStartSLOduration=3.92591877 podStartE2EDuration="7.623763484s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.697844714s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:11:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:11:36.622161011 +0000 UTC m=+1872.066022586" watchObservedRunningTime="2026-09-16 19:11:36.623763484 +0000 UTC m=+1872.067625060"
Sep 16 19:11:37.583460 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:37.583401    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-prp88" podUID="99d3c63d-b16e-4a7f-8929-cbd6357e5a80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 16 19:11:37.583852 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:37.583462    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-kthkb" podUID="7cc11d70-7c6c-4ae6-85d1-c27b73152ba5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 16 19:11:39.169981 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:39.169934    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-rktzp" podUID="61efacc4-37a7-4825-9edd-c3737a03217f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused"
Sep 16 19:11:40.377447 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:40.377405    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-866dcdbf79-5gnxs" podUID="6bf274d5-34ca-40b1-a1d9-3625e7cd7791" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 16 19:11:42.588485 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:42.588448    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-prp88"
Sep 16 19:11:42.588914 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:42.588764    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-kthkb"
Sep 16 19:11:42.588914 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:42.588891    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-prp88" podUID="99d3c63d-b16e-4a7f-8929-cbd6357e5a80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 16 19:11:42.589260 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:42.589217    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-kthkb" podUID="7cc11d70-7c6c-4ae6-85d1-c27b73152ba5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 16 19:11:44.169766 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:44.169718    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-rktzp" podUID="61efacc4-37a7-4825-9edd-c3737a03217f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused"
Sep 16 19:11:49.169751 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:49.169698    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-rktzp" podUID="61efacc4-37a7-4825-9edd-c3737a03217f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused"
Sep 16 19:11:49.318282 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:49.318255    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-rktzp"
Sep 16 19:11:49.415060 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:49.415026    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61efacc4-37a7-4825-9edd-c3737a03217f-proxy-tls\") pod \"61efacc4-37a7-4825-9edd-c3737a03217f\" (UID: \"61efacc4-37a7-4825-9edd-c3737a03217f\") "
Sep 16 19:11:49.415060 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:49.415067    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x9sz7\" (UniqueName: \"kubernetes.io/projected/61efacc4-37a7-4825-9edd-c3737a03217f-kube-api-access-x9sz7\") pod \"61efacc4-37a7-4825-9edd-c3737a03217f\" (UID: \"61efacc4-37a7-4825-9edd-c3737a03217f\") "
Sep 16 19:11:49.415324 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:49.415132    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/61efacc4-37a7-4825-9edd-c3737a03217f-cabundle-cert\") pod \"61efacc4-37a7-4825-9edd-c3737a03217f\" (UID: \"61efacc4-37a7-4825-9edd-c3737a03217f\") "
Sep 16 19:11:49.415324 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:49.415154    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61efacc4-37a7-4825-9edd-c3737a03217f-kube-rbac-proxy-sar-config\") pod \"61efacc4-37a7-4825-9edd-c3737a03217f\" (UID: \"61efacc4-37a7-4825-9edd-c3737a03217f\") "
Sep 16 19:11:49.415324 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:49.415204    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61efacc4-37a7-4825-9edd-c3737a03217f-kserve-provision-location\") pod \"61efacc4-37a7-4825-9edd-c3737a03217f\" (UID: \"61efacc4-37a7-4825-9edd-c3737a03217f\") "
Sep 16 19:11:49.415563 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:49.415534    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61efacc4-37a7-4825-9edd-c3737a03217f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "61efacc4-37a7-4825-9edd-c3737a03217f" (UID: "61efacc4-37a7-4825-9edd-c3737a03217f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:49.415621 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:49.415561    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61efacc4-37a7-4825-9edd-c3737a03217f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "61efacc4-37a7-4825-9edd-c3737a03217f" (UID: "61efacc4-37a7-4825-9edd-c3737a03217f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:11:49.415621 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:49.415572    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61efacc4-37a7-4825-9edd-c3737a03217f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "61efacc4-37a7-4825-9edd-c3737a03217f" (UID: "61efacc4-37a7-4825-9edd-c3737a03217f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:11:49.417310 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:49.417283    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61efacc4-37a7-4825-9edd-c3737a03217f-kube-api-access-x9sz7" (OuterVolumeSpecName: "kube-api-access-x9sz7") pod "61efacc4-37a7-4825-9edd-c3737a03217f" (UID: "61efacc4-37a7-4825-9edd-c3737a03217f"). InnerVolumeSpecName "kube-api-access-x9sz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:11:49.417408 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:49.417325    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61efacc4-37a7-4825-9edd-c3737a03217f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "61efacc4-37a7-4825-9edd-c3737a03217f" (UID: "61efacc4-37a7-4825-9edd-c3737a03217f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:11:49.516403 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:49.516307    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/61efacc4-37a7-4825-9edd-c3737a03217f-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:11:49.516403 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:49.516338    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61efacc4-37a7-4825-9edd-c3737a03217f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:11:49.516403 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:49.516353    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61efacc4-37a7-4825-9edd-c3737a03217f-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:11:49.516403 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:49.516366    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61efacc4-37a7-4825-9edd-c3737a03217f-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:11:49.516403 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:49.516377    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9sz7\" (UniqueName: \"kubernetes.io/projected/61efacc4-37a7-4825-9edd-c3737a03217f-kube-api-access-x9sz7\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:11:49.645760 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:49.645727    2586 generic.go:356] "Generic (PLEG): container finished" podID="61efacc4-37a7-4825-9edd-c3737a03217f" containerID="d6eac8b79103b16b73e92bb4517c3bf39c49da622d50e46dc9119d024b911e3b" exitCode=137
Sep 16 19:11:49.645945 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:49.645793    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-rktzp"
Sep 16 19:11:49.645945 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:49.645798    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-rktzp" event={"ID":"61efacc4-37a7-4825-9edd-c3737a03217f","Type":"ContainerDied","Data":"d6eac8b79103b16b73e92bb4517c3bf39c49da622d50e46dc9119d024b911e3b"}
Sep 16 19:11:49.645945 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:49.645863    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-rktzp" event={"ID":"61efacc4-37a7-4825-9edd-c3737a03217f","Type":"ContainerDied","Data":"8adb8111464299456e6d78a8c31a503f8f4c6597fa0aa9a71f84fad20e119143"}
Sep 16 19:11:49.645945 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:49.645891    2586 scope.go:117] "RemoveContainer" containerID="d7c97af5a1430a0f69e8acaf4f109867afb154e68ee77dc3fc837e3b4b2ad8a5"
Sep 16 19:11:49.655519 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:49.655503    2586 scope.go:117] "RemoveContainer" containerID="d6eac8b79103b16b73e92bb4517c3bf39c49da622d50e46dc9119d024b911e3b"
Sep 16 19:11:49.663239 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:49.663214    2586 scope.go:117] "RemoveContainer" containerID="55f9d3016da973368b9208e07e43115593844dbfad5c230fec24fe706a5953a2"
Sep 16 19:11:49.670560 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:49.670540    2586 scope.go:117] "RemoveContainer" containerID="d7c97af5a1430a0f69e8acaf4f109867afb154e68ee77dc3fc837e3b4b2ad8a5"
Sep 16 19:11:49.677510 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:11:49.671216    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7c97af5a1430a0f69e8acaf4f109867afb154e68ee77dc3fc837e3b4b2ad8a5\": container with ID starting with d7c97af5a1430a0f69e8acaf4f109867afb154e68ee77dc3fc837e3b4b2ad8a5 not found: ID does not exist" containerID="d7c97af5a1430a0f69e8acaf4f109867afb154e68ee77dc3fc837e3b4b2ad8a5"
Sep 16 19:11:49.677510 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:49.671275    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7c97af5a1430a0f69e8acaf4f109867afb154e68ee77dc3fc837e3b4b2ad8a5"} err="failed to get container status \"d7c97af5a1430a0f69e8acaf4f109867afb154e68ee77dc3fc837e3b4b2ad8a5\": rpc error: code = NotFound desc = could not find container \"d7c97af5a1430a0f69e8acaf4f109867afb154e68ee77dc3fc837e3b4b2ad8a5\": container with ID starting with d7c97af5a1430a0f69e8acaf4f109867afb154e68ee77dc3fc837e3b4b2ad8a5 not found: ID does not exist"
Sep 16 19:11:49.677510 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:49.671304    2586 scope.go:117] "RemoveContainer" containerID="d6eac8b79103b16b73e92bb4517c3bf39c49da622d50e46dc9119d024b911e3b"
Sep 16 19:11:49.677758 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:11:49.677596    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6eac8b79103b16b73e92bb4517c3bf39c49da622d50e46dc9119d024b911e3b\": container with ID starting with d6eac8b79103b16b73e92bb4517c3bf39c49da622d50e46dc9119d024b911e3b not found: ID does not exist" containerID="d6eac8b79103b16b73e92bb4517c3bf39c49da622d50e46dc9119d024b911e3b"
Sep 16 19:11:49.677758 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:49.677662    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6eac8b79103b16b73e92bb4517c3bf39c49da622d50e46dc9119d024b911e3b"} err="failed to get container status \"d6eac8b79103b16b73e92bb4517c3bf39c49da622d50e46dc9119d024b911e3b\": rpc error: code = NotFound desc = could not find container \"d6eac8b79103b16b73e92bb4517c3bf39c49da622d50e46dc9119d024b911e3b\": container with ID starting with d6eac8b79103b16b73e92bb4517c3bf39c49da622d50e46dc9119d024b911e3b not found: ID does not exist"
Sep 16 19:11:49.677758 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:49.677692    2586 scope.go:117] "RemoveContainer" containerID="55f9d3016da973368b9208e07e43115593844dbfad5c230fec24fe706a5953a2"
Sep 16 19:11:49.678200 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:11:49.678172    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55f9d3016da973368b9208e07e43115593844dbfad5c230fec24fe706a5953a2\": container with ID starting with 55f9d3016da973368b9208e07e43115593844dbfad5c230fec24fe706a5953a2 not found: ID does not exist" containerID="55f9d3016da973368b9208e07e43115593844dbfad5c230fec24fe706a5953a2"
Sep 16 19:11:49.678320 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:49.678207    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55f9d3016da973368b9208e07e43115593844dbfad5c230fec24fe706a5953a2"} err="failed to get container status \"55f9d3016da973368b9208e07e43115593844dbfad5c230fec24fe706a5953a2\": rpc error: code = NotFound desc = could not find container \"55f9d3016da973368b9208e07e43115593844dbfad5c230fec24fe706a5953a2\": container with ID starting with 55f9d3016da973368b9208e07e43115593844dbfad5c230fec24fe706a5953a2 not found: ID does not exist"
Sep 16 19:11:49.679284 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:49.679262    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-rktzp"]
Sep 16 19:11:49.682640 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:49.682619    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-rktzp"]
Sep 16 19:11:50.377627 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:50.377582    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-866dcdbf79-5gnxs" podUID="6bf274d5-34ca-40b1-a1d9-3625e7cd7791" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 16 19:11:51.092860 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:51.092829    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61efacc4-37a7-4825-9edd-c3737a03217f" path="/var/lib/kubelet/pods/61efacc4-37a7-4825-9edd-c3737a03217f/volumes"
Sep 16 19:11:52.588884 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:52.588843    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-prp88" podUID="99d3c63d-b16e-4a7f-8929-cbd6357e5a80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 16 19:11:52.589398 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:11:52.589336    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-kthkb" podUID="7cc11d70-7c6c-4ae6-85d1-c27b73152ba5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 16 19:12:00.377009 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:00.376967    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-866dcdbf79-5gnxs" podUID="6bf274d5-34ca-40b1-a1d9-3625e7cd7791" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 16 19:12:02.589788 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:02.589747    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-prp88" podUID="99d3c63d-b16e-4a7f-8929-cbd6357e5a80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 16 19:12:02.590175 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:02.589747    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-kthkb" podUID="7cc11d70-7c6c-4ae6-85d1-c27b73152ba5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 16 19:12:05.089032 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:05.088990    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-866dcdbf79-5gnxs" podUID="6bf274d5-34ca-40b1-a1d9-3625e7cd7791" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 16 19:12:12.589792 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:12.589752    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-kthkb" podUID="7cc11d70-7c6c-4ae6-85d1-c27b73152ba5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 16 19:12:12.590187 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:12.589754    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-prp88" podUID="99d3c63d-b16e-4a7f-8929-cbd6357e5a80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 16 19:12:15.092687 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:15.092657    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-866dcdbf79-5gnxs"
Sep 16 19:12:18.245493 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:18.245456    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-866dcdbf79-5gnxs"]
Sep 16 19:12:18.245965 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:18.245728    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-866dcdbf79-5gnxs" podUID="6bf274d5-34ca-40b1-a1d9-3625e7cd7791" containerName="kserve-container" containerID="cri-o://4a66cc9bbde4f9528015a8f46864d3a0678b48f188a9704b632780bde3e92d85" gracePeriod=30
Sep 16 19:12:18.245965 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:18.245762    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-866dcdbf79-5gnxs" podUID="6bf274d5-34ca-40b1-a1d9-3625e7cd7791" containerName="kube-rbac-proxy" containerID="cri-o://4d81e5e7f69af18866d97cc053cdefa75494e0907704c57c31d6eee258dc0998" gracePeriod=30
Sep 16 19:12:18.791716 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:18.791679    2586 generic.go:356] "Generic (PLEG): container finished" podID="6bf274d5-34ca-40b1-a1d9-3625e7cd7791" containerID="4d81e5e7f69af18866d97cc053cdefa75494e0907704c57c31d6eee258dc0998" exitCode=2
Sep 16 19:12:18.791898 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:18.791757    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-866dcdbf79-5gnxs" event={"ID":"6bf274d5-34ca-40b1-a1d9-3625e7cd7791","Type":"ContainerDied","Data":"4d81e5e7f69af18866d97cc053cdefa75494e0907704c57c31d6eee258dc0998"}
Sep 16 19:12:20.371715 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:20.371659    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-866dcdbf79-5gnxs" podUID="6bf274d5-34ca-40b1-a1d9-3625e7cd7791" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
Sep 16 19:12:22.588902 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:22.588861    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-prp88" podUID="99d3c63d-b16e-4a7f-8929-cbd6357e5a80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 16 19:12:22.589374 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:22.589133    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-kthkb" podUID="7cc11d70-7c6c-4ae6-85d1-c27b73152ba5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 16 19:12:23.100993 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.100967    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-866dcdbf79-5gnxs"
Sep 16 19:12:23.198568 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.198488    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6bf274d5-34ca-40b1-a1d9-3625e7cd7791-kserve-provision-location\") pod \"6bf274d5-34ca-40b1-a1d9-3625e7cd7791\" (UID: \"6bf274d5-34ca-40b1-a1d9-3625e7cd7791\") "
Sep 16 19:12:23.198568 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.198551    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6bf274d5-34ca-40b1-a1d9-3625e7cd7791-kube-rbac-proxy-sar-config\") pod \"6bf274d5-34ca-40b1-a1d9-3625e7cd7791\" (UID: \"6bf274d5-34ca-40b1-a1d9-3625e7cd7791\") "
Sep 16 19:12:23.198754 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.198586    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4cwrf\" (UniqueName: \"kubernetes.io/projected/6bf274d5-34ca-40b1-a1d9-3625e7cd7791-kube-api-access-4cwrf\") pod \"6bf274d5-34ca-40b1-a1d9-3625e7cd7791\" (UID: \"6bf274d5-34ca-40b1-a1d9-3625e7cd7791\") "
Sep 16 19:12:23.198754 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.198653    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6bf274d5-34ca-40b1-a1d9-3625e7cd7791-proxy-tls\") pod \"6bf274d5-34ca-40b1-a1d9-3625e7cd7791\" (UID: \"6bf274d5-34ca-40b1-a1d9-3625e7cd7791\") "
Sep 16 19:12:23.198754 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.198693    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6bf274d5-34ca-40b1-a1d9-3625e7cd7791-cabundle-cert\") pod \"6bf274d5-34ca-40b1-a1d9-3625e7cd7791\" (UID: \"6bf274d5-34ca-40b1-a1d9-3625e7cd7791\") "
Sep 16 19:12:23.198864 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.198848    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bf274d5-34ca-40b1-a1d9-3625e7cd7791-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6bf274d5-34ca-40b1-a1d9-3625e7cd7791" (UID: "6bf274d5-34ca-40b1-a1d9-3625e7cd7791"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:12:23.198931 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.198917    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6bf274d5-34ca-40b1-a1d9-3625e7cd7791-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:12:23.199003 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.198969    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bf274d5-34ca-40b1-a1d9-3625e7cd7791-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6bf274d5-34ca-40b1-a1d9-3625e7cd7791" (UID: "6bf274d5-34ca-40b1-a1d9-3625e7cd7791"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:12:23.199081 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.199063    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bf274d5-34ca-40b1-a1d9-3625e7cd7791-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6bf274d5-34ca-40b1-a1d9-3625e7cd7791" (UID: "6bf274d5-34ca-40b1-a1d9-3625e7cd7791"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:12:23.200844 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.200816    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bf274d5-34ca-40b1-a1d9-3625e7cd7791-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6bf274d5-34ca-40b1-a1d9-3625e7cd7791" (UID: "6bf274d5-34ca-40b1-a1d9-3625e7cd7791"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:12:23.200844 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.200832    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bf274d5-34ca-40b1-a1d9-3625e7cd7791-kube-api-access-4cwrf" (OuterVolumeSpecName: "kube-api-access-4cwrf") pod "6bf274d5-34ca-40b1-a1d9-3625e7cd7791" (UID: "6bf274d5-34ca-40b1-a1d9-3625e7cd7791"). InnerVolumeSpecName "kube-api-access-4cwrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:12:23.300411 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.300369    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6bf274d5-34ca-40b1-a1d9-3625e7cd7791-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:12:23.300411 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.300414    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6bf274d5-34ca-40b1-a1d9-3625e7cd7791-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:12:23.300602 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.300431    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cwrf\" (UniqueName: \"kubernetes.io/projected/6bf274d5-34ca-40b1-a1d9-3625e7cd7791-kube-api-access-4cwrf\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:12:23.300602 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.300448    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6bf274d5-34ca-40b1-a1d9-3625e7cd7791-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:12:23.560019 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.559944    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j"]
Sep 16 19:12:23.560323 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.560310    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61efacc4-37a7-4825-9edd-c3737a03217f" containerName="storage-initializer"
Sep 16 19:12:23.560372 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.560325    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="61efacc4-37a7-4825-9edd-c3737a03217f" containerName="storage-initializer"
Sep 16 19:12:23.560372 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.560336    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61efacc4-37a7-4825-9edd-c3737a03217f" containerName="kserve-container"
Sep 16 19:12:23.560372 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.560341    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="61efacc4-37a7-4825-9edd-c3737a03217f" containerName="kserve-container"
Sep 16 19:12:23.560372 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.560348    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bf274d5-34ca-40b1-a1d9-3625e7cd7791" containerName="storage-initializer"
Sep 16 19:12:23.560372 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.560354    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bf274d5-34ca-40b1-a1d9-3625e7cd7791" containerName="storage-initializer"
Sep 16 19:12:23.560526 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.560375    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bf274d5-34ca-40b1-a1d9-3625e7cd7791" containerName="kserve-container"
Sep 16 19:12:23.560526 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.560381    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bf274d5-34ca-40b1-a1d9-3625e7cd7791" containerName="kserve-container"
Sep 16 19:12:23.560526 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.560387    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61efacc4-37a7-4825-9edd-c3737a03217f" containerName="kube-rbac-proxy"
Sep 16 19:12:23.560526 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.560393    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="61efacc4-37a7-4825-9edd-c3737a03217f" containerName="kube-rbac-proxy"
Sep 16 19:12:23.560526 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.560399    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bf274d5-34ca-40b1-a1d9-3625e7cd7791" containerName="kube-rbac-proxy"
Sep 16 19:12:23.560526 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.560405    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bf274d5-34ca-40b1-a1d9-3625e7cd7791" containerName="kube-rbac-proxy"
Sep 16 19:12:23.560526 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.560460    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bf274d5-34ca-40b1-a1d9-3625e7cd7791" containerName="kserve-container"
Sep 16 19:12:23.560526 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.560468    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="61efacc4-37a7-4825-9edd-c3737a03217f" containerName="kube-rbac-proxy"
Sep 16 19:12:23.560526 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.560475    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="61efacc4-37a7-4825-9edd-c3737a03217f" containerName="kserve-container"
Sep 16 19:12:23.560526 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.560483    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bf274d5-34ca-40b1-a1d9-3625e7cd7791" containerName="kube-rbac-proxy"
Sep 16 19:12:23.563748 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.563732    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j"
Sep 16 19:12:23.570121 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.570092    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Sep 16 19:12:23.570331 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.570123    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Sep 16 19:12:23.587339 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.587312    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j"]
Sep 16 19:12:23.602416 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.602387    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j\" (UID: \"f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j"
Sep 16 19:12:23.602776 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.602430    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j\" (UID: \"f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j"
Sep 16 19:12:23.602776 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.602456    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j\" (UID: \"f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j"
Sep 16 19:12:23.602776 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.602475    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swv2p\" (UniqueName: \"kubernetes.io/projected/f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2-kube-api-access-swv2p\") pod \"isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j\" (UID: \"f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j"
Sep 16 19:12:23.602776 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.602550    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j\" (UID: \"f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j"
Sep 16 19:12:23.703085 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.703049    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j\" (UID: \"f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j"
Sep 16 19:12:23.703310 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.703093    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j\" (UID: \"f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j"
Sep 16 19:12:23.703310 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.703215    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j\" (UID: \"f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j"
Sep 16 19:12:23.703310 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.703272    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j\" (UID: \"f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j"
Sep 16 19:12:23.703310 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.703301    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swv2p\" (UniqueName: \"kubernetes.io/projected/f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2-kube-api-access-swv2p\") pod \"isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j\" (UID: \"f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j"
Sep 16 19:12:23.703567 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.703544    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j\" (UID: \"f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j"
Sep 16 19:12:23.703897 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.703875    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j\" (UID: \"f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j"
Sep 16 19:12:23.704028 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.704010    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j\" (UID: \"f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j"
Sep 16 19:12:23.705666 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.705641    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j\" (UID: \"f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j"
Sep 16 19:12:23.732048 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.732023    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swv2p\" (UniqueName: \"kubernetes.io/projected/f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2-kube-api-access-swv2p\") pod \"isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j\" (UID: \"f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j"
Sep 16 19:12:23.818281 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.818179    2586 generic.go:356] "Generic (PLEG): container finished" podID="6bf274d5-34ca-40b1-a1d9-3625e7cd7791" containerID="4a66cc9bbde4f9528015a8f46864d3a0678b48f188a9704b632780bde3e92d85" exitCode=0
Sep 16 19:12:23.818281 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.818265    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-866dcdbf79-5gnxs"
Sep 16 19:12:23.818281 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.818269    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-866dcdbf79-5gnxs" event={"ID":"6bf274d5-34ca-40b1-a1d9-3625e7cd7791","Type":"ContainerDied","Data":"4a66cc9bbde4f9528015a8f46864d3a0678b48f188a9704b632780bde3e92d85"}
Sep 16 19:12:23.818482 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.818341    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-866dcdbf79-5gnxs" event={"ID":"6bf274d5-34ca-40b1-a1d9-3625e7cd7791","Type":"ContainerDied","Data":"cc86e480e586fe50fcfa968555792ff63efb11dfba055a47ace086b1f2f5b05b"}
Sep 16 19:12:23.818482 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.818379    2586 scope.go:117] "RemoveContainer" containerID="4d81e5e7f69af18866d97cc053cdefa75494e0907704c57c31d6eee258dc0998"
Sep 16 19:12:23.827717 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.827698    2586 scope.go:117] "RemoveContainer" containerID="4a66cc9bbde4f9528015a8f46864d3a0678b48f188a9704b632780bde3e92d85"
Sep 16 19:12:23.836926 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.836905    2586 scope.go:117] "RemoveContainer" containerID="cb250998980025b51f17392d415f6a3d442f8b3697a06a0b1797532ec7ac105c"
Sep 16 19:12:23.844600 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.844581    2586 scope.go:117] "RemoveContainer" containerID="4d81e5e7f69af18866d97cc053cdefa75494e0907704c57c31d6eee258dc0998"
Sep 16 19:12:23.844890 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:12:23.844863    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d81e5e7f69af18866d97cc053cdefa75494e0907704c57c31d6eee258dc0998\": container with ID starting with 4d81e5e7f69af18866d97cc053cdefa75494e0907704c57c31d6eee258dc0998 not found: ID does not exist" containerID="4d81e5e7f69af18866d97cc053cdefa75494e0907704c57c31d6eee258dc0998"
Sep 16 19:12:23.844961 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.844897    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d81e5e7f69af18866d97cc053cdefa75494e0907704c57c31d6eee258dc0998"} err="failed to get container status \"4d81e5e7f69af18866d97cc053cdefa75494e0907704c57c31d6eee258dc0998\": rpc error: code = NotFound desc = could not find container \"4d81e5e7f69af18866d97cc053cdefa75494e0907704c57c31d6eee258dc0998\": container with ID starting with 4d81e5e7f69af18866d97cc053cdefa75494e0907704c57c31d6eee258dc0998 not found: ID does not exist"
Sep 16 19:12:23.844961 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.844916    2586 scope.go:117] "RemoveContainer" containerID="4a66cc9bbde4f9528015a8f46864d3a0678b48f188a9704b632780bde3e92d85"
Sep 16 19:12:23.845158 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:12:23.845141    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a66cc9bbde4f9528015a8f46864d3a0678b48f188a9704b632780bde3e92d85\": container with ID starting with 4a66cc9bbde4f9528015a8f46864d3a0678b48f188a9704b632780bde3e92d85 not found: ID does not exist" containerID="4a66cc9bbde4f9528015a8f46864d3a0678b48f188a9704b632780bde3e92d85"
Sep 16 19:12:23.845204 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.845164    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a66cc9bbde4f9528015a8f46864d3a0678b48f188a9704b632780bde3e92d85"} err="failed to get container status \"4a66cc9bbde4f9528015a8f46864d3a0678b48f188a9704b632780bde3e92d85\": rpc error: code = NotFound desc = could not find container \"4a66cc9bbde4f9528015a8f46864d3a0678b48f188a9704b632780bde3e92d85\": container with ID starting with 4a66cc9bbde4f9528015a8f46864d3a0678b48f188a9704b632780bde3e92d85 not found: ID does not exist"
Sep 16 19:12:23.845204 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.845179    2586 scope.go:117] "RemoveContainer" containerID="cb250998980025b51f17392d415f6a3d442f8b3697a06a0b1797532ec7ac105c"
Sep 16 19:12:23.845453 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:12:23.845436    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb250998980025b51f17392d415f6a3d442f8b3697a06a0b1797532ec7ac105c\": container with ID starting with cb250998980025b51f17392d415f6a3d442f8b3697a06a0b1797532ec7ac105c not found: ID does not exist" containerID="cb250998980025b51f17392d415f6a3d442f8b3697a06a0b1797532ec7ac105c"
Sep 16 19:12:23.845512 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.845458    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb250998980025b51f17392d415f6a3d442f8b3697a06a0b1797532ec7ac105c"} err="failed to get container status \"cb250998980025b51f17392d415f6a3d442f8b3697a06a0b1797532ec7ac105c\": rpc error: code = NotFound desc = could not find container \"cb250998980025b51f17392d415f6a3d442f8b3697a06a0b1797532ec7ac105c\": container with ID starting with cb250998980025b51f17392d415f6a3d442f8b3697a06a0b1797532ec7ac105c not found: ID does not exist"
Sep 16 19:12:23.871114 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.871087    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-866dcdbf79-5gnxs"]
Sep 16 19:12:23.873369 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.873347    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-866dcdbf79-5gnxs"]
Sep 16 19:12:23.874632 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:23.874605    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j"
Sep 16 19:12:24.175779 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:24.175757    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j"]
Sep 16 19:12:24.178095 ip-10-0-129-190 kubenswrapper[2586]: W0916 19:12:24.178068    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf02e7f8b_f3d4_45b6_a35b_b8ec36b091c2.slice/crio-27c018eed356b48b7854ab613bdd42e5be2f018106ae1d011bcafb824b5627cb WatchSource:0}: Error finding container 27c018eed356b48b7854ab613bdd42e5be2f018106ae1d011bcafb824b5627cb: Status 404 returned error can't find the container with id 27c018eed356b48b7854ab613bdd42e5be2f018106ae1d011bcafb824b5627cb
Sep 16 19:12:24.826762 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:24.826724    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j" event={"ID":"f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2","Type":"ContainerStarted","Data":"2302c31f719c39687f376f5d179c35c22fa82a59ad25bef725773bece4643936"}
Sep 16 19:12:24.827145 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:24.826813    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j" event={"ID":"f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2","Type":"ContainerStarted","Data":"27c018eed356b48b7854ab613bdd42e5be2f018106ae1d011bcafb824b5627cb"}
Sep 16 19:12:25.094954 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:25.094873    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bf274d5-34ca-40b1-a1d9-3625e7cd7791" path="/var/lib/kubelet/pods/6bf274d5-34ca-40b1-a1d9-3625e7cd7791/volumes"
Sep 16 19:12:28.845798 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:28.845764    2586 generic.go:356] "Generic (PLEG): container finished" podID="f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2" containerID="2302c31f719c39687f376f5d179c35c22fa82a59ad25bef725773bece4643936" exitCode=0
Sep 16 19:12:28.845798 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:28.845803    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j" event={"ID":"f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2","Type":"ContainerDied","Data":"2302c31f719c39687f376f5d179c35c22fa82a59ad25bef725773bece4643936"}
Sep 16 19:12:29.852779 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:29.852742    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j" event={"ID":"f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2","Type":"ContainerStarted","Data":"ab34a762962723df6447ac53698f37a2c7d28459066de699107bfb0fc4e30576"}
Sep 16 19:12:29.853194 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:29.852829    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j" event={"ID":"f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2","Type":"ContainerStarted","Data":"9630ab9d74778a9a6afab321141bec50eebe08ff117e08887dafaef53808dda0"}
Sep 16 19:12:29.853194 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:29.853116    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j"
Sep 16 19:12:29.853352 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:29.853289    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j"
Sep 16 19:12:29.854666 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:29.854643    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j" podUID="f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 19:12:29.881025 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:29.880977    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j" podStartSLOduration=3.231980004 podStartE2EDuration="6.880956248s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.648976244s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:12:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:12:29.880458219 +0000 UTC m=+1925.324319818" watchObservedRunningTime="2026-09-16 19:12:29.880956248 +0000 UTC m=+1925.324817823"
Sep 16 19:12:30.857759 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:30.857720    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j" podUID="f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 19:12:32.589483 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:32.589441    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-kthkb" podUID="7cc11d70-7c6c-4ae6-85d1-c27b73152ba5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 16 19:12:32.589933 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:32.589445    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-prp88" podUID="99d3c63d-b16e-4a7f-8929-cbd6357e5a80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 16 19:12:35.862441 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:35.862404    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j"
Sep 16 19:12:35.863026 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:35.862999    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j" podUID="f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 19:12:42.589453 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:42.589410    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-kthkb" podUID="7cc11d70-7c6c-4ae6-85d1-c27b73152ba5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 16 19:12:42.589905 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:42.589410    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-prp88" podUID="99d3c63d-b16e-4a7f-8929-cbd6357e5a80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 16 19:12:45.863159 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:45.863113    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j" podUID="f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 19:12:48.087720 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:48.087677    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-kthkb" podUID="7cc11d70-7c6c-4ae6-85d1-c27b73152ba5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 16 19:12:51.087423 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:51.087385    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-prp88" podUID="99d3c63d-b16e-4a7f-8929-cbd6357e5a80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 16 19:12:55.863946 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:55.863901    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j" podUID="f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 19:12:58.088205 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:12:58.088170    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-kthkb"
Sep 16 19:13:01.092274 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:01.092217    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-prp88"
Sep 16 19:13:01.398156 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:01.398118    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-kthkb"]
Sep 16 19:13:01.398431 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:01.398406    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-kthkb" podUID="7cc11d70-7c6c-4ae6-85d1-c27b73152ba5" containerName="kserve-container" containerID="cri-o://caf634bde1172f2e5f49ab93d0267eaa2f2877a0837cf3691fdc43f4d534346e" gracePeriod=30
Sep 16 19:13:01.398486 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:01.398419    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-kthkb" podUID="7cc11d70-7c6c-4ae6-85d1-c27b73152ba5" containerName="kube-rbac-proxy" containerID="cri-o://2f828450d83c3b3868e54197008d9d3a9fb0d8c1a5c73ca6d62a10f557a6f03b" gracePeriod=30
Sep 16 19:13:02.007356 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:02.007316    2586 generic.go:356] "Generic (PLEG): container finished" podID="7cc11d70-7c6c-4ae6-85d1-c27b73152ba5" containerID="2f828450d83c3b3868e54197008d9d3a9fb0d8c1a5c73ca6d62a10f557a6f03b" exitCode=2
Sep 16 19:13:02.007356 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:02.007344    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-kthkb" event={"ID":"7cc11d70-7c6c-4ae6-85d1-c27b73152ba5","Type":"ContainerDied","Data":"2f828450d83c3b3868e54197008d9d3a9fb0d8c1a5c73ca6d62a10f557a6f03b"}
Sep 16 19:13:02.584366 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:02.584316    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-kthkb" podUID="7cc11d70-7c6c-4ae6-85d1-c27b73152ba5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.48:8643/healthz\": dial tcp 10.134.0.48:8643: connect: connection refused"
Sep 16 19:13:05.863868 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:05.863827    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j" podUID="f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 19:13:06.250345 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:06.250322    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-kthkb"
Sep 16 19:13:06.278466 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:06.278444    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7cc11d70-7c6c-4ae6-85d1-c27b73152ba5-kube-rbac-proxy-sar-config\") pod \"7cc11d70-7c6c-4ae6-85d1-c27b73152ba5\" (UID: \"7cc11d70-7c6c-4ae6-85d1-c27b73152ba5\") "
Sep 16 19:13:06.278542 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:06.278485    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cc11d70-7c6c-4ae6-85d1-c27b73152ba5-proxy-tls\") pod \"7cc11d70-7c6c-4ae6-85d1-c27b73152ba5\" (UID: \"7cc11d70-7c6c-4ae6-85d1-c27b73152ba5\") "
Sep 16 19:13:06.278542 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:06.278507    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j6hh9\" (UniqueName: \"kubernetes.io/projected/7cc11d70-7c6c-4ae6-85d1-c27b73152ba5-kube-api-access-j6hh9\") pod \"7cc11d70-7c6c-4ae6-85d1-c27b73152ba5\" (UID: \"7cc11d70-7c6c-4ae6-85d1-c27b73152ba5\") "
Sep 16 19:13:06.278542 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:06.278535    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7cc11d70-7c6c-4ae6-85d1-c27b73152ba5-kserve-provision-location\") pod \"7cc11d70-7c6c-4ae6-85d1-c27b73152ba5\" (UID: \"7cc11d70-7c6c-4ae6-85d1-c27b73152ba5\") "
Sep 16 19:13:06.278704 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:06.278562    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7cc11d70-7c6c-4ae6-85d1-c27b73152ba5-cabundle-cert\") pod \"7cc11d70-7c6c-4ae6-85d1-c27b73152ba5\" (UID: \"7cc11d70-7c6c-4ae6-85d1-c27b73152ba5\") "
Sep 16 19:13:06.278824 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:06.278793    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cc11d70-7c6c-4ae6-85d1-c27b73152ba5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7cc11d70-7c6c-4ae6-85d1-c27b73152ba5" (UID: "7cc11d70-7c6c-4ae6-85d1-c27b73152ba5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:13:06.278958 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:06.278863    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cc11d70-7c6c-4ae6-85d1-c27b73152ba5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7cc11d70-7c6c-4ae6-85d1-c27b73152ba5" (UID: "7cc11d70-7c6c-4ae6-85d1-c27b73152ba5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:06.279017 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:06.278949    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cc11d70-7c6c-4ae6-85d1-c27b73152ba5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7cc11d70-7c6c-4ae6-85d1-c27b73152ba5" (UID: "7cc11d70-7c6c-4ae6-85d1-c27b73152ba5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:13:06.280733 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:06.280711    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cc11d70-7c6c-4ae6-85d1-c27b73152ba5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7cc11d70-7c6c-4ae6-85d1-c27b73152ba5" (UID: "7cc11d70-7c6c-4ae6-85d1-c27b73152ba5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:13:06.280833 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:06.280768    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cc11d70-7c6c-4ae6-85d1-c27b73152ba5-kube-api-access-j6hh9" (OuterVolumeSpecName: "kube-api-access-j6hh9") pod "7cc11d70-7c6c-4ae6-85d1-c27b73152ba5" (UID: "7cc11d70-7c6c-4ae6-85d1-c27b73152ba5"). InnerVolumeSpecName "kube-api-access-j6hh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:13:06.379902 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:06.379799    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7cc11d70-7c6c-4ae6-85d1-c27b73152ba5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:13:06.379902 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:06.379823    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cc11d70-7c6c-4ae6-85d1-c27b73152ba5-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:13:06.379902 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:06.379832    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6hh9\" (UniqueName: \"kubernetes.io/projected/7cc11d70-7c6c-4ae6-85d1-c27b73152ba5-kube-api-access-j6hh9\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:13:06.379902 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:06.379840    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7cc11d70-7c6c-4ae6-85d1-c27b73152ba5-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:13:06.379902 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:06.379851    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7cc11d70-7c6c-4ae6-85d1-c27b73152ba5-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:13:07.032810 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:07.032770    2586 generic.go:356] "Generic (PLEG): container finished" podID="7cc11d70-7c6c-4ae6-85d1-c27b73152ba5" containerID="caf634bde1172f2e5f49ab93d0267eaa2f2877a0837cf3691fdc43f4d534346e" exitCode=0
Sep 16 19:13:07.033203 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:07.032864    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-kthkb" event={"ID":"7cc11d70-7c6c-4ae6-85d1-c27b73152ba5","Type":"ContainerDied","Data":"caf634bde1172f2e5f49ab93d0267eaa2f2877a0837cf3691fdc43f4d534346e"}
Sep 16 19:13:07.033203 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:07.032885    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-kthkb"
Sep 16 19:13:07.033203 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:07.032934    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-kthkb" event={"ID":"7cc11d70-7c6c-4ae6-85d1-c27b73152ba5","Type":"ContainerDied","Data":"228188b50918ab4e46cbf20cc558b1727649eaa728d279e34a4c9cd78eab8ae9"}
Sep 16 19:13:07.033203 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:07.032973    2586 scope.go:117] "RemoveContainer" containerID="2f828450d83c3b3868e54197008d9d3a9fb0d8c1a5c73ca6d62a10f557a6f03b"
Sep 16 19:13:07.042536 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:07.042519    2586 scope.go:117] "RemoveContainer" containerID="caf634bde1172f2e5f49ab93d0267eaa2f2877a0837cf3691fdc43f4d534346e"
Sep 16 19:13:07.050099 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:07.050079    2586 scope.go:117] "RemoveContainer" containerID="07fa13b16fda310eebec247ffd895a2a71b7d4c527a18c0320cc44a64c0a1ee2"
Sep 16 19:13:07.057212 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:07.057196    2586 scope.go:117] "RemoveContainer" containerID="2f828450d83c3b3868e54197008d9d3a9fb0d8c1a5c73ca6d62a10f557a6f03b"
Sep 16 19:13:07.057504 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:13:07.057485    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f828450d83c3b3868e54197008d9d3a9fb0d8c1a5c73ca6d62a10f557a6f03b\": container with ID starting with 2f828450d83c3b3868e54197008d9d3a9fb0d8c1a5c73ca6d62a10f557a6f03b not found: ID does not exist" containerID="2f828450d83c3b3868e54197008d9d3a9fb0d8c1a5c73ca6d62a10f557a6f03b"
Sep 16 19:13:07.057576 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:07.057518    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f828450d83c3b3868e54197008d9d3a9fb0d8c1a5c73ca6d62a10f557a6f03b"} err="failed to get container status \"2f828450d83c3b3868e54197008d9d3a9fb0d8c1a5c73ca6d62a10f557a6f03b\": rpc error: code = NotFound desc = could not find container \"2f828450d83c3b3868e54197008d9d3a9fb0d8c1a5c73ca6d62a10f557a6f03b\": container with ID starting with 2f828450d83c3b3868e54197008d9d3a9fb0d8c1a5c73ca6d62a10f557a6f03b not found: ID does not exist"
Sep 16 19:13:07.057576 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:07.057543    2586 scope.go:117] "RemoveContainer" containerID="caf634bde1172f2e5f49ab93d0267eaa2f2877a0837cf3691fdc43f4d534346e"
Sep 16 19:13:07.057803 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:13:07.057784    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caf634bde1172f2e5f49ab93d0267eaa2f2877a0837cf3691fdc43f4d534346e\": container with ID starting with caf634bde1172f2e5f49ab93d0267eaa2f2877a0837cf3691fdc43f4d534346e not found: ID does not exist" containerID="caf634bde1172f2e5f49ab93d0267eaa2f2877a0837cf3691fdc43f4d534346e"
Sep 16 19:13:07.057843 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:07.057810    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caf634bde1172f2e5f49ab93d0267eaa2f2877a0837cf3691fdc43f4d534346e"} err="failed to get container status \"caf634bde1172f2e5f49ab93d0267eaa2f2877a0837cf3691fdc43f4d534346e\": rpc error: code = NotFound desc = could not find container \"caf634bde1172f2e5f49ab93d0267eaa2f2877a0837cf3691fdc43f4d534346e\": container with ID starting with caf634bde1172f2e5f49ab93d0267eaa2f2877a0837cf3691fdc43f4d534346e not found: ID does not exist"
Sep 16 19:13:07.057843 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:07.057825    2586 scope.go:117] "RemoveContainer" containerID="07fa13b16fda310eebec247ffd895a2a71b7d4c527a18c0320cc44a64c0a1ee2"
Sep 16 19:13:07.058055 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:13:07.058039    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07fa13b16fda310eebec247ffd895a2a71b7d4c527a18c0320cc44a64c0a1ee2\": container with ID starting with 07fa13b16fda310eebec247ffd895a2a71b7d4c527a18c0320cc44a64c0a1ee2 not found: ID does not exist" containerID="07fa13b16fda310eebec247ffd895a2a71b7d4c527a18c0320cc44a64c0a1ee2"
Sep 16 19:13:07.058099 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:07.058061    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07fa13b16fda310eebec247ffd895a2a71b7d4c527a18c0320cc44a64c0a1ee2"} err="failed to get container status \"07fa13b16fda310eebec247ffd895a2a71b7d4c527a18c0320cc44a64c0a1ee2\": rpc error: code = NotFound desc = could not find container \"07fa13b16fda310eebec247ffd895a2a71b7d4c527a18c0320cc44a64c0a1ee2\": container with ID starting with 07fa13b16fda310eebec247ffd895a2a71b7d4c527a18c0320cc44a64c0a1ee2 not found: ID does not exist"
Sep 16 19:13:07.066679 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:07.066656    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-kthkb"]
Sep 16 19:13:07.070822 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:07.070801    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-kthkb"]
Sep 16 19:13:07.092330 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:07.092305    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cc11d70-7c6c-4ae6-85d1-c27b73152ba5" path="/var/lib/kubelet/pods/7cc11d70-7c6c-4ae6-85d1-c27b73152ba5/volumes"
Sep 16 19:13:15.863063 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:15.863011    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j" podUID="f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 19:13:25.863680 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:25.863632    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j" podUID="f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 19:13:35.863565 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:35.863525    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j" podUID="f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 19:13:45.864288 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:45.864249    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j"
Sep 16 19:13:53.489888 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:53.489852    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j"]
Sep 16 19:13:53.490296 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:53.490090    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j" podUID="f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2" containerName="kserve-container" containerID="cri-o://9630ab9d74778a9a6afab321141bec50eebe08ff117e08887dafaef53808dda0" gracePeriod=30
Sep 16 19:13:53.490296 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:53.490130    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j" podUID="f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2" containerName="kube-rbac-proxy" containerID="cri-o://ab34a762962723df6447ac53698f37a2c7d28459066de699107bfb0fc4e30576" gracePeriod=30
Sep 16 19:13:54.254922 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:54.254891    2586 generic.go:356] "Generic (PLEG): container finished" podID="f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2" containerID="ab34a762962723df6447ac53698f37a2c7d28459066de699107bfb0fc4e30576" exitCode=2
Sep 16 19:13:54.255099 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:54.254970    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j" event={"ID":"f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2","Type":"ContainerDied","Data":"ab34a762962723df6447ac53698f37a2c7d28459066de699107bfb0fc4e30576"}
Sep 16 19:13:55.858817 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:55.858782    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j" podUID="f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.49:8643/healthz\": dial tcp 10.134.0.49:8643: connect: connection refused"
Sep 16 19:13:55.863146 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:55.863118    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j" podUID="f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 19:13:58.239357 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.239331    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j"
Sep 16 19:13:58.281312 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.281278    2586 generic.go:356] "Generic (PLEG): container finished" podID="f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2" containerID="9630ab9d74778a9a6afab321141bec50eebe08ff117e08887dafaef53808dda0" exitCode=0
Sep 16 19:13:58.281470 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.281330    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j"
Sep 16 19:13:58.281470 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.281361    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j" event={"ID":"f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2","Type":"ContainerDied","Data":"9630ab9d74778a9a6afab321141bec50eebe08ff117e08887dafaef53808dda0"}
Sep 16 19:13:58.281470 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.281416    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j" event={"ID":"f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2","Type":"ContainerDied","Data":"27c018eed356b48b7854ab613bdd42e5be2f018106ae1d011bcafb824b5627cb"}
Sep 16 19:13:58.281470 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.281442    2586 scope.go:117] "RemoveContainer" containerID="ab34a762962723df6447ac53698f37a2c7d28459066de699107bfb0fc4e30576"
Sep 16 19:13:58.293296 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.293279    2586 scope.go:117] "RemoveContainer" containerID="9630ab9d74778a9a6afab321141bec50eebe08ff117e08887dafaef53808dda0"
Sep 16 19:13:58.300765 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.300745    2586 scope.go:117] "RemoveContainer" containerID="2302c31f719c39687f376f5d179c35c22fa82a59ad25bef725773bece4643936"
Sep 16 19:13:58.308182 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.308160    2586 scope.go:117] "RemoveContainer" containerID="ab34a762962723df6447ac53698f37a2c7d28459066de699107bfb0fc4e30576"
Sep 16 19:13:58.308493 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:13:58.308471    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab34a762962723df6447ac53698f37a2c7d28459066de699107bfb0fc4e30576\": container with ID starting with ab34a762962723df6447ac53698f37a2c7d28459066de699107bfb0fc4e30576 not found: ID does not exist" containerID="ab34a762962723df6447ac53698f37a2c7d28459066de699107bfb0fc4e30576"
Sep 16 19:13:58.308565 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.308507    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab34a762962723df6447ac53698f37a2c7d28459066de699107bfb0fc4e30576"} err="failed to get container status \"ab34a762962723df6447ac53698f37a2c7d28459066de699107bfb0fc4e30576\": rpc error: code = NotFound desc = could not find container \"ab34a762962723df6447ac53698f37a2c7d28459066de699107bfb0fc4e30576\": container with ID starting with ab34a762962723df6447ac53698f37a2c7d28459066de699107bfb0fc4e30576 not found: ID does not exist"
Sep 16 19:13:58.308565 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.308525    2586 scope.go:117] "RemoveContainer" containerID="9630ab9d74778a9a6afab321141bec50eebe08ff117e08887dafaef53808dda0"
Sep 16 19:13:58.308821 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:13:58.308803    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9630ab9d74778a9a6afab321141bec50eebe08ff117e08887dafaef53808dda0\": container with ID starting with 9630ab9d74778a9a6afab321141bec50eebe08ff117e08887dafaef53808dda0 not found: ID does not exist" containerID="9630ab9d74778a9a6afab321141bec50eebe08ff117e08887dafaef53808dda0"
Sep 16 19:13:58.308878 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.308828    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9630ab9d74778a9a6afab321141bec50eebe08ff117e08887dafaef53808dda0"} err="failed to get container status \"9630ab9d74778a9a6afab321141bec50eebe08ff117e08887dafaef53808dda0\": rpc error: code = NotFound desc = could not find container \"9630ab9d74778a9a6afab321141bec50eebe08ff117e08887dafaef53808dda0\": container with ID starting with 9630ab9d74778a9a6afab321141bec50eebe08ff117e08887dafaef53808dda0 not found: ID does not exist"
Sep 16 19:13:58.308878 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.308856    2586 scope.go:117] "RemoveContainer" containerID="2302c31f719c39687f376f5d179c35c22fa82a59ad25bef725773bece4643936"
Sep 16 19:13:58.309090 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:13:58.309075    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2302c31f719c39687f376f5d179c35c22fa82a59ad25bef725773bece4643936\": container with ID starting with 2302c31f719c39687f376f5d179c35c22fa82a59ad25bef725773bece4643936 not found: ID does not exist" containerID="2302c31f719c39687f376f5d179c35c22fa82a59ad25bef725773bece4643936"
Sep 16 19:13:58.309140 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.309094    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2302c31f719c39687f376f5d179c35c22fa82a59ad25bef725773bece4643936"} err="failed to get container status \"2302c31f719c39687f376f5d179c35c22fa82a59ad25bef725773bece4643936\": rpc error: code = NotFound desc = could not find container \"2302c31f719c39687f376f5d179c35c22fa82a59ad25bef725773bece4643936\": container with ID starting with 2302c31f719c39687f376f5d179c35c22fa82a59ad25bef725773bece4643936 not found: ID does not exist"
Sep 16 19:13:58.428338 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.428310    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2-kserve-provision-location\") pod \"f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2\" (UID: \"f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2\") "
Sep 16 19:13:58.428510 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.428346    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-swv2p\" (UniqueName: \"kubernetes.io/projected/f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2-kube-api-access-swv2p\") pod \"f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2\" (UID: \"f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2\") "
Sep 16 19:13:58.428510 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.428449    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2-kube-rbac-proxy-sar-config\") pod \"f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2\" (UID: \"f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2\") "
Sep 16 19:13:58.428510 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.428493    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2-cabundle-cert\") pod \"f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2\" (UID: \"f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2\") "
Sep 16 19:13:58.428692 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.428530    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2-proxy-tls\") pod \"f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2\" (UID: \"f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2\") "
Sep 16 19:13:58.428692 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.428663    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2" (UID: "f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:58.428803 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.428795    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:13:58.428897 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.428862    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2" (UID: "f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:13:58.428897 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.428872    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2" (UID: "f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:13:58.430550 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.430521    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2-kube-api-access-swv2p" (OuterVolumeSpecName: "kube-api-access-swv2p") pod "f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2" (UID: "f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2"). InnerVolumeSpecName "kube-api-access-swv2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:13:58.430633 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.430525    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2" (UID: "f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:13:58.529735 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.529693    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swv2p\" (UniqueName: \"kubernetes.io/projected/f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2-kube-api-access-swv2p\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:13:58.529735 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.529732    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:13:58.529932 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.529746    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:13:58.529932 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.529760    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:13:58.611923 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.611887    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j"]
Sep 16 19:13:58.621304 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.621277    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6c8c596f78-skz5j"]
Sep 16 19:13:58.730108 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.730071    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xch6v"]
Sep 16 19:13:58.730631 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.730579    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2" containerName="storage-initializer"
Sep 16 19:13:58.730631 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.730634    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2" containerName="storage-initializer"
Sep 16 19:13:58.730780 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.730648    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cc11d70-7c6c-4ae6-85d1-c27b73152ba5" containerName="kserve-container"
Sep 16 19:13:58.730780 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.730653    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cc11d70-7c6c-4ae6-85d1-c27b73152ba5" containerName="kserve-container"
Sep 16 19:13:58.730780 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.730661    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2" containerName="kube-rbac-proxy"
Sep 16 19:13:58.730780 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.730667    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2" containerName="kube-rbac-proxy"
Sep 16 19:13:58.730780 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.730680    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2" containerName="kserve-container"
Sep 16 19:13:58.730780 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.730688    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2" containerName="kserve-container"
Sep 16 19:13:58.730780 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.730717    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cc11d70-7c6c-4ae6-85d1-c27b73152ba5" containerName="kube-rbac-proxy"
Sep 16 19:13:58.730780 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.730725    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cc11d70-7c6c-4ae6-85d1-c27b73152ba5" containerName="kube-rbac-proxy"
Sep 16 19:13:58.730780 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.730736    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cc11d70-7c6c-4ae6-85d1-c27b73152ba5" containerName="storage-initializer"
Sep 16 19:13:58.730780 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.730741    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cc11d70-7c6c-4ae6-85d1-c27b73152ba5" containerName="storage-initializer"
Sep 16 19:13:58.731088 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.730817    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2" containerName="kube-rbac-proxy"
Sep 16 19:13:58.731088 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.730826    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2" containerName="kserve-container"
Sep 16 19:13:58.731088 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.730833    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cc11d70-7c6c-4ae6-85d1-c27b73152ba5" containerName="kube-rbac-proxy"
Sep 16 19:13:58.731088 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.730844    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cc11d70-7c6c-4ae6-85d1-c27b73152ba5" containerName="kserve-container"
Sep 16 19:13:58.735339 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.735322    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xch6v"
Sep 16 19:13:58.739865 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.739843    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-predictor-serving-cert\""
Sep 16 19:13:58.739970 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.739880    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
Sep 16 19:13:58.816127 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.816093    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xch6v"]
Sep 16 19:13:58.831322 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.831291    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e54bf78-94ea-4ce1-b76e-30b5f3110e2f-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-xch6v\" (UID: \"7e54bf78-94ea-4ce1-b76e-30b5f3110e2f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xch6v"
Sep 16 19:13:58.831475 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.831331    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e54bf78-94ea-4ce1-b76e-30b5f3110e2f-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-xch6v\" (UID: \"7e54bf78-94ea-4ce1-b76e-30b5f3110e2f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xch6v"
Sep 16 19:13:58.831475 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.831388    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e54bf78-94ea-4ce1-b76e-30b5f3110e2f-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-xch6v\" (UID: \"7e54bf78-94ea-4ce1-b76e-30b5f3110e2f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xch6v"
Sep 16 19:13:58.831475 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.831449    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e54bf78-94ea-4ce1-b76e-30b5f3110e2f-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-xch6v\" (UID: \"7e54bf78-94ea-4ce1-b76e-30b5f3110e2f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xch6v"
Sep 16 19:13:58.831593 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.831481    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dqwz\" (UniqueName: \"kubernetes.io/projected/7e54bf78-94ea-4ce1-b76e-30b5f3110e2f-kube-api-access-4dqwz\") pod \"isvc-tensorflow-predictor-547dfc5669-xch6v\" (UID: \"7e54bf78-94ea-4ce1-b76e-30b5f3110e2f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xch6v"
Sep 16 19:13:58.931874 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.931845    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dqwz\" (UniqueName: \"kubernetes.io/projected/7e54bf78-94ea-4ce1-b76e-30b5f3110e2f-kube-api-access-4dqwz\") pod \"isvc-tensorflow-predictor-547dfc5669-xch6v\" (UID: \"7e54bf78-94ea-4ce1-b76e-30b5f3110e2f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xch6v"
Sep 16 19:13:58.932052 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.931896    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e54bf78-94ea-4ce1-b76e-30b5f3110e2f-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-xch6v\" (UID: \"7e54bf78-94ea-4ce1-b76e-30b5f3110e2f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xch6v"
Sep 16 19:13:58.932052 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.931918    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e54bf78-94ea-4ce1-b76e-30b5f3110e2f-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-xch6v\" (UID: \"7e54bf78-94ea-4ce1-b76e-30b5f3110e2f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xch6v"
Sep 16 19:13:58.932052 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.931941    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e54bf78-94ea-4ce1-b76e-30b5f3110e2f-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-xch6v\" (UID: \"7e54bf78-94ea-4ce1-b76e-30b5f3110e2f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xch6v"
Sep 16 19:13:58.932052 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.931979    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e54bf78-94ea-4ce1-b76e-30b5f3110e2f-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-xch6v\" (UID: \"7e54bf78-94ea-4ce1-b76e-30b5f3110e2f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xch6v"
Sep 16 19:13:58.932331 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:13:58.932075    2586 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-tensorflow-predictor-serving-cert: secret "isvc-tensorflow-predictor-serving-cert" not found
Sep 16 19:13:58.932331 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:13:58.932134    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e54bf78-94ea-4ce1-b76e-30b5f3110e2f-proxy-tls podName:7e54bf78-94ea-4ce1-b76e-30b5f3110e2f nodeName:}" failed. No retries permitted until 2026-09-16 19:13:59.432115887 +0000 UTC m=+2014.875977440 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7e54bf78-94ea-4ce1-b76e-30b5f3110e2f-proxy-tls") pod "isvc-tensorflow-predictor-547dfc5669-xch6v" (UID: "7e54bf78-94ea-4ce1-b76e-30b5f3110e2f") : secret "isvc-tensorflow-predictor-serving-cert" not found
Sep 16 19:13:58.932437 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.932330    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e54bf78-94ea-4ce1-b76e-30b5f3110e2f-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-xch6v\" (UID: \"7e54bf78-94ea-4ce1-b76e-30b5f3110e2f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xch6v"
Sep 16 19:13:58.932597 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.932578    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e54bf78-94ea-4ce1-b76e-30b5f3110e2f-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-xch6v\" (UID: \"7e54bf78-94ea-4ce1-b76e-30b5f3110e2f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xch6v"
Sep 16 19:13:58.932646 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.932629    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e54bf78-94ea-4ce1-b76e-30b5f3110e2f-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-xch6v\" (UID: \"7e54bf78-94ea-4ce1-b76e-30b5f3110e2f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xch6v"
Sep 16 19:13:58.953189 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:58.953161    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dqwz\" (UniqueName: \"kubernetes.io/projected/7e54bf78-94ea-4ce1-b76e-30b5f3110e2f-kube-api-access-4dqwz\") pod \"isvc-tensorflow-predictor-547dfc5669-xch6v\" (UID: \"7e54bf78-94ea-4ce1-b76e-30b5f3110e2f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xch6v"
Sep 16 19:13:59.092428 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:59.092395    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2" path="/var/lib/kubelet/pods/f02e7f8b-f3d4-45b6-a35b-b8ec36b091c2/volumes"
Sep 16 19:13:59.436022 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:59.435984    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e54bf78-94ea-4ce1-b76e-30b5f3110e2f-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-xch6v\" (UID: \"7e54bf78-94ea-4ce1-b76e-30b5f3110e2f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xch6v"
Sep 16 19:13:59.438495 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:59.438477    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e54bf78-94ea-4ce1-b76e-30b5f3110e2f-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-xch6v\" (UID: \"7e54bf78-94ea-4ce1-b76e-30b5f3110e2f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xch6v"
Sep 16 19:13:59.646012 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:59.645974    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xch6v"
Sep 16 19:13:59.930778 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:13:59.930741    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xch6v"]
Sep 16 19:13:59.934126 ip-10-0-129-190 kubenswrapper[2586]: W0916 19:13:59.934098    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e54bf78_94ea_4ce1_b76e_30b5f3110e2f.slice/crio-f170d70e7c9554585a49c4ddb1cdefb357ba2290f8941f0be8519f6312324821 WatchSource:0}: Error finding container f170d70e7c9554585a49c4ddb1cdefb357ba2290f8941f0be8519f6312324821: Status 404 returned error can't find the container with id f170d70e7c9554585a49c4ddb1cdefb357ba2290f8941f0be8519f6312324821
Sep 16 19:14:00.295982 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:00.295894    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xch6v" event={"ID":"7e54bf78-94ea-4ce1-b76e-30b5f3110e2f","Type":"ContainerStarted","Data":"c042fb4d31840124c1301afa03a1be673ac93ffdc144c94bca16a72212e4139f"}
Sep 16 19:14:00.296143 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:00.296020    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xch6v" event={"ID":"7e54bf78-94ea-4ce1-b76e-30b5f3110e2f","Type":"ContainerStarted","Data":"f170d70e7c9554585a49c4ddb1cdefb357ba2290f8941f0be8519f6312324821"}
Sep 16 19:14:05.319802 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:05.319767    2586 generic.go:356] "Generic (PLEG): container finished" podID="7e54bf78-94ea-4ce1-b76e-30b5f3110e2f" containerID="c042fb4d31840124c1301afa03a1be673ac93ffdc144c94bca16a72212e4139f" exitCode=0
Sep 16 19:14:05.320202 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:05.319841    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xch6v" event={"ID":"7e54bf78-94ea-4ce1-b76e-30b5f3110e2f","Type":"ContainerDied","Data":"c042fb4d31840124c1301afa03a1be673ac93ffdc144c94bca16a72212e4139f"}
Sep 16 19:14:09.341885 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:09.341795    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xch6v" event={"ID":"7e54bf78-94ea-4ce1-b76e-30b5f3110e2f","Type":"ContainerStarted","Data":"85d601c372da0980f09cfab393ca6b97c06485edad668c6b3e9644ab2a7b6f6b"}
Sep 16 19:14:09.341885 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:09.341855    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xch6v" event={"ID":"7e54bf78-94ea-4ce1-b76e-30b5f3110e2f","Type":"ContainerStarted","Data":"7798c43426650094463dd99f4d7d660dcc0b29dc94f0d7857f3829199bd088a9"}
Sep 16 19:14:09.342401 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:09.342116    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xch6v"
Sep 16 19:14:09.382564 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:09.382506    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xch6v" podStartSLOduration=3.063422936 podStartE2EDuration="11.382486219s" totalImagesPullingTime="3.532145888s" totalInitContainerRuntime="4.786917395s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:13:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:14:09.380709725 +0000 UTC m=+2024.824571301" watchObservedRunningTime="2026-09-16 19:14:09.382486219 +0000 UTC m=+2024.826347794"
Sep 16 19:14:10.346252 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:10.346196    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xch6v"
Sep 16 19:14:10.347578 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:10.347549    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xch6v" podUID="7e54bf78-94ea-4ce1-b76e-30b5f3110e2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 16 19:14:11.350965 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:11.350920    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xch6v" podUID="7e54bf78-94ea-4ce1-b76e-30b5f3110e2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 16 19:14:16.355652 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:16.355624    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xch6v"
Sep 16 19:14:16.356167 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:16.356137    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xch6v" podUID="7e54bf78-94ea-4ce1-b76e-30b5f3110e2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 16 19:14:26.356464 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:26.356433    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xch6v"
Sep 16 19:14:35.234170 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:35.234130    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-prp88"]
Sep 16 19:14:35.234758 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:35.234390    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-prp88" podUID="99d3c63d-b16e-4a7f-8929-cbd6357e5a80" containerName="kserve-container" containerID="cri-o://a759c24c00893e78b0f127f14b2cf9c27bad700037bec5472ca15ecb4b1c1709" gracePeriod=30
Sep 16 19:14:35.234758 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:35.234474    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-prp88" podUID="99d3c63d-b16e-4a7f-8929-cbd6357e5a80" containerName="kube-rbac-proxy" containerID="cri-o://c8d1642fd0f078cbde64a2bec99a0d2a8b85b2a32b272b8e58e36994f46fa39c" gracePeriod=30
Sep 16 19:14:35.464504 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:35.464469    2586 generic.go:356] "Generic (PLEG): container finished" podID="99d3c63d-b16e-4a7f-8929-cbd6357e5a80" containerID="c8d1642fd0f078cbde64a2bec99a0d2a8b85b2a32b272b8e58e36994f46fa39c" exitCode=2
Sep 16 19:14:35.464684 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:35.464548    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-prp88" event={"ID":"99d3c63d-b16e-4a7f-8929-cbd6357e5a80","Type":"ContainerDied","Data":"c8d1642fd0f078cbde64a2bec99a0d2a8b85b2a32b272b8e58e36994f46fa39c"}
Sep 16 19:14:37.584347 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:37.584297    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-prp88" podUID="99d3c63d-b16e-4a7f-8929-cbd6357e5a80" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.47:8643/healthz\": dial tcp 10.134.0.47:8643: connect: connection refused"
Sep 16 19:14:39.735178 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:39.735140    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xch6v"]
Sep 16 19:14:39.735618 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:39.735397    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xch6v" podUID="7e54bf78-94ea-4ce1-b76e-30b5f3110e2f" containerName="kserve-container" containerID="cri-o://7798c43426650094463dd99f4d7d660dcc0b29dc94f0d7857f3829199bd088a9" gracePeriod=30
Sep 16 19:14:39.735618 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:39.735437    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xch6v" podUID="7e54bf78-94ea-4ce1-b76e-30b5f3110e2f" containerName="kube-rbac-proxy" containerID="cri-o://85d601c372da0980f09cfab393ca6b97c06485edad668c6b3e9644ab2a7b6f6b" gracePeriod=30
Sep 16 19:14:40.188464 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.188442    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-prp88"
Sep 16 19:14:40.292852 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.292771    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99d3c63d-b16e-4a7f-8929-cbd6357e5a80-kserve-provision-location\") pod \"99d3c63d-b16e-4a7f-8929-cbd6357e5a80\" (UID: \"99d3c63d-b16e-4a7f-8929-cbd6357e5a80\") "
Sep 16 19:14:40.292852 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.292808    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7fv5k\" (UniqueName: \"kubernetes.io/projected/99d3c63d-b16e-4a7f-8929-cbd6357e5a80-kube-api-access-7fv5k\") pod \"99d3c63d-b16e-4a7f-8929-cbd6357e5a80\" (UID: \"99d3c63d-b16e-4a7f-8929-cbd6357e5a80\") "
Sep 16 19:14:40.293076 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.292880    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99d3c63d-b16e-4a7f-8929-cbd6357e5a80-cabundle-cert\") pod \"99d3c63d-b16e-4a7f-8929-cbd6357e5a80\" (UID: \"99d3c63d-b16e-4a7f-8929-cbd6357e5a80\") "
Sep 16 19:14:40.293076 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.292932    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99d3c63d-b16e-4a7f-8929-cbd6357e5a80-proxy-tls\") pod \"99d3c63d-b16e-4a7f-8929-cbd6357e5a80\" (UID: \"99d3c63d-b16e-4a7f-8929-cbd6357e5a80\") "
Sep 16 19:14:40.293076 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.292973    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99d3c63d-b16e-4a7f-8929-cbd6357e5a80-kube-rbac-proxy-sar-config\") pod \"99d3c63d-b16e-4a7f-8929-cbd6357e5a80\" (UID: \"99d3c63d-b16e-4a7f-8929-cbd6357e5a80\") "
Sep 16 19:14:40.293224 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.293088    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99d3c63d-b16e-4a7f-8929-cbd6357e5a80-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "99d3c63d-b16e-4a7f-8929-cbd6357e5a80" (UID: "99d3c63d-b16e-4a7f-8929-cbd6357e5a80"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:40.293317 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.293254    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99d3c63d-b16e-4a7f-8929-cbd6357e5a80-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:14:40.293317 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.293292    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99d3c63d-b16e-4a7f-8929-cbd6357e5a80-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "99d3c63d-b16e-4a7f-8929-cbd6357e5a80" (UID: "99d3c63d-b16e-4a7f-8929-cbd6357e5a80"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:14:40.293418 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.293398    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99d3c63d-b16e-4a7f-8929-cbd6357e5a80-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "99d3c63d-b16e-4a7f-8929-cbd6357e5a80" (UID: "99d3c63d-b16e-4a7f-8929-cbd6357e5a80"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:14:40.295009 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.294986    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99d3c63d-b16e-4a7f-8929-cbd6357e5a80-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "99d3c63d-b16e-4a7f-8929-cbd6357e5a80" (UID: "99d3c63d-b16e-4a7f-8929-cbd6357e5a80"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:14:40.295098 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.295026    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99d3c63d-b16e-4a7f-8929-cbd6357e5a80-kube-api-access-7fv5k" (OuterVolumeSpecName: "kube-api-access-7fv5k") pod "99d3c63d-b16e-4a7f-8929-cbd6357e5a80" (UID: "99d3c63d-b16e-4a7f-8929-cbd6357e5a80"). InnerVolumeSpecName "kube-api-access-7fv5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:14:40.394350 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.394320    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fv5k\" (UniqueName: \"kubernetes.io/projected/99d3c63d-b16e-4a7f-8929-cbd6357e5a80-kube-api-access-7fv5k\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:14:40.394350 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.394347    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99d3c63d-b16e-4a7f-8929-cbd6357e5a80-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:14:40.394350 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.394357    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99d3c63d-b16e-4a7f-8929-cbd6357e5a80-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:14:40.394640 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.394365    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99d3c63d-b16e-4a7f-8929-cbd6357e5a80-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:14:40.493121 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.493083    2586 generic.go:356] "Generic (PLEG): container finished" podID="99d3c63d-b16e-4a7f-8929-cbd6357e5a80" containerID="a759c24c00893e78b0f127f14b2cf9c27bad700037bec5472ca15ecb4b1c1709" exitCode=0
Sep 16 19:14:40.493352 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.493160    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-prp88"
Sep 16 19:14:40.493352 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.493176    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-prp88" event={"ID":"99d3c63d-b16e-4a7f-8929-cbd6357e5a80","Type":"ContainerDied","Data":"a759c24c00893e78b0f127f14b2cf9c27bad700037bec5472ca15ecb4b1c1709"}
Sep 16 19:14:40.493352 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.493257    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-prp88" event={"ID":"99d3c63d-b16e-4a7f-8929-cbd6357e5a80","Type":"ContainerDied","Data":"bc87cf99a6ec7319821424ec3416eba9b046fa443531e2ca1b79ad6d1c921982"}
Sep 16 19:14:40.493352 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.493301    2586 scope.go:117] "RemoveContainer" containerID="c8d1642fd0f078cbde64a2bec99a0d2a8b85b2a32b272b8e58e36994f46fa39c"
Sep 16 19:14:40.496836 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.496805    2586 generic.go:356] "Generic (PLEG): container finished" podID="7e54bf78-94ea-4ce1-b76e-30b5f3110e2f" containerID="85d601c372da0980f09cfab393ca6b97c06485edad668c6b3e9644ab2a7b6f6b" exitCode=2
Sep 16 19:14:40.496968 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.496853    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xch6v" event={"ID":"7e54bf78-94ea-4ce1-b76e-30b5f3110e2f","Type":"ContainerDied","Data":"85d601c372da0980f09cfab393ca6b97c06485edad668c6b3e9644ab2a7b6f6b"}
Sep 16 19:14:40.504215 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.504195    2586 scope.go:117] "RemoveContainer" containerID="a759c24c00893e78b0f127f14b2cf9c27bad700037bec5472ca15ecb4b1c1709"
Sep 16 19:14:40.513489 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.513473    2586 scope.go:117] "RemoveContainer" containerID="464409f1825fc60beacd220b013332d0a4a51dde765c55d4b8dfb4c1c10e9889"
Sep 16 19:14:40.521186 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.521162    2586 scope.go:117] "RemoveContainer" containerID="c8d1642fd0f078cbde64a2bec99a0d2a8b85b2a32b272b8e58e36994f46fa39c"
Sep 16 19:14:40.521788 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:14:40.521755    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8d1642fd0f078cbde64a2bec99a0d2a8b85b2a32b272b8e58e36994f46fa39c\": container with ID starting with c8d1642fd0f078cbde64a2bec99a0d2a8b85b2a32b272b8e58e36994f46fa39c not found: ID does not exist" containerID="c8d1642fd0f078cbde64a2bec99a0d2a8b85b2a32b272b8e58e36994f46fa39c"
Sep 16 19:14:40.521895 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.521797    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8d1642fd0f078cbde64a2bec99a0d2a8b85b2a32b272b8e58e36994f46fa39c"} err="failed to get container status \"c8d1642fd0f078cbde64a2bec99a0d2a8b85b2a32b272b8e58e36994f46fa39c\": rpc error: code = NotFound desc = could not find container \"c8d1642fd0f078cbde64a2bec99a0d2a8b85b2a32b272b8e58e36994f46fa39c\": container with ID starting with c8d1642fd0f078cbde64a2bec99a0d2a8b85b2a32b272b8e58e36994f46fa39c not found: ID does not exist"
Sep 16 19:14:40.521895 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.521820    2586 scope.go:117] "RemoveContainer" containerID="a759c24c00893e78b0f127f14b2cf9c27bad700037bec5472ca15ecb4b1c1709"
Sep 16 19:14:40.522182 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:14:40.522151    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a759c24c00893e78b0f127f14b2cf9c27bad700037bec5472ca15ecb4b1c1709\": container with ID starting with a759c24c00893e78b0f127f14b2cf9c27bad700037bec5472ca15ecb4b1c1709 not found: ID does not exist" containerID="a759c24c00893e78b0f127f14b2cf9c27bad700037bec5472ca15ecb4b1c1709"
Sep 16 19:14:40.522296 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.522186    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a759c24c00893e78b0f127f14b2cf9c27bad700037bec5472ca15ecb4b1c1709"} err="failed to get container status \"a759c24c00893e78b0f127f14b2cf9c27bad700037bec5472ca15ecb4b1c1709\": rpc error: code = NotFound desc = could not find container \"a759c24c00893e78b0f127f14b2cf9c27bad700037bec5472ca15ecb4b1c1709\": container with ID starting with a759c24c00893e78b0f127f14b2cf9c27bad700037bec5472ca15ecb4b1c1709 not found: ID does not exist"
Sep 16 19:14:40.522296 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.522207    2586 scope.go:117] "RemoveContainer" containerID="464409f1825fc60beacd220b013332d0a4a51dde765c55d4b8dfb4c1c10e9889"
Sep 16 19:14:40.522578 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:14:40.522523    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"464409f1825fc60beacd220b013332d0a4a51dde765c55d4b8dfb4c1c10e9889\": container with ID starting with 464409f1825fc60beacd220b013332d0a4a51dde765c55d4b8dfb4c1c10e9889 not found: ID does not exist" containerID="464409f1825fc60beacd220b013332d0a4a51dde765c55d4b8dfb4c1c10e9889"
Sep 16 19:14:40.522578 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.522557    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"464409f1825fc60beacd220b013332d0a4a51dde765c55d4b8dfb4c1c10e9889"} err="failed to get container status \"464409f1825fc60beacd220b013332d0a4a51dde765c55d4b8dfb4c1c10e9889\": rpc error: code = NotFound desc = could not find container \"464409f1825fc60beacd220b013332d0a4a51dde765c55d4b8dfb4c1c10e9889\": container with ID starting with 464409f1825fc60beacd220b013332d0a4a51dde765c55d4b8dfb4c1c10e9889 not found: ID does not exist"
Sep 16 19:14:40.525030 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.525009    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-prp88"]
Sep 16 19:14:40.529198 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.529177    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-prp88"]
Sep 16 19:14:40.569696 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.569630    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs"]
Sep 16 19:14:40.569968 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.569955    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99d3c63d-b16e-4a7f-8929-cbd6357e5a80" containerName="kube-rbac-proxy"
Sep 16 19:14:40.570010 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.569969    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="99d3c63d-b16e-4a7f-8929-cbd6357e5a80" containerName="kube-rbac-proxy"
Sep 16 19:14:40.570010 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.569988    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99d3c63d-b16e-4a7f-8929-cbd6357e5a80" containerName="storage-initializer"
Sep 16 19:14:40.570010 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.569994    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="99d3c63d-b16e-4a7f-8929-cbd6357e5a80" containerName="storage-initializer"
Sep 16 19:14:40.570010 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.570004    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99d3c63d-b16e-4a7f-8929-cbd6357e5a80" containerName="kserve-container"
Sep 16 19:14:40.570010 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.570009    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="99d3c63d-b16e-4a7f-8929-cbd6357e5a80" containerName="kserve-container"
Sep 16 19:14:40.570162 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.570070    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="99d3c63d-b16e-4a7f-8929-cbd6357e5a80" containerName="kserve-container"
Sep 16 19:14:40.570162 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.570078    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="99d3c63d-b16e-4a7f-8929-cbd6357e5a80" containerName="kube-rbac-proxy"
Sep 16 19:14:40.581453 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.581423    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs"
Sep 16 19:14:40.584529 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.584507    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Sep 16 19:14:40.584776 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.584760    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 16 19:14:40.601502 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.601477    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs"]
Sep 16 19:14:40.681646 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.681608    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg"]
Sep 16 19:14:40.685287 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.685265    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg"
Sep 16 19:14:40.689028 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.689009    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 16 19:14:40.696839 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.696818    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4dd240d8-1a39-455d-bad7-8e89d291909a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs\" (UID: \"4dd240d8-1a39-455d-bad7-8e89d291909a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs"
Sep 16 19:14:40.696949 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.696852    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4dd240d8-1a39-455d-bad7-8e89d291909a-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs\" (UID: \"4dd240d8-1a39-455d-bad7-8e89d291909a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs"
Sep 16 19:14:40.697022 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.696956    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4dd240d8-1a39-455d-bad7-8e89d291909a-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs\" (UID: \"4dd240d8-1a39-455d-bad7-8e89d291909a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs"
Sep 16 19:14:40.697022 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.696988    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4dd240d8-1a39-455d-bad7-8e89d291909a-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs\" (UID: \"4dd240d8-1a39-455d-bad7-8e89d291909a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs"
Sep 16 19:14:40.697112 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.697018    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4gtq\" (UniqueName: \"kubernetes.io/projected/4dd240d8-1a39-455d-bad7-8e89d291909a-kube-api-access-d4gtq\") pod \"isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs\" (UID: \"4dd240d8-1a39-455d-bad7-8e89d291909a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs"
Sep 16 19:14:40.705817 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.705793    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg"]
Sep 16 19:14:40.798191 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.798155    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4dd240d8-1a39-455d-bad7-8e89d291909a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs\" (UID: \"4dd240d8-1a39-455d-bad7-8e89d291909a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs"
Sep 16 19:14:40.798191 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.798192    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4dd240d8-1a39-455d-bad7-8e89d291909a-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs\" (UID: \"4dd240d8-1a39-455d-bad7-8e89d291909a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs"
Sep 16 19:14:40.798757 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.798222    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72e4a42d-de46-4845-b1d4-6485b4edc309-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg\" (UID: \"72e4a42d-de46-4845-b1d4-6485b4edc309\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg"
Sep 16 19:14:40.798757 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.798291    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72e4a42d-de46-4845-b1d4-6485b4edc309-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg\" (UID: \"72e4a42d-de46-4845-b1d4-6485b4edc309\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg"
Sep 16 19:14:40.798757 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.798309    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nqkh\" (UniqueName: \"kubernetes.io/projected/72e4a42d-de46-4845-b1d4-6485b4edc309-kube-api-access-4nqkh\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg\" (UID: \"72e4a42d-de46-4845-b1d4-6485b4edc309\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg"
Sep 16 19:14:40.798757 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.798332    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4dd240d8-1a39-455d-bad7-8e89d291909a-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs\" (UID: \"4dd240d8-1a39-455d-bad7-8e89d291909a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs"
Sep 16 19:14:40.798757 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.798359    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4dd240d8-1a39-455d-bad7-8e89d291909a-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs\" (UID: \"4dd240d8-1a39-455d-bad7-8e89d291909a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs"
Sep 16 19:14:40.798757 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.798421    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72e4a42d-de46-4845-b1d4-6485b4edc309-proxy-tls\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg\" (UID: \"72e4a42d-de46-4845-b1d4-6485b4edc309\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg"
Sep 16 19:14:40.798757 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.798494    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4gtq\" (UniqueName: \"kubernetes.io/projected/4dd240d8-1a39-455d-bad7-8e89d291909a-kube-api-access-d4gtq\") pod \"isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs\" (UID: \"4dd240d8-1a39-455d-bad7-8e89d291909a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs"
Sep 16 19:14:40.798757 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.798530    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72e4a42d-de46-4845-b1d4-6485b4edc309-cabundle-cert\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg\" (UID: \"72e4a42d-de46-4845-b1d4-6485b4edc309\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg"
Sep 16 19:14:40.798757 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.798654    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4dd240d8-1a39-455d-bad7-8e89d291909a-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs\" (UID: \"4dd240d8-1a39-455d-bad7-8e89d291909a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs"
Sep 16 19:14:40.799180 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.799023    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4dd240d8-1a39-455d-bad7-8e89d291909a-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs\" (UID: \"4dd240d8-1a39-455d-bad7-8e89d291909a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs"
Sep 16 19:14:40.799180 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.799047    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4dd240d8-1a39-455d-bad7-8e89d291909a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs\" (UID: \"4dd240d8-1a39-455d-bad7-8e89d291909a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs"
Sep 16 19:14:40.800847 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.800827    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4dd240d8-1a39-455d-bad7-8e89d291909a-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs\" (UID: \"4dd240d8-1a39-455d-bad7-8e89d291909a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs"
Sep 16 19:14:40.810688 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.810666    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4gtq\" (UniqueName: \"kubernetes.io/projected/4dd240d8-1a39-455d-bad7-8e89d291909a-kube-api-access-d4gtq\") pod \"isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs\" (UID: \"4dd240d8-1a39-455d-bad7-8e89d291909a\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs"
Sep 16 19:14:40.891986 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.891949    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs"
Sep 16 19:14:40.899467 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.899440    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72e4a42d-de46-4845-b1d4-6485b4edc309-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg\" (UID: \"72e4a42d-de46-4845-b1d4-6485b4edc309\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg"
Sep 16 19:14:40.899467 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.899470    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nqkh\" (UniqueName: \"kubernetes.io/projected/72e4a42d-de46-4845-b1d4-6485b4edc309-kube-api-access-4nqkh\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg\" (UID: \"72e4a42d-de46-4845-b1d4-6485b4edc309\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg"
Sep 16 19:14:40.899675 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.899492    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72e4a42d-de46-4845-b1d4-6485b4edc309-proxy-tls\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg\" (UID: \"72e4a42d-de46-4845-b1d4-6485b4edc309\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg"
Sep 16 19:14:40.899675 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.899522    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72e4a42d-de46-4845-b1d4-6485b4edc309-cabundle-cert\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg\" (UID: \"72e4a42d-de46-4845-b1d4-6485b4edc309\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg"
Sep 16 19:14:40.899675 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.899590    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72e4a42d-de46-4845-b1d4-6485b4edc309-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg\" (UID: \"72e4a42d-de46-4845-b1d4-6485b4edc309\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg"
Sep 16 19:14:40.899966 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.899946    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72e4a42d-de46-4845-b1d4-6485b4edc309-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg\" (UID: \"72e4a42d-de46-4845-b1d4-6485b4edc309\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg"
Sep 16 19:14:40.900171 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.900148    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72e4a42d-de46-4845-b1d4-6485b4edc309-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg\" (UID: \"72e4a42d-de46-4845-b1d4-6485b4edc309\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg"
Sep 16 19:14:40.900254 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.900206    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72e4a42d-de46-4845-b1d4-6485b4edc309-cabundle-cert\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg\" (UID: \"72e4a42d-de46-4845-b1d4-6485b4edc309\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg"
Sep 16 19:14:40.902044 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.902020    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72e4a42d-de46-4845-b1d4-6485b4edc309-proxy-tls\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg\" (UID: \"72e4a42d-de46-4845-b1d4-6485b4edc309\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg"
Sep 16 19:14:40.913285 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.913257    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nqkh\" (UniqueName: \"kubernetes.io/projected/72e4a42d-de46-4845-b1d4-6485b4edc309-kube-api-access-4nqkh\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg\" (UID: \"72e4a42d-de46-4845-b1d4-6485b4edc309\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg"
Sep 16 19:14:40.995711 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:40.995678    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg"
Sep 16 19:14:41.092524 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:41.092494    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99d3c63d-b16e-4a7f-8929-cbd6357e5a80" path="/var/lib/kubelet/pods/99d3c63d-b16e-4a7f-8929-cbd6357e5a80/volumes"
Sep 16 19:14:41.180463 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:41.180392    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs"]
Sep 16 19:14:41.182835 ip-10-0-129-190 kubenswrapper[2586]: W0916 19:14:41.182811    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dd240d8_1a39_455d_bad7_8e89d291909a.slice/crio-a01b8eb04c9cb8feffc9483b04d13cf9d18b34747a13f7c3eda6adcb9455406d WatchSource:0}: Error finding container a01b8eb04c9cb8feffc9483b04d13cf9d18b34747a13f7c3eda6adcb9455406d: Status 404 returned error can't find the container with id a01b8eb04c9cb8feffc9483b04d13cf9d18b34747a13f7c3eda6adcb9455406d
Sep 16 19:14:41.286408 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:41.286324    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg"]
Sep 16 19:14:41.289047 ip-10-0-129-190 kubenswrapper[2586]: W0916 19:14:41.289026    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72e4a42d_de46_4845_b1d4_6485b4edc309.slice/crio-c068d40dbef0793d3acbeb0a28afd18fd88e3a666937c1e8c6d6fd3d05b39f7a WatchSource:0}: Error finding container c068d40dbef0793d3acbeb0a28afd18fd88e3a666937c1e8c6d6fd3d05b39f7a: Status 404 returned error can't find the container with id c068d40dbef0793d3acbeb0a28afd18fd88e3a666937c1e8c6d6fd3d05b39f7a
Sep 16 19:14:41.351775 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:41.351738    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xch6v" podUID="7e54bf78-94ea-4ce1-b76e-30b5f3110e2f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.50:8643/healthz\": dial tcp 10.134.0.50:8643: connect: connection refused"
Sep 16 19:14:41.505541 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:41.505433    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs" event={"ID":"4dd240d8-1a39-455d-bad7-8e89d291909a","Type":"ContainerStarted","Data":"4318346665dbf01a05af1d17974751102ca168697e63210366b2c66b61c7204e"}
Sep 16 19:14:41.505541 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:41.505519    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs" event={"ID":"4dd240d8-1a39-455d-bad7-8e89d291909a","Type":"ContainerStarted","Data":"a01b8eb04c9cb8feffc9483b04d13cf9d18b34747a13f7c3eda6adcb9455406d"}
Sep 16 19:14:41.508170 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:41.508141    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg" event={"ID":"72e4a42d-de46-4845-b1d4-6485b4edc309","Type":"ContainerStarted","Data":"16fbf89cc7d80f1ec1c9b9266603fa18c31b52916e93e7fd3c2e0562e34ed0c4"}
Sep 16 19:14:41.508342 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:41.508197    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg" event={"ID":"72e4a42d-de46-4845-b1d4-6485b4edc309","Type":"ContainerStarted","Data":"c068d40dbef0793d3acbeb0a28afd18fd88e3a666937c1e8c6d6fd3d05b39f7a"}
Sep 16 19:14:45.528216 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:45.528178    2586 generic.go:356] "Generic (PLEG): container finished" podID="72e4a42d-de46-4845-b1d4-6485b4edc309" containerID="16fbf89cc7d80f1ec1c9b9266603fa18c31b52916e93e7fd3c2e0562e34ed0c4" exitCode=0
Sep 16 19:14:45.528686 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:45.528207    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg" event={"ID":"72e4a42d-de46-4845-b1d4-6485b4edc309","Type":"ContainerDied","Data":"16fbf89cc7d80f1ec1c9b9266603fa18c31b52916e93e7fd3c2e0562e34ed0c4"}
Sep 16 19:14:45.529871 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:45.529852    2586 generic.go:356] "Generic (PLEG): container finished" podID="4dd240d8-1a39-455d-bad7-8e89d291909a" containerID="4318346665dbf01a05af1d17974751102ca168697e63210366b2c66b61c7204e" exitCode=0
Sep 16 19:14:45.529971 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:45.529883    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs" event={"ID":"4dd240d8-1a39-455d-bad7-8e89d291909a","Type":"ContainerDied","Data":"4318346665dbf01a05af1d17974751102ca168697e63210366b2c66b61c7204e"}
Sep 16 19:14:46.351971 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:46.351926    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xch6v" podUID="7e54bf78-94ea-4ce1-b76e-30b5f3110e2f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.50:8643/healthz\": dial tcp 10.134.0.50:8643: connect: connection refused"
Sep 16 19:14:46.538176 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:46.538137    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs" event={"ID":"4dd240d8-1a39-455d-bad7-8e89d291909a","Type":"ContainerStarted","Data":"a687e09e2359ec857d3af7832950107578c2ae4a7911b22fcd917b16babb9f67"}
Sep 16 19:14:46.538667 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:46.538213    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs" event={"ID":"4dd240d8-1a39-455d-bad7-8e89d291909a","Type":"ContainerStarted","Data":"8b5b9040bd81c207f7b0d7c337945367a8ce17dce424487f240e12428f7eae67"}
Sep 16 19:14:46.538667 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:46.538438    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs"
Sep 16 19:14:46.540571 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:46.540548    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg" event={"ID":"72e4a42d-de46-4845-b1d4-6485b4edc309","Type":"ContainerStarted","Data":"6869d7082d0e2dac07ecc568efa4d2462514b44818665eac4df96bd645f37cb9"}
Sep 16 19:14:46.540703 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:46.540603    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg" event={"ID":"72e4a42d-de46-4845-b1d4-6485b4edc309","Type":"ContainerStarted","Data":"8bb6df660c15293598f80c861dfbd9c5424d46745b766ebd96cb9dc23869840f"}
Sep 16 19:14:46.540816 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:46.540798    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg"
Sep 16 19:14:46.572493 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:46.572450    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs" podStartSLOduration=2.700136788 podStartE2EDuration="6.57243547s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.872298682s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:14:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:14:46.570700868 +0000 UTC m=+2062.014562449" watchObservedRunningTime="2026-09-16 19:14:46.57243547 +0000 UTC m=+2062.016297045"
Sep 16 19:14:46.614092 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:46.614014    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg" podStartSLOduration=3.024056 podStartE2EDuration="6.614000202s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.589944202s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:14:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:14:46.61279285 +0000 UTC m=+2062.056654426" watchObservedRunningTime="2026-09-16 19:14:46.614000202 +0000 UTC m=+2062.057861778"
Sep 16 19:14:47.545604 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:47.545512    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg"
Sep 16 19:14:47.545604 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:47.545578    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs"
Sep 16 19:14:47.546616 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:47.546588    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg" podUID="72e4a42d-de46-4845-b1d4-6485b4edc309" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 16 19:14:47.546693 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:47.546588    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs" podUID="4dd240d8-1a39-455d-bad7-8e89d291909a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 16 19:14:48.550366 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:48.550323    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs" podUID="4dd240d8-1a39-455d-bad7-8e89d291909a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 16 19:14:48.550366 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:48.550352    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg" podUID="72e4a42d-de46-4845-b1d4-6485b4edc309" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 16 19:14:51.352186 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:51.352141    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xch6v" podUID="7e54bf78-94ea-4ce1-b76e-30b5f3110e2f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.50:8643/healthz\": dial tcp 10.134.0.50:8643: connect: connection refused"
Sep 16 19:14:51.352673 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:51.352295    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xch6v"
Sep 16 19:14:53.555521 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:53.555492    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs"
Sep 16 19:14:53.555964 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:53.555813    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg"
Sep 16 19:14:53.555964 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:53.555936    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs" podUID="4dd240d8-1a39-455d-bad7-8e89d291909a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 16 19:14:53.556111 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:53.556087    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg" podUID="72e4a42d-de46-4845-b1d4-6485b4edc309" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 16 19:14:56.352166 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:14:56.352120    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xch6v" podUID="7e54bf78-94ea-4ce1-b76e-30b5f3110e2f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.50:8643/healthz\": dial tcp 10.134.0.50:8643: connect: connection refused"
Sep 16 19:15:01.351168 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:01.351123    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xch6v" podUID="7e54bf78-94ea-4ce1-b76e-30b5f3110e2f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.50:8643/healthz\": dial tcp 10.134.0.50:8643: connect: connection refused"
Sep 16 19:15:03.556810 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:03.556769    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg" podUID="72e4a42d-de46-4845-b1d4-6485b4edc309" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 16 19:15:03.557175 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:03.556781    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs" podUID="4dd240d8-1a39-455d-bad7-8e89d291909a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 16 19:15:06.351693 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:06.351641    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xch6v" podUID="7e54bf78-94ea-4ce1-b76e-30b5f3110e2f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.50:8643/healthz\": dial tcp 10.134.0.50:8643: connect: connection refused"
Sep 16 19:15:10.389422 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:10.389391    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xch6v"
Sep 16 19:15:10.447434 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:10.447401    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e54bf78-94ea-4ce1-b76e-30b5f3110e2f-proxy-tls\") pod \"7e54bf78-94ea-4ce1-b76e-30b5f3110e2f\" (UID: \"7e54bf78-94ea-4ce1-b76e-30b5f3110e2f\") "
Sep 16 19:15:10.447612 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:10.447443    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4dqwz\" (UniqueName: \"kubernetes.io/projected/7e54bf78-94ea-4ce1-b76e-30b5f3110e2f-kube-api-access-4dqwz\") pod \"7e54bf78-94ea-4ce1-b76e-30b5f3110e2f\" (UID: \"7e54bf78-94ea-4ce1-b76e-30b5f3110e2f\") "
Sep 16 19:15:10.447612 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:10.447501    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e54bf78-94ea-4ce1-b76e-30b5f3110e2f-kserve-provision-location\") pod \"7e54bf78-94ea-4ce1-b76e-30b5f3110e2f\" (UID: \"7e54bf78-94ea-4ce1-b76e-30b5f3110e2f\") "
Sep 16 19:15:10.447612 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:10.447536    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e54bf78-94ea-4ce1-b76e-30b5f3110e2f-kube-rbac-proxy-sar-config\") pod \"7e54bf78-94ea-4ce1-b76e-30b5f3110e2f\" (UID: \"7e54bf78-94ea-4ce1-b76e-30b5f3110e2f\") "
Sep 16 19:15:10.447789 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:10.447625    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e54bf78-94ea-4ce1-b76e-30b5f3110e2f-cabundle-cert\") pod \"7e54bf78-94ea-4ce1-b76e-30b5f3110e2f\" (UID: \"7e54bf78-94ea-4ce1-b76e-30b5f3110e2f\") "
Sep 16 19:15:10.448080 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:10.448013    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e54bf78-94ea-4ce1-b76e-30b5f3110e2f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7e54bf78-94ea-4ce1-b76e-30b5f3110e2f" (UID: "7e54bf78-94ea-4ce1-b76e-30b5f3110e2f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:15:10.448187 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:10.448079    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e54bf78-94ea-4ce1-b76e-30b5f3110e2f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7e54bf78-94ea-4ce1-b76e-30b5f3110e2f" (UID: "7e54bf78-94ea-4ce1-b76e-30b5f3110e2f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:15:10.449682 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:10.449651    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e54bf78-94ea-4ce1-b76e-30b5f3110e2f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7e54bf78-94ea-4ce1-b76e-30b5f3110e2f" (UID: "7e54bf78-94ea-4ce1-b76e-30b5f3110e2f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:15:10.449782 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:10.449725    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e54bf78-94ea-4ce1-b76e-30b5f3110e2f-kube-api-access-4dqwz" (OuterVolumeSpecName: "kube-api-access-4dqwz") pod "7e54bf78-94ea-4ce1-b76e-30b5f3110e2f" (UID: "7e54bf78-94ea-4ce1-b76e-30b5f3110e2f"). InnerVolumeSpecName "kube-api-access-4dqwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:15:10.458064 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:10.458039    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e54bf78-94ea-4ce1-b76e-30b5f3110e2f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7e54bf78-94ea-4ce1-b76e-30b5f3110e2f" (UID: "7e54bf78-94ea-4ce1-b76e-30b5f3110e2f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:10.549258 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:10.549140    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e54bf78-94ea-4ce1-b76e-30b5f3110e2f-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:15:10.549258 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:10.549170    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e54bf78-94ea-4ce1-b76e-30b5f3110e2f-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:15:10.549258 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:10.549178    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dqwz\" (UniqueName: \"kubernetes.io/projected/7e54bf78-94ea-4ce1-b76e-30b5f3110e2f-kube-api-access-4dqwz\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:15:10.549258 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:10.549187    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e54bf78-94ea-4ce1-b76e-30b5f3110e2f-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:15:10.549258 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:10.549197    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e54bf78-94ea-4ce1-b76e-30b5f3110e2f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:15:10.656054 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:10.656021    2586 generic.go:356] "Generic (PLEG): container finished" podID="7e54bf78-94ea-4ce1-b76e-30b5f3110e2f" containerID="7798c43426650094463dd99f4d7d660dcc0b29dc94f0d7857f3829199bd088a9" exitCode=137
Sep 16 19:15:10.656219 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:10.656079    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xch6v"
Sep 16 19:15:10.656219 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:10.656110    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xch6v" event={"ID":"7e54bf78-94ea-4ce1-b76e-30b5f3110e2f","Type":"ContainerDied","Data":"7798c43426650094463dd99f4d7d660dcc0b29dc94f0d7857f3829199bd088a9"}
Sep 16 19:15:10.656219 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:10.656173    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xch6v" event={"ID":"7e54bf78-94ea-4ce1-b76e-30b5f3110e2f","Type":"ContainerDied","Data":"f170d70e7c9554585a49c4ddb1cdefb357ba2290f8941f0be8519f6312324821"}
Sep 16 19:15:10.656219 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:10.656202    2586 scope.go:117] "RemoveContainer" containerID="85d601c372da0980f09cfab393ca6b97c06485edad668c6b3e9644ab2a7b6f6b"
Sep 16 19:15:10.665293 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:10.665217    2586 scope.go:117] "RemoveContainer" containerID="7798c43426650094463dd99f4d7d660dcc0b29dc94f0d7857f3829199bd088a9"
Sep 16 19:15:10.672662 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:10.672645    2586 scope.go:117] "RemoveContainer" containerID="c042fb4d31840124c1301afa03a1be673ac93ffdc144c94bca16a72212e4139f"
Sep 16 19:15:10.679765 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:10.679744    2586 scope.go:117] "RemoveContainer" containerID="85d601c372da0980f09cfab393ca6b97c06485edad668c6b3e9644ab2a7b6f6b"
Sep 16 19:15:10.680026 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:15:10.680008    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85d601c372da0980f09cfab393ca6b97c06485edad668c6b3e9644ab2a7b6f6b\": container with ID starting with 85d601c372da0980f09cfab393ca6b97c06485edad668c6b3e9644ab2a7b6f6b not found: ID does not exist" containerID="85d601c372da0980f09cfab393ca6b97c06485edad668c6b3e9644ab2a7b6f6b"
Sep 16 19:15:10.680075 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:10.680033    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85d601c372da0980f09cfab393ca6b97c06485edad668c6b3e9644ab2a7b6f6b"} err="failed to get container status \"85d601c372da0980f09cfab393ca6b97c06485edad668c6b3e9644ab2a7b6f6b\": rpc error: code = NotFound desc = could not find container \"85d601c372da0980f09cfab393ca6b97c06485edad668c6b3e9644ab2a7b6f6b\": container with ID starting with 85d601c372da0980f09cfab393ca6b97c06485edad668c6b3e9644ab2a7b6f6b not found: ID does not exist"
Sep 16 19:15:10.680075 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:10.680048    2586 scope.go:117] "RemoveContainer" containerID="7798c43426650094463dd99f4d7d660dcc0b29dc94f0d7857f3829199bd088a9"
Sep 16 19:15:10.680344 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:15:10.680323    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7798c43426650094463dd99f4d7d660dcc0b29dc94f0d7857f3829199bd088a9\": container with ID starting with 7798c43426650094463dd99f4d7d660dcc0b29dc94f0d7857f3829199bd088a9 not found: ID does not exist" containerID="7798c43426650094463dd99f4d7d660dcc0b29dc94f0d7857f3829199bd088a9"
Sep 16 19:15:10.680443 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:10.680353    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7798c43426650094463dd99f4d7d660dcc0b29dc94f0d7857f3829199bd088a9"} err="failed to get container status \"7798c43426650094463dd99f4d7d660dcc0b29dc94f0d7857f3829199bd088a9\": rpc error: code = NotFound desc = could not find container \"7798c43426650094463dd99f4d7d660dcc0b29dc94f0d7857f3829199bd088a9\": container with ID starting with 7798c43426650094463dd99f4d7d660dcc0b29dc94f0d7857f3829199bd088a9 not found: ID does not exist"
Sep 16 19:15:10.680443 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:10.680374    2586 scope.go:117] "RemoveContainer" containerID="c042fb4d31840124c1301afa03a1be673ac93ffdc144c94bca16a72212e4139f"
Sep 16 19:15:10.680686 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:15:10.680661    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c042fb4d31840124c1301afa03a1be673ac93ffdc144c94bca16a72212e4139f\": container with ID starting with c042fb4d31840124c1301afa03a1be673ac93ffdc144c94bca16a72212e4139f not found: ID does not exist" containerID="c042fb4d31840124c1301afa03a1be673ac93ffdc144c94bca16a72212e4139f"
Sep 16 19:15:10.680756 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:10.680692    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c042fb4d31840124c1301afa03a1be673ac93ffdc144c94bca16a72212e4139f"} err="failed to get container status \"c042fb4d31840124c1301afa03a1be673ac93ffdc144c94bca16a72212e4139f\": rpc error: code = NotFound desc = could not find container \"c042fb4d31840124c1301afa03a1be673ac93ffdc144c94bca16a72212e4139f\": container with ID starting with c042fb4d31840124c1301afa03a1be673ac93ffdc144c94bca16a72212e4139f not found: ID does not exist"
Sep 16 19:15:10.685755 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:10.685733    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xch6v"]
Sep 16 19:15:10.687032 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:10.687013    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-xch6v"]
Sep 16 19:15:11.092057 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:11.092023    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e54bf78-94ea-4ce1-b76e-30b5f3110e2f" path="/var/lib/kubelet/pods/7e54bf78-94ea-4ce1-b76e-30b5f3110e2f/volumes"
Sep 16 19:15:13.556884 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:13.556847    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg" podUID="72e4a42d-de46-4845-b1d4-6485b4edc309" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 16 19:15:13.557285 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:13.556849    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs" podUID="4dd240d8-1a39-455d-bad7-8e89d291909a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 16 19:15:23.556401 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:23.556353    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs" podUID="4dd240d8-1a39-455d-bad7-8e89d291909a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 16 19:15:23.556401 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:23.556372    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg" podUID="72e4a42d-de46-4845-b1d4-6485b4edc309" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 16 19:15:31.520310 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:31.520274    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-f8flm"]
Sep 16 19:15:31.520715 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:31.520677    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e54bf78-94ea-4ce1-b76e-30b5f3110e2f" containerName="storage-initializer"
Sep 16 19:15:31.520715 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:31.520698    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e54bf78-94ea-4ce1-b76e-30b5f3110e2f" containerName="storage-initializer"
Sep 16 19:15:31.520803 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:31.520715    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e54bf78-94ea-4ce1-b76e-30b5f3110e2f" containerName="kube-rbac-proxy"
Sep 16 19:15:31.520803 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:31.520724    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e54bf78-94ea-4ce1-b76e-30b5f3110e2f" containerName="kube-rbac-proxy"
Sep 16 19:15:31.520803 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:31.520736    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e54bf78-94ea-4ce1-b76e-30b5f3110e2f" containerName="kserve-container"
Sep 16 19:15:31.520803 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:31.520742    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e54bf78-94ea-4ce1-b76e-30b5f3110e2f" containerName="kserve-container"
Sep 16 19:15:31.520941 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:31.520821    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e54bf78-94ea-4ce1-b76e-30b5f3110e2f" containerName="kube-rbac-proxy"
Sep 16 19:15:31.520941 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:31.520834    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e54bf78-94ea-4ce1-b76e-30b5f3110e2f" containerName="kserve-container"
Sep 16 19:15:31.524053 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:31.524033    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-f8flm"
Sep 16 19:15:31.527161 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:31.527136    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-kube-rbac-proxy-sar-config\""
Sep 16 19:15:31.527551 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:31.527532    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-predictor-serving-cert\""
Sep 16 19:15:31.539058 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:31.539027    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-f8flm"]
Sep 16 19:15:31.639787 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:31.639745    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwcds\" (UniqueName: \"kubernetes.io/projected/81b89f31-91f5-4d65-85c3-aba3321b5cd5-kube-api-access-bwcds\") pod \"isvc-triton-predictor-5d7db58676-f8flm\" (UID: \"81b89f31-91f5-4d65-85c3-aba3321b5cd5\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-f8flm"
Sep 16 19:15:31.639787 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:31.639785    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81b89f31-91f5-4d65-85c3-aba3321b5cd5-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-f8flm\" (UID: \"81b89f31-91f5-4d65-85c3-aba3321b5cd5\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-f8flm"
Sep 16 19:15:31.640035 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:31.639891    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81b89f31-91f5-4d65-85c3-aba3321b5cd5-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-f8flm\" (UID: \"81b89f31-91f5-4d65-85c3-aba3321b5cd5\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-f8flm"
Sep 16 19:15:31.640035 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:31.639928    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81b89f31-91f5-4d65-85c3-aba3321b5cd5-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-f8flm\" (UID: \"81b89f31-91f5-4d65-85c3-aba3321b5cd5\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-f8flm"
Sep 16 19:15:31.640035 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:31.639964    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81b89f31-91f5-4d65-85c3-aba3321b5cd5-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-f8flm\" (UID: \"81b89f31-91f5-4d65-85c3-aba3321b5cd5\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-f8flm"
Sep 16 19:15:31.741104 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:31.741069    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwcds\" (UniqueName: \"kubernetes.io/projected/81b89f31-91f5-4d65-85c3-aba3321b5cd5-kube-api-access-bwcds\") pod \"isvc-triton-predictor-5d7db58676-f8flm\" (UID: \"81b89f31-91f5-4d65-85c3-aba3321b5cd5\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-f8flm"
Sep 16 19:15:31.741304 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:31.741108    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81b89f31-91f5-4d65-85c3-aba3321b5cd5-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-f8flm\" (UID: \"81b89f31-91f5-4d65-85c3-aba3321b5cd5\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-f8flm"
Sep 16 19:15:31.741304 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:31.741156    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81b89f31-91f5-4d65-85c3-aba3321b5cd5-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-f8flm\" (UID: \"81b89f31-91f5-4d65-85c3-aba3321b5cd5\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-f8flm"
Sep 16 19:15:31.741304 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:31.741170    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81b89f31-91f5-4d65-85c3-aba3321b5cd5-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-f8flm\" (UID: \"81b89f31-91f5-4d65-85c3-aba3321b5cd5\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-f8flm"
Sep 16 19:15:31.741304 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:31.741191    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81b89f31-91f5-4d65-85c3-aba3321b5cd5-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-f8flm\" (UID: \"81b89f31-91f5-4d65-85c3-aba3321b5cd5\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-f8flm"
Sep 16 19:15:31.741701 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:31.741675    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81b89f31-91f5-4d65-85c3-aba3321b5cd5-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-f8flm\" (UID: \"81b89f31-91f5-4d65-85c3-aba3321b5cd5\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-f8flm"
Sep 16 19:15:31.741861 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:31.741839    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81b89f31-91f5-4d65-85c3-aba3321b5cd5-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-f8flm\" (UID: \"81b89f31-91f5-4d65-85c3-aba3321b5cd5\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-f8flm"
Sep 16 19:15:31.741949 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:31.741929    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81b89f31-91f5-4d65-85c3-aba3321b5cd5-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-f8flm\" (UID: \"81b89f31-91f5-4d65-85c3-aba3321b5cd5\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-f8flm"
Sep 16 19:15:31.743759 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:31.743735    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81b89f31-91f5-4d65-85c3-aba3321b5cd5-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-f8flm\" (UID: \"81b89f31-91f5-4d65-85c3-aba3321b5cd5\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-f8flm"
Sep 16 19:15:31.753356 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:31.753334    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwcds\" (UniqueName: \"kubernetes.io/projected/81b89f31-91f5-4d65-85c3-aba3321b5cd5-kube-api-access-bwcds\") pod \"isvc-triton-predictor-5d7db58676-f8flm\" (UID: \"81b89f31-91f5-4d65-85c3-aba3321b5cd5\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-f8flm"
Sep 16 19:15:31.835394 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:31.835305    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-f8flm"
Sep 16 19:15:32.114411 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:32.114385    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-f8flm"]
Sep 16 19:15:32.117037 ip-10-0-129-190 kubenswrapper[2586]: W0916 19:15:32.117012    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81b89f31_91f5_4d65_85c3_aba3321b5cd5.slice/crio-76456ad3bfdd38e74fe26231deb2d91d23547d4e825770bace4caddf5617e17b WatchSource:0}: Error finding container 76456ad3bfdd38e74fe26231deb2d91d23547d4e825770bace4caddf5617e17b: Status 404 returned error can't find the container with id 76456ad3bfdd38e74fe26231deb2d91d23547d4e825770bace4caddf5617e17b
Sep 16 19:15:32.767600 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:32.767560    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-f8flm" event={"ID":"81b89f31-91f5-4d65-85c3-aba3321b5cd5","Type":"ContainerStarted","Data":"66ee273b95f9b0eeb39c2fa455cff83b143e7e8783c6a875246074ca3c135824"}
Sep 16 19:15:32.768088 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:32.767661    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-f8flm" event={"ID":"81b89f31-91f5-4d65-85c3-aba3321b5cd5","Type":"ContainerStarted","Data":"76456ad3bfdd38e74fe26231deb2d91d23547d4e825770bace4caddf5617e17b"}
Sep 16 19:15:33.556880 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:33.556839    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs" podUID="4dd240d8-1a39-455d-bad7-8e89d291909a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 16 19:15:33.557050 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:33.556843    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg" podUID="72e4a42d-de46-4845-b1d4-6485b4edc309" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 16 19:15:36.789589 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:36.789555    2586 generic.go:356] "Generic (PLEG): container finished" podID="81b89f31-91f5-4d65-85c3-aba3321b5cd5" containerID="66ee273b95f9b0eeb39c2fa455cff83b143e7e8783c6a875246074ca3c135824" exitCode=0
Sep 16 19:15:36.790076 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:36.789630    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-f8flm" event={"ID":"81b89f31-91f5-4d65-85c3-aba3321b5cd5","Type":"ContainerDied","Data":"66ee273b95f9b0eeb39c2fa455cff83b143e7e8783c6a875246074ca3c135824"}
Sep 16 19:15:36.790899 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:36.790884    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:15:43.556695 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:43.556201    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg" podUID="72e4a42d-de46-4845-b1d4-6485b4edc309" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 16 19:15:43.556695 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:43.556192    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs" podUID="4dd240d8-1a39-455d-bad7-8e89d291909a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 16 19:15:53.556265 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:53.556149    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs" podUID="4dd240d8-1a39-455d-bad7-8e89d291909a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 16 19:15:53.556265 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:15:53.556216    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg" podUID="72e4a42d-de46-4845-b1d4-6485b4edc309" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 16 19:16:00.086950 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:00.086903    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs" podUID="4dd240d8-1a39-455d-bad7-8e89d291909a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 16 19:16:03.556696 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:03.556649    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg" podUID="72e4a42d-de46-4845-b1d4-6485b4edc309" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 16 19:16:10.087084 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:10.087034    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs" podUID="4dd240d8-1a39-455d-bad7-8e89d291909a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 16 19:16:13.557370 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:13.557211    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg"
Sep 16 19:16:20.088407 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:20.088110    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs"
Sep 16 19:16:20.286408 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:20.286370    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg"]
Sep 16 19:16:20.286693 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:20.286663    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg" podUID="72e4a42d-de46-4845-b1d4-6485b4edc309" containerName="kserve-container" containerID="cri-o://8bb6df660c15293598f80c861dfbd9c5424d46745b766ebd96cb9dc23869840f" gracePeriod=30
Sep 16 19:16:20.286852 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:20.286709    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg" podUID="72e4a42d-de46-4845-b1d4-6485b4edc309" containerName="kube-rbac-proxy" containerID="cri-o://6869d7082d0e2dac07ecc568efa4d2462514b44818665eac4df96bd645f37cb9" gracePeriod=30
Sep 16 19:16:20.501541 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:20.501502    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs"]
Sep 16 19:16:20.582103 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:20.582056    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9"]
Sep 16 19:16:20.601537 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:20.601016    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9"]
Sep 16 19:16:20.601537 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:20.601139    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9"
Sep 16 19:16:20.613100 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:20.613064    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd-cabundle-cert\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9\" (UID: \"1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9"
Sep 16 19:16:20.613281 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:20.613114    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9\" (UID: \"1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9"
Sep 16 19:16:20.613281 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:20.613217    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd-proxy-tls\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9\" (UID: \"1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9"
Sep 16 19:16:20.613281 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:20.613274    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9z2f\" (UniqueName: \"kubernetes.io/projected/1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd-kube-api-access-n9z2f\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9\" (UID: \"1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9"
Sep 16 19:16:20.613473 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:20.613311    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9\" (UID: \"1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9"
Sep 16 19:16:20.714440 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:20.714396    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd-cabundle-cert\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9\" (UID: \"1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9"
Sep 16 19:16:20.714648 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:20.714446    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9\" (UID: \"1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9"
Sep 16 19:16:20.714648 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:20.714556    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd-proxy-tls\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9\" (UID: \"1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9"
Sep 16 19:16:20.714648 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:20.714595    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9z2f\" (UniqueName: \"kubernetes.io/projected/1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd-kube-api-access-n9z2f\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9\" (UID: \"1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9"
Sep 16 19:16:20.714648 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:20.714634    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9\" (UID: \"1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9"
Sep 16 19:16:20.714928 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:20.714903    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9\" (UID: \"1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9"
Sep 16 19:16:20.715240 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:20.715191    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd-cabundle-cert\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9\" (UID: \"1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9"
Sep 16 19:16:20.715368 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:20.715316    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9\" (UID: \"1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9"
Sep 16 19:16:20.717699 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:20.717657    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd-proxy-tls\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9\" (UID: \"1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9"
Sep 16 19:16:20.727286 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:20.727223    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9z2f\" (UniqueName: \"kubernetes.io/projected/1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd-kube-api-access-n9z2f\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9\" (UID: \"1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9"
Sep 16 19:16:20.916982 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:20.916941    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9"
Sep 16 19:16:21.098572 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:21.098533    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs" podUID="4dd240d8-1a39-455d-bad7-8e89d291909a" containerName="kserve-container" containerID="cri-o://8b5b9040bd81c207f7b0d7c337945367a8ce17dce424487f240e12428f7eae67" gracePeriod=30
Sep 16 19:16:21.099451 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:21.098996    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs" podUID="4dd240d8-1a39-455d-bad7-8e89d291909a" containerName="kube-rbac-proxy" containerID="cri-o://a687e09e2359ec857d3af7832950107578c2ae4a7911b22fcd917b16babb9f67" gracePeriod=30
Sep 16 19:16:23.550857 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:23.550804    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs" podUID="4dd240d8-1a39-455d-bad7-8e89d291909a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.51:8643/healthz\": dial tcp 10.134.0.51:8643: connect: connection refused"
Sep 16 19:16:23.551377 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:23.550804    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg" podUID="72e4a42d-de46-4845-b1d4-6485b4edc309" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.52:8643/healthz\": dial tcp 10.134.0.52:8643: connect: connection refused"
Sep 16 19:16:23.556289 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:23.556253    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg" podUID="72e4a42d-de46-4845-b1d4-6485b4edc309" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 16 19:16:28.550899 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:28.550855    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs" podUID="4dd240d8-1a39-455d-bad7-8e89d291909a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.51:8643/healthz\": dial tcp 10.134.0.51:8643: connect: connection refused"
Sep 16 19:16:28.551348 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:28.550849    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg" podUID="72e4a42d-de46-4845-b1d4-6485b4edc309" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.52:8643/healthz\": dial tcp 10.134.0.52:8643: connect: connection refused"
Sep 16 19:16:30.086893 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:30.086851    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs" podUID="4dd240d8-1a39-455d-bad7-8e89d291909a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 16 19:16:33.550645 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:33.550599    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg" podUID="72e4a42d-de46-4845-b1d4-6485b4edc309" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.52:8643/healthz\": dial tcp 10.134.0.52:8643: connect: connection refused"
Sep 16 19:16:33.551262 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:33.550606    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs" podUID="4dd240d8-1a39-455d-bad7-8e89d291909a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.51:8643/healthz\": dial tcp 10.134.0.51:8643: connect: connection refused"
Sep 16 19:16:33.551262 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:33.550731    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg"
Sep 16 19:16:33.551262 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:33.550823    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs"
Sep 16 19:16:33.556945 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:33.556910    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg" podUID="72e4a42d-de46-4845-b1d4-6485b4edc309" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 16 19:16:38.551168 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:38.551124    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs" podUID="4dd240d8-1a39-455d-bad7-8e89d291909a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.51:8643/healthz\": dial tcp 10.134.0.51:8643: connect: connection refused"
Sep 16 19:16:38.551703 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:38.551124    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg" podUID="72e4a42d-de46-4845-b1d4-6485b4edc309" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.52:8643/healthz\": dial tcp 10.134.0.52:8643: connect: connection refused"
Sep 16 19:16:40.086990 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:40.086945    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs" podUID="4dd240d8-1a39-455d-bad7-8e89d291909a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 16 19:16:43.550659 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:43.550618    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg" podUID="72e4a42d-de46-4845-b1d4-6485b4edc309" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.52:8643/healthz\": dial tcp 10.134.0.52:8643: connect: connection refused"
Sep 16 19:16:43.551112 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:43.550617    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs" podUID="4dd240d8-1a39-455d-bad7-8e89d291909a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.51:8643/healthz\": dial tcp 10.134.0.51:8643: connect: connection refused"
Sep 16 19:16:43.556184 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:43.556149    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg" podUID="72e4a42d-de46-4845-b1d4-6485b4edc309" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 16 19:16:43.556328 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:43.556308    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg"
Sep 16 19:16:48.550865 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:48.550817    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs" podUID="4dd240d8-1a39-455d-bad7-8e89d291909a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.51:8643/healthz\": dial tcp 10.134.0.51:8643: connect: connection refused"
Sep 16 19:16:48.551399 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:48.550828    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg" podUID="72e4a42d-de46-4845-b1d4-6485b4edc309" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.52:8643/healthz\": dial tcp 10.134.0.52:8643: connect: connection refused"
Sep 16 19:16:50.119196 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:50.119167    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg"
Sep 16 19:16:50.123454 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:50.123435    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs"
Sep 16 19:16:50.197163 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:50.197129    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72e4a42d-de46-4845-b1d4-6485b4edc309-kube-rbac-proxy-sar-config\") pod \"72e4a42d-de46-4845-b1d4-6485b4edc309\" (UID: \"72e4a42d-de46-4845-b1d4-6485b4edc309\") "
Sep 16 19:16:50.197407 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:50.197181    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4nqkh\" (UniqueName: \"kubernetes.io/projected/72e4a42d-de46-4845-b1d4-6485b4edc309-kube-api-access-4nqkh\") pod \"72e4a42d-de46-4845-b1d4-6485b4edc309\" (UID: \"72e4a42d-de46-4845-b1d4-6485b4edc309\") "
Sep 16 19:16:50.197407 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:50.197252    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72e4a42d-de46-4845-b1d4-6485b4edc309-kserve-provision-location\") pod \"72e4a42d-de46-4845-b1d4-6485b4edc309\" (UID: \"72e4a42d-de46-4845-b1d4-6485b4edc309\") "
Sep 16 19:16:50.197407 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:50.197276    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72e4a42d-de46-4845-b1d4-6485b4edc309-proxy-tls\") pod \"72e4a42d-de46-4845-b1d4-6485b4edc309\" (UID: \"72e4a42d-de46-4845-b1d4-6485b4edc309\") "
Sep 16 19:16:50.197407 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:50.197302    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72e4a42d-de46-4845-b1d4-6485b4edc309-cabundle-cert\") pod \"72e4a42d-de46-4845-b1d4-6485b4edc309\" (UID: \"72e4a42d-de46-4845-b1d4-6485b4edc309\") "
Sep 16 19:16:50.197650 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:50.197557    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72e4a42d-de46-4845-b1d4-6485b4edc309-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "72e4a42d-de46-4845-b1d4-6485b4edc309" (UID: "72e4a42d-de46-4845-b1d4-6485b4edc309"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:16:50.197804 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:50.197779    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72e4a42d-de46-4845-b1d4-6485b4edc309-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "72e4a42d-de46-4845-b1d4-6485b4edc309" (UID: "72e4a42d-de46-4845-b1d4-6485b4edc309"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:16:50.197877 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:50.197814    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72e4a42d-de46-4845-b1d4-6485b4edc309-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "72e4a42d-de46-4845-b1d4-6485b4edc309" (UID: "72e4a42d-de46-4845-b1d4-6485b4edc309"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:16:50.199746 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:50.199720    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72e4a42d-de46-4845-b1d4-6485b4edc309-kube-api-access-4nqkh" (OuterVolumeSpecName: "kube-api-access-4nqkh") pod "72e4a42d-de46-4845-b1d4-6485b4edc309" (UID: "72e4a42d-de46-4845-b1d4-6485b4edc309"). InnerVolumeSpecName "kube-api-access-4nqkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:16:50.199908 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:50.199868    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72e4a42d-de46-4845-b1d4-6485b4edc309-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "72e4a42d-de46-4845-b1d4-6485b4edc309" (UID: "72e4a42d-de46-4845-b1d4-6485b4edc309"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:16:50.258082 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:50.258043    2586 generic.go:356] "Generic (PLEG): container finished" podID="4dd240d8-1a39-455d-bad7-8e89d291909a" containerID="a687e09e2359ec857d3af7832950107578c2ae4a7911b22fcd917b16babb9f67" exitCode=2
Sep 16 19:16:50.258082 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:50.258079    2586 generic.go:356] "Generic (PLEG): container finished" podID="4dd240d8-1a39-455d-bad7-8e89d291909a" containerID="8b5b9040bd81c207f7b0d7c337945367a8ce17dce424487f240e12428f7eae67" exitCode=0
Sep 16 19:16:50.258355 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:50.258105    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs"
Sep 16 19:16:50.258355 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:50.258137    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs" event={"ID":"4dd240d8-1a39-455d-bad7-8e89d291909a","Type":"ContainerDied","Data":"a687e09e2359ec857d3af7832950107578c2ae4a7911b22fcd917b16babb9f67"}
Sep 16 19:16:50.258355 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:50.258240    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs" event={"ID":"4dd240d8-1a39-455d-bad7-8e89d291909a","Type":"ContainerDied","Data":"8b5b9040bd81c207f7b0d7c337945367a8ce17dce424487f240e12428f7eae67"}
Sep 16 19:16:50.258355 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:50.258274    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs" event={"ID":"4dd240d8-1a39-455d-bad7-8e89d291909a","Type":"ContainerDied","Data":"a01b8eb04c9cb8feffc9483b04d13cf9d18b34747a13f7c3eda6adcb9455406d"}
Sep 16 19:16:50.258355 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:50.258301    2586 scope.go:117] "RemoveContainer" containerID="a687e09e2359ec857d3af7832950107578c2ae4a7911b22fcd917b16babb9f67"
Sep 16 19:16:50.260599 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:50.260574    2586 generic.go:356] "Generic (PLEG): container finished" podID="72e4a42d-de46-4845-b1d4-6485b4edc309" containerID="6869d7082d0e2dac07ecc568efa4d2462514b44818665eac4df96bd645f37cb9" exitCode=2
Sep 16 19:16:50.260599 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:50.260597    2586 generic.go:356] "Generic (PLEG): container finished" podID="72e4a42d-de46-4845-b1d4-6485b4edc309" containerID="8bb6df660c15293598f80c861dfbd9c5424d46745b766ebd96cb9dc23869840f" exitCode=0
Sep 16 19:16:50.260775 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:50.260645    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg"
Sep 16 19:16:50.260775 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:50.260663    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg" event={"ID":"72e4a42d-de46-4845-b1d4-6485b4edc309","Type":"ContainerDied","Data":"6869d7082d0e2dac07ecc568efa4d2462514b44818665eac4df96bd645f37cb9"}
Sep 16 19:16:50.260775 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:50.260716    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg" event={"ID":"72e4a42d-de46-4845-b1d4-6485b4edc309","Type":"ContainerDied","Data":"8bb6df660c15293598f80c861dfbd9c5424d46745b766ebd96cb9dc23869840f"}
Sep 16 19:16:50.260775 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:50.260748    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg" event={"ID":"72e4a42d-de46-4845-b1d4-6485b4edc309","Type":"ContainerDied","Data":"c068d40dbef0793d3acbeb0a28afd18fd88e3a666937c1e8c6d6fd3d05b39f7a"}
Sep 16 19:16:50.298459 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:50.298354    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4gtq\" (UniqueName: \"kubernetes.io/projected/4dd240d8-1a39-455d-bad7-8e89d291909a-kube-api-access-d4gtq\") pod \"4dd240d8-1a39-455d-bad7-8e89d291909a\" (UID: \"4dd240d8-1a39-455d-bad7-8e89d291909a\") "
Sep 16 19:16:50.298459 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:50.298426    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4dd240d8-1a39-455d-bad7-8e89d291909a-cabundle-cert\") pod \"4dd240d8-1a39-455d-bad7-8e89d291909a\" (UID: \"4dd240d8-1a39-455d-bad7-8e89d291909a\") "
Sep 16 19:16:50.298669 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:50.298462    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4dd240d8-1a39-455d-bad7-8e89d291909a-kube-rbac-proxy-sar-config\") pod \"4dd240d8-1a39-455d-bad7-8e89d291909a\" (UID: \"4dd240d8-1a39-455d-bad7-8e89d291909a\") "
Sep 16 19:16:50.298669 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:50.298503    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4dd240d8-1a39-455d-bad7-8e89d291909a-kserve-provision-location\") pod \"4dd240d8-1a39-455d-bad7-8e89d291909a\" (UID: \"4dd240d8-1a39-455d-bad7-8e89d291909a\") "
Sep 16 19:16:50.298669 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:50.298536    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4dd240d8-1a39-455d-bad7-8e89d291909a-proxy-tls\") pod \"4dd240d8-1a39-455d-bad7-8e89d291909a\" (UID: \"4dd240d8-1a39-455d-bad7-8e89d291909a\") "
Sep 16 19:16:50.298877 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:50.298851    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72e4a42d-de46-4845-b1d4-6485b4edc309-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:16:50.298877 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:50.298874    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72e4a42d-de46-4845-b1d4-6485b4edc309-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:16:50.298994 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:50.298890    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72e4a42d-de46-4845-b1d4-6485b4edc309-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:16:50.298994 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:50.298903    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72e4a42d-de46-4845-b1d4-6485b4edc309-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:16:50.298994 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:50.298920    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nqkh\" (UniqueName: \"kubernetes.io/projected/72e4a42d-de46-4845-b1d4-6485b4edc309-kube-api-access-4nqkh\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:16:50.299157 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:50.299095    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dd240d8-1a39-455d-bad7-8e89d291909a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4dd240d8-1a39-455d-bad7-8e89d291909a" (UID: "4dd240d8-1a39-455d-bad7-8e89d291909a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:16:50.299302 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:50.299269    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dd240d8-1a39-455d-bad7-8e89d291909a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4dd240d8-1a39-455d-bad7-8e89d291909a" (UID: "4dd240d8-1a39-455d-bad7-8e89d291909a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:16:50.299575 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:50.299491    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dd240d8-1a39-455d-bad7-8e89d291909a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4dd240d8-1a39-455d-bad7-8e89d291909a" (UID: "4dd240d8-1a39-455d-bad7-8e89d291909a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:16:50.301103 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:50.301077    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dd240d8-1a39-455d-bad7-8e89d291909a-kube-api-access-d4gtq" (OuterVolumeSpecName: "kube-api-access-d4gtq") pod "4dd240d8-1a39-455d-bad7-8e89d291909a" (UID: "4dd240d8-1a39-455d-bad7-8e89d291909a"). InnerVolumeSpecName "kube-api-access-d4gtq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:16:50.301349 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:50.301324    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dd240d8-1a39-455d-bad7-8e89d291909a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4dd240d8-1a39-455d-bad7-8e89d291909a" (UID: "4dd240d8-1a39-455d-bad7-8e89d291909a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:16:50.301453 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:50.301424    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg"]
Sep 16 19:16:50.304682 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:50.304662    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-wnwvg"]
Sep 16 19:16:50.326438 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:50.326416    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9"]
Sep 16 19:16:50.400279 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:50.400225    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4gtq\" (UniqueName: \"kubernetes.io/projected/4dd240d8-1a39-455d-bad7-8e89d291909a-kube-api-access-d4gtq\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:16:50.400279 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:50.400272    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4dd240d8-1a39-455d-bad7-8e89d291909a-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:16:50.400279 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:50.400284    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4dd240d8-1a39-455d-bad7-8e89d291909a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:16:50.400592 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:50.400293    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4dd240d8-1a39-455d-bad7-8e89d291909a-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:16:50.400592 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:50.400303    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4dd240d8-1a39-455d-bad7-8e89d291909a-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:16:50.591357 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:50.591325    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs"]
Sep 16 19:16:50.603557 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:50.603527    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs"]
Sep 16 19:16:50.893417 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:50.893190    2586 scope.go:117] "RemoveContainer" containerID="8b5b9040bd81c207f7b0d7c337945367a8ce17dce424487f240e12428f7eae67"
Sep 16 19:16:50.894578 ip-10-0-129-190 kubenswrapper[2586]: W0916 19:16:50.894553    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a8d984e_ef64_429b_9f5a_ffd8ba8e6dfd.slice/crio-32032c976d403e5585bd47e91ad15393941def13c67fce57da6f930337a7805e WatchSource:0}: Error finding container 32032c976d403e5585bd47e91ad15393941def13c67fce57da6f930337a7805e: Status 404 returned error can't find the container with id 32032c976d403e5585bd47e91ad15393941def13c67fce57da6f930337a7805e
Sep 16 19:16:51.087830 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:51.087785    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-7pcqs" podUID="4dd240d8-1a39-455d-bad7-8e89d291909a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: i/o timeout"
Sep 16 19:16:51.093367 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:51.093339    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4dd240d8-1a39-455d-bad7-8e89d291909a" path="/var/lib/kubelet/pods/4dd240d8-1a39-455d-bad7-8e89d291909a/volumes"
Sep 16 19:16:51.094004 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:51.093980    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72e4a42d-de46-4845-b1d4-6485b4edc309" path="/var/lib/kubelet/pods/72e4a42d-de46-4845-b1d4-6485b4edc309/volumes"
Sep 16 19:16:51.269484 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:51.269408    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9" event={"ID":"1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd","Type":"ContainerStarted","Data":"32032c976d403e5585bd47e91ad15393941def13c67fce57da6f930337a7805e"}
Sep 16 19:16:51.989754 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:51.989724    2586 scope.go:117] "RemoveContainer" containerID="4318346665dbf01a05af1d17974751102ca168697e63210366b2c66b61c7204e"
Sep 16 19:16:55.604560 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:55.604535    2586 scope.go:117] "RemoveContainer" containerID="a687e09e2359ec857d3af7832950107578c2ae4a7911b22fcd917b16babb9f67"
Sep 16 19:16:55.604864 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:16:55.604838    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a687e09e2359ec857d3af7832950107578c2ae4a7911b22fcd917b16babb9f67\": container with ID starting with a687e09e2359ec857d3af7832950107578c2ae4a7911b22fcd917b16babb9f67 not found: ID does not exist" containerID="a687e09e2359ec857d3af7832950107578c2ae4a7911b22fcd917b16babb9f67"
Sep 16 19:16:55.604914 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:55.604873    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a687e09e2359ec857d3af7832950107578c2ae4a7911b22fcd917b16babb9f67"} err="failed to get container status \"a687e09e2359ec857d3af7832950107578c2ae4a7911b22fcd917b16babb9f67\": rpc error: code = NotFound desc = could not find container \"a687e09e2359ec857d3af7832950107578c2ae4a7911b22fcd917b16babb9f67\": container with ID starting with a687e09e2359ec857d3af7832950107578c2ae4a7911b22fcd917b16babb9f67 not found: ID does not exist"
Sep 16 19:16:55.604914 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:55.604891    2586 scope.go:117] "RemoveContainer" containerID="8b5b9040bd81c207f7b0d7c337945367a8ce17dce424487f240e12428f7eae67"
Sep 16 19:16:55.605176 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:16:55.605161    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b5b9040bd81c207f7b0d7c337945367a8ce17dce424487f240e12428f7eae67\": container with ID starting with 8b5b9040bd81c207f7b0d7c337945367a8ce17dce424487f240e12428f7eae67 not found: ID does not exist" containerID="8b5b9040bd81c207f7b0d7c337945367a8ce17dce424487f240e12428f7eae67"
Sep 16 19:16:55.605240 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:55.605180    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b5b9040bd81c207f7b0d7c337945367a8ce17dce424487f240e12428f7eae67"} err="failed to get container status \"8b5b9040bd81c207f7b0d7c337945367a8ce17dce424487f240e12428f7eae67\": rpc error: code = NotFound desc = could not find container \"8b5b9040bd81c207f7b0d7c337945367a8ce17dce424487f240e12428f7eae67\": container with ID starting with 8b5b9040bd81c207f7b0d7c337945367a8ce17dce424487f240e12428f7eae67 not found: ID does not exist"
Sep 16 19:16:55.605240 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:55.605193    2586 scope.go:117] "RemoveContainer" containerID="4318346665dbf01a05af1d17974751102ca168697e63210366b2c66b61c7204e"
Sep 16 19:16:55.605440 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:16:55.605422    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4318346665dbf01a05af1d17974751102ca168697e63210366b2c66b61c7204e\": container with ID starting with 4318346665dbf01a05af1d17974751102ca168697e63210366b2c66b61c7204e not found: ID does not exist" containerID="4318346665dbf01a05af1d17974751102ca168697e63210366b2c66b61c7204e"
Sep 16 19:16:55.605477 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:55.605445    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4318346665dbf01a05af1d17974751102ca168697e63210366b2c66b61c7204e"} err="failed to get container status \"4318346665dbf01a05af1d17974751102ca168697e63210366b2c66b61c7204e\": rpc error: code = NotFound desc = could not find container \"4318346665dbf01a05af1d17974751102ca168697e63210366b2c66b61c7204e\": container with ID starting with 4318346665dbf01a05af1d17974751102ca168697e63210366b2c66b61c7204e not found: ID does not exist"
Sep 16 19:16:55.605477 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:55.605459    2586 scope.go:117] "RemoveContainer" containerID="a687e09e2359ec857d3af7832950107578c2ae4a7911b22fcd917b16babb9f67"
Sep 16 19:16:55.605698 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:55.605678    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a687e09e2359ec857d3af7832950107578c2ae4a7911b22fcd917b16babb9f67"} err="failed to get container status \"a687e09e2359ec857d3af7832950107578c2ae4a7911b22fcd917b16babb9f67\": rpc error: code = NotFound desc = could not find container \"a687e09e2359ec857d3af7832950107578c2ae4a7911b22fcd917b16babb9f67\": container with ID starting with a687e09e2359ec857d3af7832950107578c2ae4a7911b22fcd917b16babb9f67 not found: ID does not exist"
Sep 16 19:16:55.605746 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:55.605699    2586 scope.go:117] "RemoveContainer" containerID="8b5b9040bd81c207f7b0d7c337945367a8ce17dce424487f240e12428f7eae67"
Sep 16 19:16:55.605933 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:55.605916    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b5b9040bd81c207f7b0d7c337945367a8ce17dce424487f240e12428f7eae67"} err="failed to get container status \"8b5b9040bd81c207f7b0d7c337945367a8ce17dce424487f240e12428f7eae67\": rpc error: code = NotFound desc = could not find container \"8b5b9040bd81c207f7b0d7c337945367a8ce17dce424487f240e12428f7eae67\": container with ID starting with 8b5b9040bd81c207f7b0d7c337945367a8ce17dce424487f240e12428f7eae67 not found: ID does not exist"
Sep 16 19:16:55.605993 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:55.605933    2586 scope.go:117] "RemoveContainer" containerID="4318346665dbf01a05af1d17974751102ca168697e63210366b2c66b61c7204e"
Sep 16 19:16:55.606193 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:55.606173    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4318346665dbf01a05af1d17974751102ca168697e63210366b2c66b61c7204e"} err="failed to get container status \"4318346665dbf01a05af1d17974751102ca168697e63210366b2c66b61c7204e\": rpc error: code = NotFound desc = could not find container \"4318346665dbf01a05af1d17974751102ca168697e63210366b2c66b61c7204e\": container with ID starting with 4318346665dbf01a05af1d17974751102ca168697e63210366b2c66b61c7204e not found: ID does not exist"
Sep 16 19:16:55.606262 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:55.606194    2586 scope.go:117] "RemoveContainer" containerID="6869d7082d0e2dac07ecc568efa4d2462514b44818665eac4df96bd645f37cb9"
Sep 16 19:16:56.339424 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:16:56.339401    2586 scope.go:117] "RemoveContainer" containerID="8bb6df660c15293598f80c861dfbd9c5424d46745b766ebd96cb9dc23869840f"
Sep 16 19:17:01.108920 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:01.108787    2586 scope.go:117] "RemoveContainer" containerID="16fbf89cc7d80f1ec1c9b9266603fa18c31b52916e93e7fd3c2e0562e34ed0c4"
Sep 16 19:17:01.335179 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:01.335138    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9" event={"ID":"1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd","Type":"ContainerStarted","Data":"cafc756dd6bd14c563f064ae3587c051d053ee11b789e64be2622af1d964dbd4"}
Sep 16 19:17:02.265515 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:02.265283    2586 scope.go:117] "RemoveContainer" containerID="6869d7082d0e2dac07ecc568efa4d2462514b44818665eac4df96bd645f37cb9"
Sep 16 19:17:02.265782 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:17:02.265677    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6869d7082d0e2dac07ecc568efa4d2462514b44818665eac4df96bd645f37cb9\": container with ID starting with 6869d7082d0e2dac07ecc568efa4d2462514b44818665eac4df96bd645f37cb9 not found: ID does not exist" containerID="6869d7082d0e2dac07ecc568efa4d2462514b44818665eac4df96bd645f37cb9"
Sep 16 19:17:02.265782 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:02.265708    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6869d7082d0e2dac07ecc568efa4d2462514b44818665eac4df96bd645f37cb9"} err="failed to get container status \"6869d7082d0e2dac07ecc568efa4d2462514b44818665eac4df96bd645f37cb9\": rpc error: code = NotFound desc = could not find container \"6869d7082d0e2dac07ecc568efa4d2462514b44818665eac4df96bd645f37cb9\": container with ID starting with 6869d7082d0e2dac07ecc568efa4d2462514b44818665eac4df96bd645f37cb9 not found: ID does not exist"
Sep 16 19:17:02.265782 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:02.265728    2586 scope.go:117] "RemoveContainer" containerID="8bb6df660c15293598f80c861dfbd9c5424d46745b766ebd96cb9dc23869840f"
Sep 16 19:17:02.265989 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:17:02.265971    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bb6df660c15293598f80c861dfbd9c5424d46745b766ebd96cb9dc23869840f\": container with ID starting with 8bb6df660c15293598f80c861dfbd9c5424d46745b766ebd96cb9dc23869840f not found: ID does not exist" containerID="8bb6df660c15293598f80c861dfbd9c5424d46745b766ebd96cb9dc23869840f"
Sep 16 19:17:02.266043 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:02.265994    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bb6df660c15293598f80c861dfbd9c5424d46745b766ebd96cb9dc23869840f"} err="failed to get container status \"8bb6df660c15293598f80c861dfbd9c5424d46745b766ebd96cb9dc23869840f\": rpc error: code = NotFound desc = could not find container \"8bb6df660c15293598f80c861dfbd9c5424d46745b766ebd96cb9dc23869840f\": container with ID starting with 8bb6df660c15293598f80c861dfbd9c5424d46745b766ebd96cb9dc23869840f not found: ID does not exist"
Sep 16 19:17:02.266043 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:02.266009    2586 scope.go:117] "RemoveContainer" containerID="16fbf89cc7d80f1ec1c9b9266603fa18c31b52916e93e7fd3c2e0562e34ed0c4"
Sep 16 19:17:02.266292 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:17:02.266269    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16fbf89cc7d80f1ec1c9b9266603fa18c31b52916e93e7fd3c2e0562e34ed0c4\": container with ID starting with 16fbf89cc7d80f1ec1c9b9266603fa18c31b52916e93e7fd3c2e0562e34ed0c4 not found: ID does not exist" containerID="16fbf89cc7d80f1ec1c9b9266603fa18c31b52916e93e7fd3c2e0562e34ed0c4"
Sep 16 19:17:02.266360 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:02.266298    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16fbf89cc7d80f1ec1c9b9266603fa18c31b52916e93e7fd3c2e0562e34ed0c4"} err="failed to get container status \"16fbf89cc7d80f1ec1c9b9266603fa18c31b52916e93e7fd3c2e0562e34ed0c4\": rpc error: code = NotFound desc = could not find container \"16fbf89cc7d80f1ec1c9b9266603fa18c31b52916e93e7fd3c2e0562e34ed0c4\": container with ID starting with 16fbf89cc7d80f1ec1c9b9266603fa18c31b52916e93e7fd3c2e0562e34ed0c4 not found: ID does not exist"
Sep 16 19:17:02.266360 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:02.266318    2586 scope.go:117] "RemoveContainer" containerID="6869d7082d0e2dac07ecc568efa4d2462514b44818665eac4df96bd645f37cb9"
Sep 16 19:17:02.266635 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:02.266616    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6869d7082d0e2dac07ecc568efa4d2462514b44818665eac4df96bd645f37cb9"} err="failed to get container status \"6869d7082d0e2dac07ecc568efa4d2462514b44818665eac4df96bd645f37cb9\": rpc error: code = NotFound desc = could not find container \"6869d7082d0e2dac07ecc568efa4d2462514b44818665eac4df96bd645f37cb9\": container with ID starting with 6869d7082d0e2dac07ecc568efa4d2462514b44818665eac4df96bd645f37cb9 not found: ID does not exist"
Sep 16 19:17:02.266688 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:02.266635    2586 scope.go:117] "RemoveContainer" containerID="8bb6df660c15293598f80c861dfbd9c5424d46745b766ebd96cb9dc23869840f"
Sep 16 19:17:02.266867 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:02.266849    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bb6df660c15293598f80c861dfbd9c5424d46745b766ebd96cb9dc23869840f"} err="failed to get container status \"8bb6df660c15293598f80c861dfbd9c5424d46745b766ebd96cb9dc23869840f\": rpc error: code = NotFound desc = could not find container \"8bb6df660c15293598f80c861dfbd9c5424d46745b766ebd96cb9dc23869840f\": container with ID starting with 8bb6df660c15293598f80c861dfbd9c5424d46745b766ebd96cb9dc23869840f not found: ID does not exist"
Sep 16 19:17:02.266925 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:02.266867    2586 scope.go:117] "RemoveContainer" containerID="16fbf89cc7d80f1ec1c9b9266603fa18c31b52916e93e7fd3c2e0562e34ed0c4"
Sep 16 19:17:02.267167 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:02.267142    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16fbf89cc7d80f1ec1c9b9266603fa18c31b52916e93e7fd3c2e0562e34ed0c4"} err="failed to get container status \"16fbf89cc7d80f1ec1c9b9266603fa18c31b52916e93e7fd3c2e0562e34ed0c4\": rpc error: code = NotFound desc = could not find container \"16fbf89cc7d80f1ec1c9b9266603fa18c31b52916e93e7fd3c2e0562e34ed0c4\": container with ID starting with 16fbf89cc7d80f1ec1c9b9266603fa18c31b52916e93e7fd3c2e0562e34ed0c4 not found: ID does not exist"
Sep 16 19:17:30.492242 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:30.492199    2586 generic.go:356] "Generic (PLEG): container finished" podID="1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd" containerID="cafc756dd6bd14c563f064ae3587c051d053ee11b789e64be2622af1d964dbd4" exitCode=0
Sep 16 19:17:30.492645 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:30.492271    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9" event={"ID":"1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd","Type":"ContainerDied","Data":"cafc756dd6bd14c563f064ae3587c051d053ee11b789e64be2622af1d964dbd4"}
Sep 16 19:17:31.499379 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:31.499281    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-f8flm" event={"ID":"81b89f31-91f5-4d65-85c3-aba3321b5cd5","Type":"ContainerStarted","Data":"aa737ca8f1b295c15e7e0cc9d48ac01a9605ed99e26f85c1374090478982bde8"}
Sep 16 19:17:31.499379 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:31.499360    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-f8flm"
Sep 16 19:17:31.499928 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:31.499395    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-f8flm"
Sep 16 19:17:31.499928 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:31.499420    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-f8flm" event={"ID":"81b89f31-91f5-4d65-85c3-aba3321b5cd5","Type":"ContainerStarted","Data":"c75002482cb6d116b520c2744165cba48c98dcd9ec1ffce0211fac19a44fa0ac"}
Sep 16 19:17:31.501165 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:31.501138    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-f8flm" podUID="81b89f31-91f5-4d65-85c3-aba3321b5cd5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 16 19:17:31.501636 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:31.501611    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9" event={"ID":"1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd","Type":"ContainerStarted","Data":"15aedbb3060bc1742888a7e7410395ebd8d8820938f5eaad8cca11c61f184ff1"}
Sep 16 19:17:31.501707 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:31.501671    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9" event={"ID":"1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd","Type":"ContainerStarted","Data":"205d8e44ad6111539481a8b4eee0eec343f2d85fcb2f8ea5ec59463d9f5b826b"}
Sep 16 19:17:31.501871 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:31.501855    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9"
Sep 16 19:17:31.527950 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:31.527900    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-f8flm" podStartSLOduration=2.125111626 podStartE2EDuration="2m0.527884608s" totalImagesPullingTime="1m54.422294234s" totalInitContainerRuntime="3.980478748s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:15:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:17:31.52540383 +0000 UTC m=+2226.969265398" watchObservedRunningTime="2026-09-16 19:17:31.527884608 +0000 UTC m=+2226.971746183"
Sep 16 19:17:31.553190 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:31.553138    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9" podStartSLOduration=66.862809872 podStartE2EDuration="1m11.553122601s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.690312729s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:16:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:17:31.55012697 +0000 UTC m=+2226.993988567" watchObservedRunningTime="2026-09-16 19:17:31.553122601 +0000 UTC m=+2226.996984176"
Sep 16 19:17:32.506380 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:32.506327    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-f8flm" podUID="81b89f31-91f5-4d65-85c3-aba3321b5cd5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 16 19:17:32.506852 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:32.506482    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9"
Sep 16 19:17:32.507805 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:32.507779    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9" podUID="1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 16 19:17:33.510435 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:33.510391    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9" podUID="1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 16 19:17:37.510859 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:37.510829    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-f8flm"
Sep 16 19:17:37.511344 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:37.511312    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-f8flm" podUID="81b89f31-91f5-4d65-85c3-aba3321b5cd5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 16 19:17:38.514863 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:38.514832    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9"
Sep 16 19:17:38.515364 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:38.515336    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9" podUID="1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 16 19:17:47.512003 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:47.511928    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-f8flm"
Sep 16 19:17:48.515552 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:48.515511    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9" podUID="1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 16 19:17:54.221556 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:54.221515    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-f8flm"]
Sep 16 19:17:54.221999 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:54.221769    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-f8flm" podUID="81b89f31-91f5-4d65-85c3-aba3321b5cd5" containerName="kserve-container" containerID="cri-o://c75002482cb6d116b520c2744165cba48c98dcd9ec1ffce0211fac19a44fa0ac" gracePeriod=30
Sep 16 19:17:54.221999 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:54.221887    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-f8flm" podUID="81b89f31-91f5-4d65-85c3-aba3321b5cd5" containerName="kube-rbac-proxy" containerID="cri-o://aa737ca8f1b295c15e7e0cc9d48ac01a9605ed99e26f85c1374090478982bde8" gracePeriod=30
Sep 16 19:17:54.609748 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:54.609713    2586 generic.go:356] "Generic (PLEG): container finished" podID="81b89f31-91f5-4d65-85c3-aba3321b5cd5" containerID="aa737ca8f1b295c15e7e0cc9d48ac01a9605ed99e26f85c1374090478982bde8" exitCode=2
Sep 16 19:17:54.609915 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:54.609768    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-f8flm" event={"ID":"81b89f31-91f5-4d65-85c3-aba3321b5cd5","Type":"ContainerDied","Data":"aa737ca8f1b295c15e7e0cc9d48ac01a9605ed99e26f85c1374090478982bde8"}
Sep 16 19:17:57.205583 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:57.205505    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-f8flm"
Sep 16 19:17:57.283414 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:57.283381    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwcds\" (UniqueName: \"kubernetes.io/projected/81b89f31-91f5-4d65-85c3-aba3321b5cd5-kube-api-access-bwcds\") pod \"81b89f31-91f5-4d65-85c3-aba3321b5cd5\" (UID: \"81b89f31-91f5-4d65-85c3-aba3321b5cd5\") "
Sep 16 19:17:57.283608 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:57.283432    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81b89f31-91f5-4d65-85c3-aba3321b5cd5-kube-rbac-proxy-sar-config\") pod \"81b89f31-91f5-4d65-85c3-aba3321b5cd5\" (UID: \"81b89f31-91f5-4d65-85c3-aba3321b5cd5\") "
Sep 16 19:17:57.283608 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:57.283492    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81b89f31-91f5-4d65-85c3-aba3321b5cd5-cabundle-cert\") pod \"81b89f31-91f5-4d65-85c3-aba3321b5cd5\" (UID: \"81b89f31-91f5-4d65-85c3-aba3321b5cd5\") "
Sep 16 19:17:57.283608 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:57.283517    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81b89f31-91f5-4d65-85c3-aba3321b5cd5-kserve-provision-location\") pod \"81b89f31-91f5-4d65-85c3-aba3321b5cd5\" (UID: \"81b89f31-91f5-4d65-85c3-aba3321b5cd5\") "
Sep 16 19:17:57.283608 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:57.283553    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81b89f31-91f5-4d65-85c3-aba3321b5cd5-proxy-tls\") pod \"81b89f31-91f5-4d65-85c3-aba3321b5cd5\" (UID: \"81b89f31-91f5-4d65-85c3-aba3321b5cd5\") "
Sep 16 19:17:57.283846 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:57.283804    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81b89f31-91f5-4d65-85c3-aba3321b5cd5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "81b89f31-91f5-4d65-85c3-aba3321b5cd5" (UID: "81b89f31-91f5-4d65-85c3-aba3321b5cd5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:17:57.283915 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:57.283848    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81b89f31-91f5-4d65-85c3-aba3321b5cd5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "81b89f31-91f5-4d65-85c3-aba3321b5cd5" (UID: "81b89f31-91f5-4d65-85c3-aba3321b5cd5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:17:57.283915 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:57.283882    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b89f31-91f5-4d65-85c3-aba3321b5cd5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "81b89f31-91f5-4d65-85c3-aba3321b5cd5" (UID: "81b89f31-91f5-4d65-85c3-aba3321b5cd5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:57.285675 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:57.285642    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81b89f31-91f5-4d65-85c3-aba3321b5cd5-kube-api-access-bwcds" (OuterVolumeSpecName: "kube-api-access-bwcds") pod "81b89f31-91f5-4d65-85c3-aba3321b5cd5" (UID: "81b89f31-91f5-4d65-85c3-aba3321b5cd5"). InnerVolumeSpecName "kube-api-access-bwcds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:17:57.285809 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:57.285750    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81b89f31-91f5-4d65-85c3-aba3321b5cd5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "81b89f31-91f5-4d65-85c3-aba3321b5cd5" (UID: "81b89f31-91f5-4d65-85c3-aba3321b5cd5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:17:57.384334 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:57.384300    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwcds\" (UniqueName: \"kubernetes.io/projected/81b89f31-91f5-4d65-85c3-aba3321b5cd5-kube-api-access-bwcds\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:17:57.384334 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:57.384331    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81b89f31-91f5-4d65-85c3-aba3321b5cd5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:17:57.384334 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:57.384340    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81b89f31-91f5-4d65-85c3-aba3321b5cd5-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:17:57.384556 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:57.384349    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81b89f31-91f5-4d65-85c3-aba3321b5cd5-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:17:57.384556 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:57.384359    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81b89f31-91f5-4d65-85c3-aba3321b5cd5-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:17:57.631149 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:57.631110    2586 generic.go:356] "Generic (PLEG): container finished" podID="81b89f31-91f5-4d65-85c3-aba3321b5cd5" containerID="c75002482cb6d116b520c2744165cba48c98dcd9ec1ffce0211fac19a44fa0ac" exitCode=0
Sep 16 19:17:57.631339 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:57.631165    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-f8flm"
Sep 16 19:17:57.631339 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:57.631184    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-f8flm" event={"ID":"81b89f31-91f5-4d65-85c3-aba3321b5cd5","Type":"ContainerDied","Data":"c75002482cb6d116b520c2744165cba48c98dcd9ec1ffce0211fac19a44fa0ac"}
Sep 16 19:17:57.631339 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:57.631287    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-f8flm" event={"ID":"81b89f31-91f5-4d65-85c3-aba3321b5cd5","Type":"ContainerDied","Data":"76456ad3bfdd38e74fe26231deb2d91d23547d4e825770bace4caddf5617e17b"}
Sep 16 19:17:57.631339 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:57.631315    2586 scope.go:117] "RemoveContainer" containerID="aa737ca8f1b295c15e7e0cc9d48ac01a9605ed99e26f85c1374090478982bde8"
Sep 16 19:17:57.640383 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:57.640363    2586 scope.go:117] "RemoveContainer" containerID="c75002482cb6d116b520c2744165cba48c98dcd9ec1ffce0211fac19a44fa0ac"
Sep 16 19:17:57.651446 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:57.651424    2586 scope.go:117] "RemoveContainer" containerID="66ee273b95f9b0eeb39c2fa455cff83b143e7e8783c6a875246074ca3c135824"
Sep 16 19:17:57.659354 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:57.659338    2586 scope.go:117] "RemoveContainer" containerID="aa737ca8f1b295c15e7e0cc9d48ac01a9605ed99e26f85c1374090478982bde8"
Sep 16 19:17:57.659747 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:17:57.659721    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa737ca8f1b295c15e7e0cc9d48ac01a9605ed99e26f85c1374090478982bde8\": container with ID starting with aa737ca8f1b295c15e7e0cc9d48ac01a9605ed99e26f85c1374090478982bde8 not found: ID does not exist" containerID="aa737ca8f1b295c15e7e0cc9d48ac01a9605ed99e26f85c1374090478982bde8"
Sep 16 19:17:57.659861 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:57.659754    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa737ca8f1b295c15e7e0cc9d48ac01a9605ed99e26f85c1374090478982bde8"} err="failed to get container status \"aa737ca8f1b295c15e7e0cc9d48ac01a9605ed99e26f85c1374090478982bde8\": rpc error: code = NotFound desc = could not find container \"aa737ca8f1b295c15e7e0cc9d48ac01a9605ed99e26f85c1374090478982bde8\": container with ID starting with aa737ca8f1b295c15e7e0cc9d48ac01a9605ed99e26f85c1374090478982bde8 not found: ID does not exist"
Sep 16 19:17:57.659861 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:57.659769    2586 scope.go:117] "RemoveContainer" containerID="c75002482cb6d116b520c2744165cba48c98dcd9ec1ffce0211fac19a44fa0ac"
Sep 16 19:17:57.660026 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:17:57.660012    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c75002482cb6d116b520c2744165cba48c98dcd9ec1ffce0211fac19a44fa0ac\": container with ID starting with c75002482cb6d116b520c2744165cba48c98dcd9ec1ffce0211fac19a44fa0ac not found: ID does not exist" containerID="c75002482cb6d116b520c2744165cba48c98dcd9ec1ffce0211fac19a44fa0ac"
Sep 16 19:17:57.660070 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:57.660029    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c75002482cb6d116b520c2744165cba48c98dcd9ec1ffce0211fac19a44fa0ac"} err="failed to get container status \"c75002482cb6d116b520c2744165cba48c98dcd9ec1ffce0211fac19a44fa0ac\": rpc error: code = NotFound desc = could not find container \"c75002482cb6d116b520c2744165cba48c98dcd9ec1ffce0211fac19a44fa0ac\": container with ID starting with c75002482cb6d116b520c2744165cba48c98dcd9ec1ffce0211fac19a44fa0ac not found: ID does not exist"
Sep 16 19:17:57.660070 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:57.660042    2586 scope.go:117] "RemoveContainer" containerID="66ee273b95f9b0eeb39c2fa455cff83b143e7e8783c6a875246074ca3c135824"
Sep 16 19:17:57.660292 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:17:57.660275    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66ee273b95f9b0eeb39c2fa455cff83b143e7e8783c6a875246074ca3c135824\": container with ID starting with 66ee273b95f9b0eeb39c2fa455cff83b143e7e8783c6a875246074ca3c135824 not found: ID does not exist" containerID="66ee273b95f9b0eeb39c2fa455cff83b143e7e8783c6a875246074ca3c135824"
Sep 16 19:17:57.660353 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:57.660297    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66ee273b95f9b0eeb39c2fa455cff83b143e7e8783c6a875246074ca3c135824"} err="failed to get container status \"66ee273b95f9b0eeb39c2fa455cff83b143e7e8783c6a875246074ca3c135824\": rpc error: code = NotFound desc = could not find container \"66ee273b95f9b0eeb39c2fa455cff83b143e7e8783c6a875246074ca3c135824\": container with ID starting with 66ee273b95f9b0eeb39c2fa455cff83b143e7e8783c6a875246074ca3c135824 not found: ID does not exist"
Sep 16 19:17:57.664632 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:57.664611    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-f8flm"]
Sep 16 19:17:57.667160 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:57.667139    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-f8flm"]
Sep 16 19:17:58.515451 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:58.515413    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9" podUID="1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 16 19:17:59.094206 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:59.094167    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81b89f31-91f5-4d65-85c3-aba3321b5cd5" path="/var/lib/kubelet/pods/81b89f31-91f5-4d65-85c3-aba3321b5cd5/volumes"
Sep 16 19:17:59.418648 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:59.418615    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-gwfg9"]
Sep 16 19:17:59.419110 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:59.419096    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72e4a42d-de46-4845-b1d4-6485b4edc309" containerName="storage-initializer"
Sep 16 19:17:59.419160 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:59.419114    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="72e4a42d-de46-4845-b1d4-6485b4edc309" containerName="storage-initializer"
Sep 16 19:17:59.419160 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:59.419140    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dd240d8-1a39-455d-bad7-8e89d291909a" containerName="storage-initializer"
Sep 16 19:17:59.419160 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:59.419149    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dd240d8-1a39-455d-bad7-8e89d291909a" containerName="storage-initializer"
Sep 16 19:17:59.419268 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:59.419166    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81b89f31-91f5-4d65-85c3-aba3321b5cd5" containerName="kserve-container"
Sep 16 19:17:59.419268 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:59.419176    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="81b89f31-91f5-4d65-85c3-aba3321b5cd5" containerName="kserve-container"
Sep 16 19:17:59.419268 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:59.419186    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81b89f31-91f5-4d65-85c3-aba3321b5cd5" containerName="kube-rbac-proxy"
Sep 16 19:17:59.419268 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:59.419194    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="81b89f31-91f5-4d65-85c3-aba3321b5cd5" containerName="kube-rbac-proxy"
Sep 16 19:17:59.419268 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:59.419204    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dd240d8-1a39-455d-bad7-8e89d291909a" containerName="kube-rbac-proxy"
Sep 16 19:17:59.419268 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:59.419212    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dd240d8-1a39-455d-bad7-8e89d291909a" containerName="kube-rbac-proxy"
Sep 16 19:17:59.419268 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:59.419223    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72e4a42d-de46-4845-b1d4-6485b4edc309" containerName="kube-rbac-proxy"
Sep 16 19:17:59.419268 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:59.419246    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="72e4a42d-de46-4845-b1d4-6485b4edc309" containerName="kube-rbac-proxy"
Sep 16 19:17:59.419268 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:59.419259    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dd240d8-1a39-455d-bad7-8e89d291909a" containerName="kserve-container"
Sep 16 19:17:59.419268 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:59.419267    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dd240d8-1a39-455d-bad7-8e89d291909a" containerName="kserve-container"
Sep 16 19:17:59.419567 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:59.419280    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72e4a42d-de46-4845-b1d4-6485b4edc309" containerName="kserve-container"
Sep 16 19:17:59.419567 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:59.419289    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="72e4a42d-de46-4845-b1d4-6485b4edc309" containerName="kserve-container"
Sep 16 19:17:59.419567 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:59.419299    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81b89f31-91f5-4d65-85c3-aba3321b5cd5" containerName="storage-initializer"
Sep 16 19:17:59.419567 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:59.419307    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="81b89f31-91f5-4d65-85c3-aba3321b5cd5" containerName="storage-initializer"
Sep 16 19:17:59.419567 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:59.419379    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="72e4a42d-de46-4845-b1d4-6485b4edc309" containerName="kserve-container"
Sep 16 19:17:59.419567 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:59.419391    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="72e4a42d-de46-4845-b1d4-6485b4edc309" containerName="kube-rbac-proxy"
Sep 16 19:17:59.419567 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:59.419400    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="81b89f31-91f5-4d65-85c3-aba3321b5cd5" containerName="kserve-container"
Sep 16 19:17:59.419567 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:59.419409    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="4dd240d8-1a39-455d-bad7-8e89d291909a" containerName="kube-rbac-proxy"
Sep 16 19:17:59.419567 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:59.419419    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="4dd240d8-1a39-455d-bad7-8e89d291909a" containerName="kserve-container"
Sep 16 19:17:59.419567 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:59.419427    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="81b89f31-91f5-4d65-85c3-aba3321b5cd5" containerName="kube-rbac-proxy"
Sep 16 19:17:59.443057 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:59.443026    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-gwfg9"]
Sep 16 19:17:59.443216 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:59.443179    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-gwfg9"
Sep 16 19:17:59.446164 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:59.446134    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\""
Sep 16 19:17:59.446164 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:59.446156    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
Sep 16 19:17:59.502403 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:59.502372    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da6c72d6-5b8f-43a0-a917-9956487c11a5-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-gwfg9\" (UID: \"da6c72d6-5b8f-43a0-a917-9956487c11a5\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-gwfg9"
Sep 16 19:17:59.502538 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:59.502414    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da6c72d6-5b8f-43a0-a917-9956487c11a5-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-gwfg9\" (UID: \"da6c72d6-5b8f-43a0-a917-9956487c11a5\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-gwfg9"
Sep 16 19:17:59.502538 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:59.502434    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da6c72d6-5b8f-43a0-a917-9956487c11a5-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-gwfg9\" (UID: \"da6c72d6-5b8f-43a0-a917-9956487c11a5\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-gwfg9"
Sep 16 19:17:59.502538 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:59.502476    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da6c72d6-5b8f-43a0-a917-9956487c11a5-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-gwfg9\" (UID: \"da6c72d6-5b8f-43a0-a917-9956487c11a5\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-gwfg9"
Sep 16 19:17:59.502642 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:59.502562    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vzwk\" (UniqueName: \"kubernetes.io/projected/da6c72d6-5b8f-43a0-a917-9956487c11a5-kube-api-access-6vzwk\") pod \"isvc-xgboost-predictor-7566d65bf7-gwfg9\" (UID: \"da6c72d6-5b8f-43a0-a917-9956487c11a5\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-gwfg9"
Sep 16 19:17:59.603529 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:59.603489    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da6c72d6-5b8f-43a0-a917-9956487c11a5-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-gwfg9\" (UID: \"da6c72d6-5b8f-43a0-a917-9956487c11a5\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-gwfg9"
Sep 16 19:17:59.603529 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:59.603531    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da6c72d6-5b8f-43a0-a917-9956487c11a5-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-gwfg9\" (UID: \"da6c72d6-5b8f-43a0-a917-9956487c11a5\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-gwfg9"
Sep 16 19:17:59.604081 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:59.603549    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da6c72d6-5b8f-43a0-a917-9956487c11a5-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-gwfg9\" (UID: \"da6c72d6-5b8f-43a0-a917-9956487c11a5\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-gwfg9"
Sep 16 19:17:59.604081 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:59.603602    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vzwk\" (UniqueName: \"kubernetes.io/projected/da6c72d6-5b8f-43a0-a917-9956487c11a5-kube-api-access-6vzwk\") pod \"isvc-xgboost-predictor-7566d65bf7-gwfg9\" (UID: \"da6c72d6-5b8f-43a0-a917-9956487c11a5\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-gwfg9"
Sep 16 19:17:59.604081 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:59.603652    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da6c72d6-5b8f-43a0-a917-9956487c11a5-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-gwfg9\" (UID: \"da6c72d6-5b8f-43a0-a917-9956487c11a5\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-gwfg9"
Sep 16 19:17:59.604276 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:59.604098    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da6c72d6-5b8f-43a0-a917-9956487c11a5-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-gwfg9\" (UID: \"da6c72d6-5b8f-43a0-a917-9956487c11a5\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-gwfg9"
Sep 16 19:17:59.604342 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:59.604328    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da6c72d6-5b8f-43a0-a917-9956487c11a5-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-gwfg9\" (UID: \"da6c72d6-5b8f-43a0-a917-9956487c11a5\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-gwfg9"
Sep 16 19:17:59.604401 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:59.604376    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da6c72d6-5b8f-43a0-a917-9956487c11a5-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-gwfg9\" (UID: \"da6c72d6-5b8f-43a0-a917-9956487c11a5\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-gwfg9"
Sep 16 19:17:59.606283 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:59.606258    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da6c72d6-5b8f-43a0-a917-9956487c11a5-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-gwfg9\" (UID: \"da6c72d6-5b8f-43a0-a917-9956487c11a5\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-gwfg9"
Sep 16 19:17:59.613987 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:59.613965    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vzwk\" (UniqueName: \"kubernetes.io/projected/da6c72d6-5b8f-43a0-a917-9956487c11a5-kube-api-access-6vzwk\") pod \"isvc-xgboost-predictor-7566d65bf7-gwfg9\" (UID: \"da6c72d6-5b8f-43a0-a917-9956487c11a5\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-gwfg9"
Sep 16 19:17:59.753970 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:17:59.753887    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-gwfg9"
Sep 16 19:18:00.055760 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:00.055668    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-gwfg9"]
Sep 16 19:18:00.058164 ip-10-0-129-190 kubenswrapper[2586]: W0916 19:18:00.058129    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda6c72d6_5b8f_43a0_a917_9956487c11a5.slice/crio-e6523d867dc03987b519844fd189c75305f93a323a4738406e2bb71fb1ab4f25 WatchSource:0}: Error finding container e6523d867dc03987b519844fd189c75305f93a323a4738406e2bb71fb1ab4f25: Status 404 returned error can't find the container with id e6523d867dc03987b519844fd189c75305f93a323a4738406e2bb71fb1ab4f25
Sep 16 19:18:00.648966 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:00.648930    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-gwfg9" event={"ID":"da6c72d6-5b8f-43a0-a917-9956487c11a5","Type":"ContainerStarted","Data":"8bebad5d1fc6168579b243e13fc891de86ec5fcb18b2adfe6ff43299c37f67fb"}
Sep 16 19:18:00.649340 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:00.649005    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-gwfg9" event={"ID":"da6c72d6-5b8f-43a0-a917-9956487c11a5","Type":"ContainerStarted","Data":"e6523d867dc03987b519844fd189c75305f93a323a4738406e2bb71fb1ab4f25"}
Sep 16 19:18:04.670224 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:04.670192    2586 generic.go:356] "Generic (PLEG): container finished" podID="da6c72d6-5b8f-43a0-a917-9956487c11a5" containerID="8bebad5d1fc6168579b243e13fc891de86ec5fcb18b2adfe6ff43299c37f67fb" exitCode=0
Sep 16 19:18:04.670729 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:04.670272    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-gwfg9" event={"ID":"da6c72d6-5b8f-43a0-a917-9956487c11a5","Type":"ContainerDied","Data":"8bebad5d1fc6168579b243e13fc891de86ec5fcb18b2adfe6ff43299c37f67fb"}
Sep 16 19:18:08.515345 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:08.515306    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9" podUID="1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 16 19:18:18.516156 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:18.516114    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9" podUID="1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 16 19:18:22.767820 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:22.767734    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-gwfg9" event={"ID":"da6c72d6-5b8f-43a0-a917-9956487c11a5","Type":"ContainerStarted","Data":"1eeb83ad76a1e8edd084f140228e1a55c11339a732cc60f484ae293a6453b3c2"}
Sep 16 19:18:22.768208 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:22.767843    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-gwfg9" event={"ID":"da6c72d6-5b8f-43a0-a917-9956487c11a5","Type":"ContainerStarted","Data":"e76079a7499900228905b117236ae6423f603bf4f2080a78fd218e24f9dbb8ae"}
Sep 16 19:18:22.768208 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:22.768114    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-gwfg9"
Sep 16 19:18:22.768208 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:22.768158    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-gwfg9"
Sep 16 19:18:22.769625 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:22.769598    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-gwfg9" podUID="da6c72d6-5b8f-43a0-a917-9956487c11a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 16 19:18:22.804396 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:22.804341    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-gwfg9" podStartSLOduration=1.768317256 podStartE2EDuration="23.804327787s" totalImagesPullingTime="17.680391821s" totalInitContainerRuntime="4.35561871s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:17:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:18:22.802817039 +0000 UTC m=+2278.246678616" watchObservedRunningTime="2026-09-16 19:18:22.804327787 +0000 UTC m=+2278.248189364"
Sep 16 19:18:23.773290 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:23.773252    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-gwfg9" podUID="da6c72d6-5b8f-43a0-a917-9956487c11a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 16 19:18:28.516127 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:28.516087    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9" podUID="1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 16 19:18:28.777826 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:28.777733    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-gwfg9"
Sep 16 19:18:28.778215 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:28.778177    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-gwfg9" podUID="da6c72d6-5b8f-43a0-a917-9956487c11a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 16 19:18:38.515299 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:38.515259    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9" podUID="1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 16 19:18:38.779097 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:38.778988    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-gwfg9" podUID="da6c72d6-5b8f-43a0-a917-9956487c11a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 16 19:18:48.516585 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:48.516551    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9"
Sep 16 19:18:48.779075 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:48.778988    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-gwfg9" podUID="da6c72d6-5b8f-43a0-a917-9956487c11a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 16 19:18:50.774953 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:50.774919    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9"]
Sep 16 19:18:50.775449 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:50.775353    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9" podUID="1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd" containerName="kserve-container" containerID="cri-o://205d8e44ad6111539481a8b4eee0eec343f2d85fcb2f8ea5ec59463d9f5b826b" gracePeriod=30
Sep 16 19:18:50.775644 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:50.775607    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9" podUID="1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd" containerName="kube-rbac-proxy" containerID="cri-o://15aedbb3060bc1742888a7e7410395ebd8d8820938f5eaad8cca11c61f184ff1" gracePeriod=30
Sep 16 19:18:50.903915 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:50.903875    2586 generic.go:356] "Generic (PLEG): container finished" podID="1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd" containerID="15aedbb3060bc1742888a7e7410395ebd8d8820938f5eaad8cca11c61f184ff1" exitCode=2
Sep 16 19:18:50.904071 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:50.903945    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9" event={"ID":"1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd","Type":"ContainerDied","Data":"15aedbb3060bc1742888a7e7410395ebd8d8820938f5eaad8cca11c61f184ff1"}
Sep 16 19:18:53.510756 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:53.510710    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9" podUID="1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.54:8643/healthz\": dial tcp 10.134.0.54:8643: connect: connection refused"
Sep 16 19:18:55.627276 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:55.627247    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9"
Sep 16 19:18:55.701705 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:55.701616    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd-proxy-tls\") pod \"1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd\" (UID: \"1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd\") "
Sep 16 19:18:55.701705 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:55.701657    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9z2f\" (UniqueName: \"kubernetes.io/projected/1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd-kube-api-access-n9z2f\") pod \"1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd\" (UID: \"1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd\") "
Sep 16 19:18:55.701925 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:55.701717    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd-kserve-provision-location\") pod \"1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd\" (UID: \"1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd\") "
Sep 16 19:18:55.701925 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:55.701748    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd-cabundle-cert\") pod \"1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd\" (UID: \"1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd\") "
Sep 16 19:18:55.701925 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:55.701770    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd-kube-rbac-proxy-sar-config\") pod \"1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd\" (UID: \"1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd\") "
Sep 16 19:18:55.702144 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:55.702109    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd" (UID: "1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:18:55.702217 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:55.702148    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd" (UID: "1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:18:55.702217 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:55.702177    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd" (UID: "1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:18:55.703832 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:55.703808    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd-kube-api-access-n9z2f" (OuterVolumeSpecName: "kube-api-access-n9z2f") pod "1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd" (UID: "1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd"). InnerVolumeSpecName "kube-api-access-n9z2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:18:55.703832 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:55.703822    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd" (UID: "1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:18:55.802988 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:55.802955    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:18:55.802988 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:55.802981    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9z2f\" (UniqueName: \"kubernetes.io/projected/1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd-kube-api-access-n9z2f\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:18:55.802988 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:55.802993    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:18:55.803223 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:55.803006    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:18:55.803223 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:55.803016    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:18:55.930586 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:55.930548    2586 generic.go:356] "Generic (PLEG): container finished" podID="1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd" containerID="205d8e44ad6111539481a8b4eee0eec343f2d85fcb2f8ea5ec59463d9f5b826b" exitCode=0
Sep 16 19:18:55.930761 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:55.930624    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9"
Sep 16 19:18:55.930761 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:55.930632    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9" event={"ID":"1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd","Type":"ContainerDied","Data":"205d8e44ad6111539481a8b4eee0eec343f2d85fcb2f8ea5ec59463d9f5b826b"}
Sep 16 19:18:55.930761 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:55.930709    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9" event={"ID":"1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd","Type":"ContainerDied","Data":"32032c976d403e5585bd47e91ad15393941def13c67fce57da6f930337a7805e"}
Sep 16 19:18:55.930761 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:55.930749    2586 scope.go:117] "RemoveContainer" containerID="15aedbb3060bc1742888a7e7410395ebd8d8820938f5eaad8cca11c61f184ff1"
Sep 16 19:18:55.938970 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:55.938938    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-qpcx4"]
Sep 16 19:18:55.939460 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:55.939438    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd" containerName="kserve-container"
Sep 16 19:18:55.939460 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:55.939461    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd" containerName="kserve-container"
Sep 16 19:18:55.939700 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:55.939495    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd" containerName="storage-initializer"
Sep 16 19:18:55.939700 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:55.939504    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd" containerName="storage-initializer"
Sep 16 19:18:55.939700 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:55.939518    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd" containerName="kube-rbac-proxy"
Sep 16 19:18:55.939700 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:55.939527    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd" containerName="kube-rbac-proxy"
Sep 16 19:18:55.939700 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:55.939602    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd" containerName="kube-rbac-proxy"
Sep 16 19:18:55.939700 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:55.939613    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd" containerName="kserve-container"
Sep 16 19:18:55.942183 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:55.942167    2586 scope.go:117] "RemoveContainer" containerID="205d8e44ad6111539481a8b4eee0eec343f2d85fcb2f8ea5ec59463d9f5b826b"
Sep 16 19:18:55.943430 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:55.942808    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-qpcx4"
Sep 16 19:18:55.945613 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:55.945588    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 16 19:18:55.945724 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:55.945611    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Sep 16 19:18:55.952380 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:55.952359    2586 scope.go:117] "RemoveContainer" containerID="cafc756dd6bd14c563f064ae3587c051d053ee11b789e64be2622af1d964dbd4"
Sep 16 19:18:55.957950 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:55.957925    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-qpcx4"]
Sep 16 19:18:55.961861 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:55.961836    2586 scope.go:117] "RemoveContainer" containerID="15aedbb3060bc1742888a7e7410395ebd8d8820938f5eaad8cca11c61f184ff1"
Sep 16 19:18:55.962209 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:18:55.962183    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15aedbb3060bc1742888a7e7410395ebd8d8820938f5eaad8cca11c61f184ff1\": container with ID starting with 15aedbb3060bc1742888a7e7410395ebd8d8820938f5eaad8cca11c61f184ff1 not found: ID does not exist" containerID="15aedbb3060bc1742888a7e7410395ebd8d8820938f5eaad8cca11c61f184ff1"
Sep 16 19:18:55.962316 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:55.962221    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15aedbb3060bc1742888a7e7410395ebd8d8820938f5eaad8cca11c61f184ff1"} err="failed to get container status \"15aedbb3060bc1742888a7e7410395ebd8d8820938f5eaad8cca11c61f184ff1\": rpc error: code = NotFound desc = could not find container \"15aedbb3060bc1742888a7e7410395ebd8d8820938f5eaad8cca11c61f184ff1\": container with ID starting with 15aedbb3060bc1742888a7e7410395ebd8d8820938f5eaad8cca11c61f184ff1 not found: ID does not exist"
Sep 16 19:18:55.962316 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:55.962260    2586 scope.go:117] "RemoveContainer" containerID="205d8e44ad6111539481a8b4eee0eec343f2d85fcb2f8ea5ec59463d9f5b826b"
Sep 16 19:18:55.962567 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:18:55.962549    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"205d8e44ad6111539481a8b4eee0eec343f2d85fcb2f8ea5ec59463d9f5b826b\": container with ID starting with 205d8e44ad6111539481a8b4eee0eec343f2d85fcb2f8ea5ec59463d9f5b826b not found: ID does not exist" containerID="205d8e44ad6111539481a8b4eee0eec343f2d85fcb2f8ea5ec59463d9f5b826b"
Sep 16 19:18:55.962636 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:55.962572    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"205d8e44ad6111539481a8b4eee0eec343f2d85fcb2f8ea5ec59463d9f5b826b"} err="failed to get container status \"205d8e44ad6111539481a8b4eee0eec343f2d85fcb2f8ea5ec59463d9f5b826b\": rpc error: code = NotFound desc = could not find container \"205d8e44ad6111539481a8b4eee0eec343f2d85fcb2f8ea5ec59463d9f5b826b\": container with ID starting with 205d8e44ad6111539481a8b4eee0eec343f2d85fcb2f8ea5ec59463d9f5b826b not found: ID does not exist"
Sep 16 19:18:55.962636 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:55.962589    2586 scope.go:117] "RemoveContainer" containerID="cafc756dd6bd14c563f064ae3587c051d053ee11b789e64be2622af1d964dbd4"
Sep 16 19:18:55.962865 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:18:55.962847    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cafc756dd6bd14c563f064ae3587c051d053ee11b789e64be2622af1d964dbd4\": container with ID starting with cafc756dd6bd14c563f064ae3587c051d053ee11b789e64be2622af1d964dbd4 not found: ID does not exist" containerID="cafc756dd6bd14c563f064ae3587c051d053ee11b789e64be2622af1d964dbd4"
Sep 16 19:18:55.962926 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:55.962873    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cafc756dd6bd14c563f064ae3587c051d053ee11b789e64be2622af1d964dbd4"} err="failed to get container status \"cafc756dd6bd14c563f064ae3587c051d053ee11b789e64be2622af1d964dbd4\": rpc error: code = NotFound desc = could not find container \"cafc756dd6bd14c563f064ae3587c051d053ee11b789e64be2622af1d964dbd4\": container with ID starting with cafc756dd6bd14c563f064ae3587c051d053ee11b789e64be2622af1d964dbd4 not found: ID does not exist"
Sep 16 19:18:56.004871 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:56.004823    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f998575-a92d-4cec-9fad-eafd7e481461-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-854695c447-qpcx4\" (UID: \"9f998575-a92d-4cec-9fad-eafd7e481461\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-qpcx4"
Sep 16 19:18:56.005045 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:56.004971    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckm4c\" (UniqueName: \"kubernetes.io/projected/9f998575-a92d-4cec-9fad-eafd7e481461-kube-api-access-ckm4c\") pod \"isvc-canary-stop-v2-predictor-854695c447-qpcx4\" (UID: \"9f998575-a92d-4cec-9fad-eafd7e481461\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-qpcx4"
Sep 16 19:18:56.005260 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:56.005192    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f998575-a92d-4cec-9fad-eafd7e481461-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-854695c447-qpcx4\" (UID: \"9f998575-a92d-4cec-9fad-eafd7e481461\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-qpcx4"
Sep 16 19:18:56.005385 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:56.005278    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f998575-a92d-4cec-9fad-eafd7e481461-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-854695c447-qpcx4\" (UID: \"9f998575-a92d-4cec-9fad-eafd7e481461\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-qpcx4"
Sep 16 19:18:56.005385 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:56.005315    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f998575-a92d-4cec-9fad-eafd7e481461-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-854695c447-qpcx4\" (UID: \"9f998575-a92d-4cec-9fad-eafd7e481461\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-qpcx4"
Sep 16 19:18:56.007535 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:56.007496    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9"]
Sep 16 19:18:56.009983 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:56.009960    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-9qsr9"]
Sep 16 19:18:56.089893 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:56.089854    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-8l7nw"]
Sep 16 19:18:56.093781 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:56.093758    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-8l7nw"
Sep 16 19:18:56.097317 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:56.097292    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 16 19:18:56.105807 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:56.105784    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f998575-a92d-4cec-9fad-eafd7e481461-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-854695c447-qpcx4\" (UID: \"9f998575-a92d-4cec-9fad-eafd7e481461\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-qpcx4"
Sep 16 19:18:56.105920 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:56.105822    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckm4c\" (UniqueName: \"kubernetes.io/projected/9f998575-a92d-4cec-9fad-eafd7e481461-kube-api-access-ckm4c\") pod \"isvc-canary-stop-v2-predictor-854695c447-qpcx4\" (UID: \"9f998575-a92d-4cec-9fad-eafd7e481461\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-qpcx4"
Sep 16 19:18:56.105920 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:56.105889    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f998575-a92d-4cec-9fad-eafd7e481461-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-854695c447-qpcx4\" (UID: \"9f998575-a92d-4cec-9fad-eafd7e481461\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-qpcx4"
Sep 16 19:18:56.106056 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:56.105937    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f998575-a92d-4cec-9fad-eafd7e481461-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-854695c447-qpcx4\" (UID: \"9f998575-a92d-4cec-9fad-eafd7e481461\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-qpcx4"
Sep 16 19:18:56.106056 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:56.106001    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f998575-a92d-4cec-9fad-eafd7e481461-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-854695c447-qpcx4\" (UID: \"9f998575-a92d-4cec-9fad-eafd7e481461\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-qpcx4"
Sep 16 19:18:56.106175 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:18:56.106093    2586 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-serving-cert: secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Sep 16 19:18:56.106175 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:18:56.106157    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f998575-a92d-4cec-9fad-eafd7e481461-proxy-tls podName:9f998575-a92d-4cec-9fad-eafd7e481461 nodeName:}" failed. No retries permitted until 2026-09-16 19:18:56.606136846 +0000 UTC m=+2312.049998421 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9f998575-a92d-4cec-9fad-eafd7e481461-proxy-tls") pod "isvc-canary-stop-v2-predictor-854695c447-qpcx4" (UID: "9f998575-a92d-4cec-9fad-eafd7e481461") : secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Sep 16 19:18:56.106391 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:56.106369    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f998575-a92d-4cec-9fad-eafd7e481461-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-854695c447-qpcx4\" (UID: \"9f998575-a92d-4cec-9fad-eafd7e481461\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-qpcx4"
Sep 16 19:18:56.106580 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:56.106560    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f998575-a92d-4cec-9fad-eafd7e481461-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-854695c447-qpcx4\" (UID: \"9f998575-a92d-4cec-9fad-eafd7e481461\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-qpcx4"
Sep 16 19:18:56.106621 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:56.106580    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f998575-a92d-4cec-9fad-eafd7e481461-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-854695c447-qpcx4\" (UID: \"9f998575-a92d-4cec-9fad-eafd7e481461\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-qpcx4"
Sep 16 19:18:56.112900 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:56.112879    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-8l7nw"]
Sep 16 19:18:56.120093 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:56.120069    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckm4c\" (UniqueName: \"kubernetes.io/projected/9f998575-a92d-4cec-9fad-eafd7e481461-kube-api-access-ckm4c\") pod \"isvc-canary-stop-v2-predictor-854695c447-qpcx4\" (UID: \"9f998575-a92d-4cec-9fad-eafd7e481461\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-qpcx4"
Sep 16 19:18:56.206716 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:56.206642    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3fa70ab6-6d52-4039-b747-30046038d673-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-8l7nw\" (UID: \"3fa70ab6-6d52-4039-b747-30046038d673\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-8l7nw"
Sep 16 19:18:56.206716 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:56.206677    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3fa70ab6-6d52-4039-b747-30046038d673-cabundle-cert\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-8l7nw\" (UID: \"3fa70ab6-6d52-4039-b747-30046038d673\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-8l7nw"
Sep 16 19:18:56.206922 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:56.206733    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fa70ab6-6d52-4039-b747-30046038d673-proxy-tls\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-8l7nw\" (UID: \"3fa70ab6-6d52-4039-b747-30046038d673\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-8l7nw"
Sep 16 19:18:56.206922 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:56.206806    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fa70ab6-6d52-4039-b747-30046038d673-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-8l7nw\" (UID: \"3fa70ab6-6d52-4039-b747-30046038d673\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-8l7nw"
Sep 16 19:18:56.206922 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:56.206829    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcbdq\" (UniqueName: \"kubernetes.io/projected/3fa70ab6-6d52-4039-b747-30046038d673-kube-api-access-mcbdq\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-8l7nw\" (UID: \"3fa70ab6-6d52-4039-b747-30046038d673\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-8l7nw"
Sep 16 19:18:56.308133 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:56.308091    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fa70ab6-6d52-4039-b747-30046038d673-proxy-tls\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-8l7nw\" (UID: \"3fa70ab6-6d52-4039-b747-30046038d673\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-8l7nw"
Sep 16 19:18:56.308348 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:56.308149    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fa70ab6-6d52-4039-b747-30046038d673-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-8l7nw\" (UID: \"3fa70ab6-6d52-4039-b747-30046038d673\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-8l7nw"
Sep 16 19:18:56.308348 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:56.308182    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcbdq\" (UniqueName: \"kubernetes.io/projected/3fa70ab6-6d52-4039-b747-30046038d673-kube-api-access-mcbdq\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-8l7nw\" (UID: \"3fa70ab6-6d52-4039-b747-30046038d673\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-8l7nw"
Sep 16 19:18:56.308451 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:56.308401    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3fa70ab6-6d52-4039-b747-30046038d673-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-8l7nw\" (UID: \"3fa70ab6-6d52-4039-b747-30046038d673\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-8l7nw"
Sep 16 19:18:56.308503 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:56.308454    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3fa70ab6-6d52-4039-b747-30046038d673-cabundle-cert\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-8l7nw\" (UID: \"3fa70ab6-6d52-4039-b747-30046038d673\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-8l7nw"
Sep 16 19:18:56.308652 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:56.308631    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fa70ab6-6d52-4039-b747-30046038d673-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-8l7nw\" (UID: \"3fa70ab6-6d52-4039-b747-30046038d673\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-8l7nw"
Sep 16 19:18:56.309068 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:56.309047    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3fa70ab6-6d52-4039-b747-30046038d673-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-8l7nw\" (UID: \"3fa70ab6-6d52-4039-b747-30046038d673\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-8l7nw"
Sep 16 19:18:56.309172 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:56.309155    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3fa70ab6-6d52-4039-b747-30046038d673-cabundle-cert\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-8l7nw\" (UID: \"3fa70ab6-6d52-4039-b747-30046038d673\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-8l7nw"
Sep 16 19:18:56.310770 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:56.310752    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fa70ab6-6d52-4039-b747-30046038d673-proxy-tls\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-8l7nw\" (UID: \"3fa70ab6-6d52-4039-b747-30046038d673\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-8l7nw"
Sep 16 19:18:56.325681 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:56.325661    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcbdq\" (UniqueName: \"kubernetes.io/projected/3fa70ab6-6d52-4039-b747-30046038d673-kube-api-access-mcbdq\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-8l7nw\" (UID: \"3fa70ab6-6d52-4039-b747-30046038d673\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-8l7nw"
Sep 16 19:18:56.405154 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:56.405120    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-8l7nw"
Sep 16 19:18:56.611191 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:56.611154    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f998575-a92d-4cec-9fad-eafd7e481461-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-854695c447-qpcx4\" (UID: \"9f998575-a92d-4cec-9fad-eafd7e481461\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-qpcx4"
Sep 16 19:18:56.613589 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:56.613560    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f998575-a92d-4cec-9fad-eafd7e481461-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-854695c447-qpcx4\" (UID: \"9f998575-a92d-4cec-9fad-eafd7e481461\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-qpcx4"
Sep 16 19:18:56.689982 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:56.689957    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-8l7nw"]
Sep 16 19:18:56.692106 ip-10-0-129-190 kubenswrapper[2586]: W0916 19:18:56.692075    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fa70ab6_6d52_4039_b747_30046038d673.slice/crio-fe27982b57ad8ae9426af824c7c813171622e316c12e286a75ace75163568047 WatchSource:0}: Error finding container fe27982b57ad8ae9426af824c7c813171622e316c12e286a75ace75163568047: Status 404 returned error can't find the container with id fe27982b57ad8ae9426af824c7c813171622e316c12e286a75ace75163568047
Sep 16 19:18:56.855774 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:56.855732    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-qpcx4"
Sep 16 19:18:56.939273 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:56.939240    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-8l7nw" event={"ID":"3fa70ab6-6d52-4039-b747-30046038d673","Type":"ContainerStarted","Data":"2d8425604cd35df637bb8015d1773c7a7ff7cd6b825ead916f0a0c73c51d1999"}
Sep 16 19:18:56.939440 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:56.939305    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-8l7nw" event={"ID":"3fa70ab6-6d52-4039-b747-30046038d673","Type":"ContainerStarted","Data":"fe27982b57ad8ae9426af824c7c813171622e316c12e286a75ace75163568047"}
Sep 16 19:18:57.093275 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:57.093220    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd" path="/var/lib/kubelet/pods/1a8d984e-ef64-429b-9f5a-ffd8ba8e6dfd/volumes"
Sep 16 19:18:57.138772 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:57.138741    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-qpcx4"]
Sep 16 19:18:57.139587 ip-10-0-129-190 kubenswrapper[2586]: W0916 19:18:57.139561    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f998575_a92d_4cec_9fad_eafd7e481461.slice/crio-86375702efe21720cfd1a8d099a6d3990813dcde296f251b7c358a8deb148482 WatchSource:0}: Error finding container 86375702efe21720cfd1a8d099a6d3990813dcde296f251b7c358a8deb148482: Status 404 returned error can't find the container with id 86375702efe21720cfd1a8d099a6d3990813dcde296f251b7c358a8deb148482
Sep 16 19:18:57.946936 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:57.946898    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-qpcx4" event={"ID":"9f998575-a92d-4cec-9fad-eafd7e481461","Type":"ContainerStarted","Data":"21e82f3b14563f2a41a9a2795ed27164db9a4de8cba3dcc2df4b6c2505f8902c"}
Sep 16 19:18:57.947387 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:57.946958    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-qpcx4" event={"ID":"9f998575-a92d-4cec-9fad-eafd7e481461","Type":"ContainerStarted","Data":"86375702efe21720cfd1a8d099a6d3990813dcde296f251b7c358a8deb148482"}
Sep 16 19:18:58.778479 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:18:58.778440    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-gwfg9" podUID="da6c72d6-5b8f-43a0-a917-9956487c11a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 16 19:19:00.968061 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:00.967972    2586 generic.go:356] "Generic (PLEG): container finished" podID="3fa70ab6-6d52-4039-b747-30046038d673" containerID="2d8425604cd35df637bb8015d1773c7a7ff7cd6b825ead916f0a0c73c51d1999" exitCode=0
Sep 16 19:19:00.968511 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:00.968051    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-8l7nw" event={"ID":"3fa70ab6-6d52-4039-b747-30046038d673","Type":"ContainerDied","Data":"2d8425604cd35df637bb8015d1773c7a7ff7cd6b825ead916f0a0c73c51d1999"}
Sep 16 19:19:01.974749 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:01.974717    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-8l7nw" event={"ID":"3fa70ab6-6d52-4039-b747-30046038d673","Type":"ContainerStarted","Data":"138b169163f2d3301761f19f4e197ba08db1c28c9c0cd03d25ac085e58d64a79"}
Sep 16 19:19:01.975186 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:01.974792    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-8l7nw" event={"ID":"3fa70ab6-6d52-4039-b747-30046038d673","Type":"ContainerStarted","Data":"a902db437ffb68e9d2c44ddaeb7e53482357887484cc6dea5e5b767ed41eba9e"}
Sep 16 19:19:01.975186 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:01.975122    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-8l7nw"
Sep 16 19:19:01.975365 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:01.975256    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-8l7nw"
Sep 16 19:19:01.976439 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:01.976413    2586 generic.go:356] "Generic (PLEG): container finished" podID="9f998575-a92d-4cec-9fad-eafd7e481461" containerID="21e82f3b14563f2a41a9a2795ed27164db9a4de8cba3dcc2df4b6c2505f8902c" exitCode=0
Sep 16 19:19:01.976530 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:01.976487    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-qpcx4" event={"ID":"9f998575-a92d-4cec-9fad-eafd7e481461","Type":"ContainerDied","Data":"21e82f3b14563f2a41a9a2795ed27164db9a4de8cba3dcc2df4b6c2505f8902c"}
Sep 16 19:19:01.976779 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:01.976752    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-8l7nw" podUID="3fa70ab6-6d52-4039-b747-30046038d673" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 16 19:19:02.033975 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:02.033912    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-8l7nw" podStartSLOduration=2.076822763 podStartE2EDuration="6.033889395s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.957066632s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:18:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:19:01.998476542 +0000 UTC m=+2317.442338119" watchObservedRunningTime="2026-09-16 19:19:02.033889395 +0000 UTC m=+2317.477750971"
Sep 16 19:19:02.983786 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:02.983744    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-qpcx4" event={"ID":"9f998575-a92d-4cec-9fad-eafd7e481461","Type":"ContainerStarted","Data":"c5916b17e13ba4d48ab1717efa40b4ea4a5d94a91f08c96340b6b67c131d469a"}
Sep 16 19:19:02.984169 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:02.983825    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-qpcx4" event={"ID":"9f998575-a92d-4cec-9fad-eafd7e481461","Type":"ContainerStarted","Data":"4f81fb9e9af1566ed546c5465609206a44a256ec032bb34b19a5cc95011d12b0"}
Sep 16 19:19:02.984281 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:02.984257    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-qpcx4"
Sep 16 19:19:02.984350 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:02.984312    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-qpcx4"
Sep 16 19:19:02.984350 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:02.984252    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-8l7nw" podUID="3fa70ab6-6d52-4039-b747-30046038d673" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 16 19:19:02.985551 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:02.985528    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-qpcx4" podUID="9f998575-a92d-4cec-9fad-eafd7e481461" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 16 19:19:03.023286 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:03.023205    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-qpcx4" podStartSLOduration=4.10651676 podStartE2EDuration="8.02318438s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.91666762s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:18:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:19:03.022157982 +0000 UTC m=+2318.466019569" watchObservedRunningTime="2026-09-16 19:19:03.02318438 +0000 UTC m=+2318.467045960"
Sep 16 19:19:03.989109 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:03.989059    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-qpcx4" podUID="9f998575-a92d-4cec-9fad-eafd7e481461" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 16 19:19:07.988958 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:07.988926    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-8l7nw"
Sep 16 19:19:07.989426 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:07.989396    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-8l7nw" podUID="3fa70ab6-6d52-4039-b747-30046038d673" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 16 19:19:08.778750 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:08.778707    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-gwfg9" podUID="da6c72d6-5b8f-43a0-a917-9956487c11a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 16 19:19:08.993703 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:08.993671    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-qpcx4"
Sep 16 19:19:08.994147 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:08.994114    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-qpcx4" podUID="9f998575-a92d-4cec-9fad-eafd7e481461" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 16 19:19:17.990365 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:17.990278    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-8l7nw" podUID="3fa70ab6-6d52-4039-b747-30046038d673" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 16 19:19:18.783050 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:18.779416    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-gwfg9" podUID="da6c72d6-5b8f-43a0-a917-9956487c11a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 16 19:19:18.994531 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:18.994487    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-qpcx4" podUID="9f998575-a92d-4cec-9fad-eafd7e481461" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 16 19:19:27.989480 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:27.989435    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-8l7nw" podUID="3fa70ab6-6d52-4039-b747-30046038d673" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 16 19:19:28.778395 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:28.778365    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-gwfg9"
Sep 16 19:19:28.994494 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:28.994450    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-qpcx4" podUID="9f998575-a92d-4cec-9fad-eafd7e481461" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 16 19:19:29.436596 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:29.436562    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-gwfg9"]
Sep 16 19:19:29.436812 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:29.436786    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-gwfg9" podUID="da6c72d6-5b8f-43a0-a917-9956487c11a5" containerName="kserve-container" containerID="cri-o://e76079a7499900228905b117236ae6423f603bf4f2080a78fd218e24f9dbb8ae" gracePeriod=30
Sep 16 19:19:29.436896 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:29.436841    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-gwfg9" podUID="da6c72d6-5b8f-43a0-a917-9956487c11a5" containerName="kube-rbac-proxy" containerID="cri-o://1eeb83ad76a1e8edd084f140228e1a55c11339a732cc60f484ae293a6453b3c2" gracePeriod=30
Sep 16 19:19:30.122238 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:30.122195    2586 generic.go:356] "Generic (PLEG): container finished" podID="da6c72d6-5b8f-43a0-a917-9956487c11a5" containerID="1eeb83ad76a1e8edd084f140228e1a55c11339a732cc60f484ae293a6453b3c2" exitCode=2
Sep 16 19:19:30.122593 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:30.122276    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-gwfg9" event={"ID":"da6c72d6-5b8f-43a0-a917-9956487c11a5","Type":"ContainerDied","Data":"1eeb83ad76a1e8edd084f140228e1a55c11339a732cc60f484ae293a6453b3c2"}
Sep 16 19:19:33.773517 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:33.773470    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-gwfg9" podUID="da6c72d6-5b8f-43a0-a917-9956487c11a5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.55:8643/healthz\": dial tcp 10.134.0.55:8643: connect: connection refused"
Sep 16 19:19:34.143601 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:34.143568    2586 generic.go:356] "Generic (PLEG): container finished" podID="da6c72d6-5b8f-43a0-a917-9956487c11a5" containerID="e76079a7499900228905b117236ae6423f603bf4f2080a78fd218e24f9dbb8ae" exitCode=0
Sep 16 19:19:34.143761 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:34.143636    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-gwfg9" event={"ID":"da6c72d6-5b8f-43a0-a917-9956487c11a5","Type":"ContainerDied","Data":"e76079a7499900228905b117236ae6423f603bf4f2080a78fd218e24f9dbb8ae"}
Sep 16 19:19:34.180178 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:34.180158    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-gwfg9"
Sep 16 19:19:34.257317 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:34.257277    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da6c72d6-5b8f-43a0-a917-9956487c11a5-cabundle-cert\") pod \"da6c72d6-5b8f-43a0-a917-9956487c11a5\" (UID: \"da6c72d6-5b8f-43a0-a917-9956487c11a5\") "
Sep 16 19:19:34.257506 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:34.257326    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da6c72d6-5b8f-43a0-a917-9956487c11a5-kserve-provision-location\") pod \"da6c72d6-5b8f-43a0-a917-9956487c11a5\" (UID: \"da6c72d6-5b8f-43a0-a917-9956487c11a5\") "
Sep 16 19:19:34.257506 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:34.257351    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vzwk\" (UniqueName: \"kubernetes.io/projected/da6c72d6-5b8f-43a0-a917-9956487c11a5-kube-api-access-6vzwk\") pod \"da6c72d6-5b8f-43a0-a917-9956487c11a5\" (UID: \"da6c72d6-5b8f-43a0-a917-9956487c11a5\") "
Sep 16 19:19:34.257506 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:34.257406    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da6c72d6-5b8f-43a0-a917-9956487c11a5-kube-rbac-proxy-sar-config\") pod \"da6c72d6-5b8f-43a0-a917-9956487c11a5\" (UID: \"da6c72d6-5b8f-43a0-a917-9956487c11a5\") "
Sep 16 19:19:34.257506 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:34.257451    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da6c72d6-5b8f-43a0-a917-9956487c11a5-proxy-tls\") pod \"da6c72d6-5b8f-43a0-a917-9956487c11a5\" (UID: \"da6c72d6-5b8f-43a0-a917-9956487c11a5\") "
Sep 16 19:19:34.257715 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:34.257648    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da6c72d6-5b8f-43a0-a917-9956487c11a5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "da6c72d6-5b8f-43a0-a917-9956487c11a5" (UID: "da6c72d6-5b8f-43a0-a917-9956487c11a5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:19:34.257715 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:34.257683    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da6c72d6-5b8f-43a0-a917-9956487c11a5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "da6c72d6-5b8f-43a0-a917-9956487c11a5" (UID: "da6c72d6-5b8f-43a0-a917-9956487c11a5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:19:34.257808 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:34.257726    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da6c72d6-5b8f-43a0-a917-9956487c11a5-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:19:34.257808 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:34.257782    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da6c72d6-5b8f-43a0-a917-9956487c11a5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "da6c72d6-5b8f-43a0-a917-9956487c11a5" (UID: "da6c72d6-5b8f-43a0-a917-9956487c11a5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:19:34.259587 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:34.259569    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da6c72d6-5b8f-43a0-a917-9956487c11a5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "da6c72d6-5b8f-43a0-a917-9956487c11a5" (UID: "da6c72d6-5b8f-43a0-a917-9956487c11a5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:19:34.259666 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:34.259651    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da6c72d6-5b8f-43a0-a917-9956487c11a5-kube-api-access-6vzwk" (OuterVolumeSpecName: "kube-api-access-6vzwk") pod "da6c72d6-5b8f-43a0-a917-9956487c11a5" (UID: "da6c72d6-5b8f-43a0-a917-9956487c11a5"). InnerVolumeSpecName "kube-api-access-6vzwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:19:34.359072 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:34.359039    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da6c72d6-5b8f-43a0-a917-9956487c11a5-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:19:34.359072 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:34.359071    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vzwk\" (UniqueName: \"kubernetes.io/projected/da6c72d6-5b8f-43a0-a917-9956487c11a5-kube-api-access-6vzwk\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:19:34.359289 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:34.359081    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da6c72d6-5b8f-43a0-a917-9956487c11a5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:19:34.359289 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:34.359093    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da6c72d6-5b8f-43a0-a917-9956487c11a5-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:19:35.150396 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:35.150365    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-gwfg9" event={"ID":"da6c72d6-5b8f-43a0-a917-9956487c11a5","Type":"ContainerDied","Data":"e6523d867dc03987b519844fd189c75305f93a323a4738406e2bb71fb1ab4f25"}
Sep 16 19:19:35.150396 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:35.150386    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-gwfg9"
Sep 16 19:19:35.150847 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:35.150434    2586 scope.go:117] "RemoveContainer" containerID="1eeb83ad76a1e8edd084f140228e1a55c11339a732cc60f484ae293a6453b3c2"
Sep 16 19:19:35.158614 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:35.158598    2586 scope.go:117] "RemoveContainer" containerID="e76079a7499900228905b117236ae6423f603bf4f2080a78fd218e24f9dbb8ae"
Sep 16 19:19:35.166030 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:35.166014    2586 scope.go:117] "RemoveContainer" containerID="8bebad5d1fc6168579b243e13fc891de86ec5fcb18b2adfe6ff43299c37f67fb"
Sep 16 19:19:35.180901 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:35.180879    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-gwfg9"]
Sep 16 19:19:35.185950 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:35.185928    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-gwfg9"]
Sep 16 19:19:37.097192 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:37.097157    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da6c72d6-5b8f-43a0-a917-9956487c11a5" path="/var/lib/kubelet/pods/da6c72d6-5b8f-43a0-a917-9956487c11a5/volumes"
Sep 16 19:19:37.989478 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:37.989439    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-8l7nw" podUID="3fa70ab6-6d52-4039-b747-30046038d673" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 16 19:19:38.994125 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:38.994079    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-qpcx4" podUID="9f998575-a92d-4cec-9fad-eafd7e481461" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 16 19:19:47.989960 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:47.989915    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-8l7nw" podUID="3fa70ab6-6d52-4039-b747-30046038d673" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 16 19:19:48.994186 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:48.994143    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-qpcx4" podUID="9f998575-a92d-4cec-9fad-eafd7e481461" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 16 19:19:57.989735 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:57.989692    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-8l7nw" podUID="3fa70ab6-6d52-4039-b747-30046038d673" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 16 19:19:58.994761 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:19:58.994722    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-qpcx4" podUID="9f998575-a92d-4cec-9fad-eafd7e481461" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 16 19:20:07.989715 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:07.989672    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-8l7nw" podUID="3fa70ab6-6d52-4039-b747-30046038d673" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 16 19:20:08.994380 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:08.994339    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-qpcx4" podUID="9f998575-a92d-4cec-9fad-eafd7e481461" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 16 19:20:11.087620 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:11.087567    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-qpcx4" podUID="9f998575-a92d-4cec-9fad-eafd7e481461" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 16 19:20:17.990251 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:17.990208    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-8l7nw"
Sep 16 19:20:21.091999 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:21.091974    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-qpcx4"
Sep 16 19:20:21.386045 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:21.386009    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-8l7nw"]
Sep 16 19:20:21.386287 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:21.386216    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-8l7nw" podUID="3fa70ab6-6d52-4039-b747-30046038d673" containerName="kserve-container" containerID="cri-o://a902db437ffb68e9d2c44ddaeb7e53482357887484cc6dea5e5b767ed41eba9e" gracePeriod=30
Sep 16 19:20:21.386287 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:21.386273    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-8l7nw" podUID="3fa70ab6-6d52-4039-b747-30046038d673" containerName="kube-rbac-proxy" containerID="cri-o://138b169163f2d3301761f19f4e197ba08db1c28c9c0cd03d25ac085e58d64a79" gracePeriod=30
Sep 16 19:20:22.382900 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:22.382859    2586 generic.go:356] "Generic (PLEG): container finished" podID="3fa70ab6-6d52-4039-b747-30046038d673" containerID="138b169163f2d3301761f19f4e197ba08db1c28c9c0cd03d25ac085e58d64a79" exitCode=2
Sep 16 19:20:22.383290 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:22.382927    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-8l7nw" event={"ID":"3fa70ab6-6d52-4039-b747-30046038d673","Type":"ContainerDied","Data":"138b169163f2d3301761f19f4e197ba08db1c28c9c0cd03d25ac085e58d64a79"}
Sep 16 19:20:22.984868 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:22.984820    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-8l7nw" podUID="3fa70ab6-6d52-4039-b747-30046038d673" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.57:8643/healthz\": dial tcp 10.134.0.57:8643: connect: connection refused"
Sep 16 19:20:25.984250 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:25.984195    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-qpcx4"]
Sep 16 19:20:25.984679 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:25.984467    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-qpcx4" podUID="9f998575-a92d-4cec-9fad-eafd7e481461" containerName="kube-rbac-proxy" containerID="cri-o://c5916b17e13ba4d48ab1717efa40b4ea4a5d94a91f08c96340b6b67c131d469a" gracePeriod=30
Sep 16 19:20:25.984679 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:25.984457    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-qpcx4" podUID="9f998575-a92d-4cec-9fad-eafd7e481461" containerName="kserve-container" containerID="cri-o://4f81fb9e9af1566ed546c5465609206a44a256ec032bb34b19a5cc95011d12b0" gracePeriod=30
Sep 16 19:20:26.136702 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:26.136679    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-8l7nw"
Sep 16 19:20:26.197881 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:26.197796    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3fa70ab6-6d52-4039-b747-30046038d673-kube-rbac-proxy-sar-config\") pod \"3fa70ab6-6d52-4039-b747-30046038d673\" (UID: \"3fa70ab6-6d52-4039-b747-30046038d673\") "
Sep 16 19:20:26.197881 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:26.197838    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3fa70ab6-6d52-4039-b747-30046038d673-cabundle-cert\") pod \"3fa70ab6-6d52-4039-b747-30046038d673\" (UID: \"3fa70ab6-6d52-4039-b747-30046038d673\") "
Sep 16 19:20:26.197881 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:26.197862    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fa70ab6-6d52-4039-b747-30046038d673-proxy-tls\") pod \"3fa70ab6-6d52-4039-b747-30046038d673\" (UID: \"3fa70ab6-6d52-4039-b747-30046038d673\") "
Sep 16 19:20:26.198150 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:26.197898    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mcbdq\" (UniqueName: \"kubernetes.io/projected/3fa70ab6-6d52-4039-b747-30046038d673-kube-api-access-mcbdq\") pod \"3fa70ab6-6d52-4039-b747-30046038d673\" (UID: \"3fa70ab6-6d52-4039-b747-30046038d673\") "
Sep 16 19:20:26.198150 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:26.197925    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fa70ab6-6d52-4039-b747-30046038d673-kserve-provision-location\") pod \"3fa70ab6-6d52-4039-b747-30046038d673\" (UID: \"3fa70ab6-6d52-4039-b747-30046038d673\") "
Sep 16 19:20:26.198340 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:26.198316    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fa70ab6-6d52-4039-b747-30046038d673-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3fa70ab6-6d52-4039-b747-30046038d673" (UID: "3fa70ab6-6d52-4039-b747-30046038d673"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:20:26.198407 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:26.198355    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fa70ab6-6d52-4039-b747-30046038d673-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3fa70ab6-6d52-4039-b747-30046038d673" (UID: "3fa70ab6-6d52-4039-b747-30046038d673"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:20:26.198465 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:26.198420    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fa70ab6-6d52-4039-b747-30046038d673-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3fa70ab6-6d52-4039-b747-30046038d673" (UID: "3fa70ab6-6d52-4039-b747-30046038d673"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:20:26.200071 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:26.200049    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fa70ab6-6d52-4039-b747-30046038d673-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3fa70ab6-6d52-4039-b747-30046038d673" (UID: "3fa70ab6-6d52-4039-b747-30046038d673"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:20:26.200173 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:26.200113    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fa70ab6-6d52-4039-b747-30046038d673-kube-api-access-mcbdq" (OuterVolumeSpecName: "kube-api-access-mcbdq") pod "3fa70ab6-6d52-4039-b747-30046038d673" (UID: "3fa70ab6-6d52-4039-b747-30046038d673"). InnerVolumeSpecName "kube-api-access-mcbdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:20:26.299136 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:26.299103    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcbdq\" (UniqueName: \"kubernetes.io/projected/3fa70ab6-6d52-4039-b747-30046038d673-kube-api-access-mcbdq\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:20:26.299136 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:26.299133    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fa70ab6-6d52-4039-b747-30046038d673-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:20:26.299136 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:26.299144    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3fa70ab6-6d52-4039-b747-30046038d673-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:20:26.299396 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:26.299154    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3fa70ab6-6d52-4039-b747-30046038d673-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:20:26.299396 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:26.299163    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fa70ab6-6d52-4039-b747-30046038d673-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:20:26.403296 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:26.403259    2586 generic.go:356] "Generic (PLEG): container finished" podID="9f998575-a92d-4cec-9fad-eafd7e481461" containerID="c5916b17e13ba4d48ab1717efa40b4ea4a5d94a91f08c96340b6b67c131d469a" exitCode=2
Sep 16 19:20:26.403471 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:26.403259    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-qpcx4" event={"ID":"9f998575-a92d-4cec-9fad-eafd7e481461","Type":"ContainerDied","Data":"c5916b17e13ba4d48ab1717efa40b4ea4a5d94a91f08c96340b6b67c131d469a"}
Sep 16 19:20:26.405192 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:26.405166    2586 generic.go:356] "Generic (PLEG): container finished" podID="3fa70ab6-6d52-4039-b747-30046038d673" containerID="a902db437ffb68e9d2c44ddaeb7e53482357887484cc6dea5e5b767ed41eba9e" exitCode=0
Sep 16 19:20:26.405327 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:26.405201    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-8l7nw" event={"ID":"3fa70ab6-6d52-4039-b747-30046038d673","Type":"ContainerDied","Data":"a902db437ffb68e9d2c44ddaeb7e53482357887484cc6dea5e5b767ed41eba9e"}
Sep 16 19:20:26.405327 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:26.405257    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-8l7nw"
Sep 16 19:20:26.405327 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:26.405271    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-8l7nw" event={"ID":"3fa70ab6-6d52-4039-b747-30046038d673","Type":"ContainerDied","Data":"fe27982b57ad8ae9426af824c7c813171622e316c12e286a75ace75163568047"}
Sep 16 19:20:26.405327 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:26.405308    2586 scope.go:117] "RemoveContainer" containerID="138b169163f2d3301761f19f4e197ba08db1c28c9c0cd03d25ac085e58d64a79"
Sep 16 19:20:26.414157 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:26.414135    2586 scope.go:117] "RemoveContainer" containerID="a902db437ffb68e9d2c44ddaeb7e53482357887484cc6dea5e5b767ed41eba9e"
Sep 16 19:20:26.422189 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:26.422166    2586 scope.go:117] "RemoveContainer" containerID="2d8425604cd35df637bb8015d1773c7a7ff7cd6b825ead916f0a0c73c51d1999"
Sep 16 19:20:26.430728 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:26.430707    2586 scope.go:117] "RemoveContainer" containerID="138b169163f2d3301761f19f4e197ba08db1c28c9c0cd03d25ac085e58d64a79"
Sep 16 19:20:26.431023 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:20:26.431003    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"138b169163f2d3301761f19f4e197ba08db1c28c9c0cd03d25ac085e58d64a79\": container with ID starting with 138b169163f2d3301761f19f4e197ba08db1c28c9c0cd03d25ac085e58d64a79 not found: ID does not exist" containerID="138b169163f2d3301761f19f4e197ba08db1c28c9c0cd03d25ac085e58d64a79"
Sep 16 19:20:26.431088 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:26.431037    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"138b169163f2d3301761f19f4e197ba08db1c28c9c0cd03d25ac085e58d64a79"} err="failed to get container status \"138b169163f2d3301761f19f4e197ba08db1c28c9c0cd03d25ac085e58d64a79\": rpc error: code = NotFound desc = could not find container \"138b169163f2d3301761f19f4e197ba08db1c28c9c0cd03d25ac085e58d64a79\": container with ID starting with 138b169163f2d3301761f19f4e197ba08db1c28c9c0cd03d25ac085e58d64a79 not found: ID does not exist"
Sep 16 19:20:26.431088 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:26.431054    2586 scope.go:117] "RemoveContainer" containerID="a902db437ffb68e9d2c44ddaeb7e53482357887484cc6dea5e5b767ed41eba9e"
Sep 16 19:20:26.431449 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:20:26.431425    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a902db437ffb68e9d2c44ddaeb7e53482357887484cc6dea5e5b767ed41eba9e\": container with ID starting with a902db437ffb68e9d2c44ddaeb7e53482357887484cc6dea5e5b767ed41eba9e not found: ID does not exist" containerID="a902db437ffb68e9d2c44ddaeb7e53482357887484cc6dea5e5b767ed41eba9e"
Sep 16 19:20:26.431524 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:26.431454    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a902db437ffb68e9d2c44ddaeb7e53482357887484cc6dea5e5b767ed41eba9e"} err="failed to get container status \"a902db437ffb68e9d2c44ddaeb7e53482357887484cc6dea5e5b767ed41eba9e\": rpc error: code = NotFound desc = could not find container \"a902db437ffb68e9d2c44ddaeb7e53482357887484cc6dea5e5b767ed41eba9e\": container with ID starting with a902db437ffb68e9d2c44ddaeb7e53482357887484cc6dea5e5b767ed41eba9e not found: ID does not exist"
Sep 16 19:20:26.431524 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:26.431467    2586 scope.go:117] "RemoveContainer" containerID="2d8425604cd35df637bb8015d1773c7a7ff7cd6b825ead916f0a0c73c51d1999"
Sep 16 19:20:26.431732 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:20:26.431716    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d8425604cd35df637bb8015d1773c7a7ff7cd6b825ead916f0a0c73c51d1999\": container with ID starting with 2d8425604cd35df637bb8015d1773c7a7ff7cd6b825ead916f0a0c73c51d1999 not found: ID does not exist" containerID="2d8425604cd35df637bb8015d1773c7a7ff7cd6b825ead916f0a0c73c51d1999"
Sep 16 19:20:26.431789 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:26.431737    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d8425604cd35df637bb8015d1773c7a7ff7cd6b825ead916f0a0c73c51d1999"} err="failed to get container status \"2d8425604cd35df637bb8015d1773c7a7ff7cd6b825ead916f0a0c73c51d1999\": rpc error: code = NotFound desc = could not find container \"2d8425604cd35df637bb8015d1773c7a7ff7cd6b825ead916f0a0c73c51d1999\": container with ID starting with 2d8425604cd35df637bb8015d1773c7a7ff7cd6b825ead916f0a0c73c51d1999 not found: ID does not exist"
Sep 16 19:20:26.438492 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:26.438467    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-8l7nw"]
Sep 16 19:20:26.445677 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:26.445650    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-8l7nw"]
Sep 16 19:20:27.092661 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:27.092632    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fa70ab6-6d52-4039-b747-30046038d673" path="/var/lib/kubelet/pods/3fa70ab6-6d52-4039-b747-30046038d673/volumes"
Sep 16 19:20:28.989783 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:28.989735    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-qpcx4" podUID="9f998575-a92d-4cec-9fad-eafd7e481461" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.56:8643/healthz\": dial tcp 10.134.0.56:8643: connect: connection refused"
Sep 16 19:20:30.838419 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:30.838398    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-qpcx4"
Sep 16 19:20:30.941044 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:30.940961    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f998575-a92d-4cec-9fad-eafd7e481461-proxy-tls\") pod \"9f998575-a92d-4cec-9fad-eafd7e481461\" (UID: \"9f998575-a92d-4cec-9fad-eafd7e481461\") "
Sep 16 19:20:30.941044 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:30.941006    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f998575-a92d-4cec-9fad-eafd7e481461-cabundle-cert\") pod \"9f998575-a92d-4cec-9fad-eafd7e481461\" (UID: \"9f998575-a92d-4cec-9fad-eafd7e481461\") "
Sep 16 19:20:30.941044 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:30.941038    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f998575-a92d-4cec-9fad-eafd7e481461-kube-rbac-proxy-sar-config\") pod \"9f998575-a92d-4cec-9fad-eafd7e481461\" (UID: \"9f998575-a92d-4cec-9fad-eafd7e481461\") "
Sep 16 19:20:30.941359 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:30.941102    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f998575-a92d-4cec-9fad-eafd7e481461-kserve-provision-location\") pod \"9f998575-a92d-4cec-9fad-eafd7e481461\" (UID: \"9f998575-a92d-4cec-9fad-eafd7e481461\") "
Sep 16 19:20:30.941359 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:30.941171    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ckm4c\" (UniqueName: \"kubernetes.io/projected/9f998575-a92d-4cec-9fad-eafd7e481461-kube-api-access-ckm4c\") pod \"9f998575-a92d-4cec-9fad-eafd7e481461\" (UID: \"9f998575-a92d-4cec-9fad-eafd7e481461\") "
Sep 16 19:20:30.941468 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:30.941430    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f998575-a92d-4cec-9fad-eafd7e481461-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9f998575-a92d-4cec-9fad-eafd7e481461" (UID: "9f998575-a92d-4cec-9fad-eafd7e481461"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:20:30.941523 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:30.941467    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f998575-a92d-4cec-9fad-eafd7e481461-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9f998575-a92d-4cec-9fad-eafd7e481461" (UID: "9f998575-a92d-4cec-9fad-eafd7e481461"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:20:30.941523 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:30.941473    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f998575-a92d-4cec-9fad-eafd7e481461-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9f998575-a92d-4cec-9fad-eafd7e481461" (UID: "9f998575-a92d-4cec-9fad-eafd7e481461"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:20:30.943120 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:30.943101    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f998575-a92d-4cec-9fad-eafd7e481461-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9f998575-a92d-4cec-9fad-eafd7e481461" (UID: "9f998575-a92d-4cec-9fad-eafd7e481461"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:20:30.943192 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:30.943136    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f998575-a92d-4cec-9fad-eafd7e481461-kube-api-access-ckm4c" (OuterVolumeSpecName: "kube-api-access-ckm4c") pod "9f998575-a92d-4cec-9fad-eafd7e481461" (UID: "9f998575-a92d-4cec-9fad-eafd7e481461"). InnerVolumeSpecName "kube-api-access-ckm4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:20:31.042756 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:31.042716    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckm4c\" (UniqueName: \"kubernetes.io/projected/9f998575-a92d-4cec-9fad-eafd7e481461-kube-api-access-ckm4c\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:20:31.042756 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:31.042748    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f998575-a92d-4cec-9fad-eafd7e481461-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:20:31.042756 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:31.042759    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f998575-a92d-4cec-9fad-eafd7e481461-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:20:31.042988 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:31.042768    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f998575-a92d-4cec-9fad-eafd7e481461-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:20:31.042988 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:31.042777    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f998575-a92d-4cec-9fad-eafd7e481461-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:20:31.439580 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:31.439539    2586 generic.go:356] "Generic (PLEG): container finished" podID="9f998575-a92d-4cec-9fad-eafd7e481461" containerID="4f81fb9e9af1566ed546c5465609206a44a256ec032bb34b19a5cc95011d12b0" exitCode=0
Sep 16 19:20:31.439741 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:31.439594    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-qpcx4"
Sep 16 19:20:31.439741 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:31.439632    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-qpcx4" event={"ID":"9f998575-a92d-4cec-9fad-eafd7e481461","Type":"ContainerDied","Data":"4f81fb9e9af1566ed546c5465609206a44a256ec032bb34b19a5cc95011d12b0"}
Sep 16 19:20:31.439741 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:31.439699    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-qpcx4" event={"ID":"9f998575-a92d-4cec-9fad-eafd7e481461","Type":"ContainerDied","Data":"86375702efe21720cfd1a8d099a6d3990813dcde296f251b7c358a8deb148482"}
Sep 16 19:20:31.439741 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:31.439728    2586 scope.go:117] "RemoveContainer" containerID="c5916b17e13ba4d48ab1717efa40b4ea4a5d94a91f08c96340b6b67c131d469a"
Sep 16 19:20:31.447783 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:31.447765    2586 scope.go:117] "RemoveContainer" containerID="4f81fb9e9af1566ed546c5465609206a44a256ec032bb34b19a5cc95011d12b0"
Sep 16 19:20:31.455491 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:31.455475    2586 scope.go:117] "RemoveContainer" containerID="21e82f3b14563f2a41a9a2795ed27164db9a4de8cba3dcc2df4b6c2505f8902c"
Sep 16 19:20:31.462534 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:31.462515    2586 scope.go:117] "RemoveContainer" containerID="c5916b17e13ba4d48ab1717efa40b4ea4a5d94a91f08c96340b6b67c131d469a"
Sep 16 19:20:31.462798 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:20:31.462780    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5916b17e13ba4d48ab1717efa40b4ea4a5d94a91f08c96340b6b67c131d469a\": container with ID starting with c5916b17e13ba4d48ab1717efa40b4ea4a5d94a91f08c96340b6b67c131d469a not found: ID does not exist" containerID="c5916b17e13ba4d48ab1717efa40b4ea4a5d94a91f08c96340b6b67c131d469a"
Sep 16 19:20:31.462840 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:31.462807    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5916b17e13ba4d48ab1717efa40b4ea4a5d94a91f08c96340b6b67c131d469a"} err="failed to get container status \"c5916b17e13ba4d48ab1717efa40b4ea4a5d94a91f08c96340b6b67c131d469a\": rpc error: code = NotFound desc = could not find container \"c5916b17e13ba4d48ab1717efa40b4ea4a5d94a91f08c96340b6b67c131d469a\": container with ID starting with c5916b17e13ba4d48ab1717efa40b4ea4a5d94a91f08c96340b6b67c131d469a not found: ID does not exist"
Sep 16 19:20:31.462840 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:31.462824    2586 scope.go:117] "RemoveContainer" containerID="4f81fb9e9af1566ed546c5465609206a44a256ec032bb34b19a5cc95011d12b0"
Sep 16 19:20:31.463096 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:20:31.463077    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f81fb9e9af1566ed546c5465609206a44a256ec032bb34b19a5cc95011d12b0\": container with ID starting with 4f81fb9e9af1566ed546c5465609206a44a256ec032bb34b19a5cc95011d12b0 not found: ID does not exist" containerID="4f81fb9e9af1566ed546c5465609206a44a256ec032bb34b19a5cc95011d12b0"
Sep 16 19:20:31.463151 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:31.463101    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f81fb9e9af1566ed546c5465609206a44a256ec032bb34b19a5cc95011d12b0"} err="failed to get container status \"4f81fb9e9af1566ed546c5465609206a44a256ec032bb34b19a5cc95011d12b0\": rpc error: code = NotFound desc = could not find container \"4f81fb9e9af1566ed546c5465609206a44a256ec032bb34b19a5cc95011d12b0\": container with ID starting with 4f81fb9e9af1566ed546c5465609206a44a256ec032bb34b19a5cc95011d12b0 not found: ID does not exist"
Sep 16 19:20:31.463151 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:31.463115    2586 scope.go:117] "RemoveContainer" containerID="21e82f3b14563f2a41a9a2795ed27164db9a4de8cba3dcc2df4b6c2505f8902c"
Sep 16 19:20:31.463424 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:20:31.463409    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21e82f3b14563f2a41a9a2795ed27164db9a4de8cba3dcc2df4b6c2505f8902c\": container with ID starting with 21e82f3b14563f2a41a9a2795ed27164db9a4de8cba3dcc2df4b6c2505f8902c not found: ID does not exist" containerID="21e82f3b14563f2a41a9a2795ed27164db9a4de8cba3dcc2df4b6c2505f8902c"
Sep 16 19:20:31.463472 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:31.463427    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21e82f3b14563f2a41a9a2795ed27164db9a4de8cba3dcc2df4b6c2505f8902c"} err="failed to get container status \"21e82f3b14563f2a41a9a2795ed27164db9a4de8cba3dcc2df4b6c2505f8902c\": rpc error: code = NotFound desc = could not find container \"21e82f3b14563f2a41a9a2795ed27164db9a4de8cba3dcc2df4b6c2505f8902c\": container with ID starting with 21e82f3b14563f2a41a9a2795ed27164db9a4de8cba3dcc2df4b6c2505f8902c not found: ID does not exist"
Sep 16 19:20:31.466646 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:31.466624    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-qpcx4"]
Sep 16 19:20:31.477837 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:31.477815    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-854695c447-qpcx4"]
Sep 16 19:20:33.092457 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:33.092426    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f998575-a92d-4cec-9fad-eafd7e481461" path="/var/lib/kubelet/pods/9f998575-a92d-4cec-9fad-eafd7e481461/volumes"
Sep 16 19:20:36.418695 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:36.418658    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-djvhk"]
Sep 16 19:20:36.419063 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:36.418965    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da6c72d6-5b8f-43a0-a917-9956487c11a5" containerName="kserve-container"
Sep 16 19:20:36.419063 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:36.418975    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="da6c72d6-5b8f-43a0-a917-9956487c11a5" containerName="kserve-container"
Sep 16 19:20:36.419063 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:36.418985    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fa70ab6-6d52-4039-b747-30046038d673" containerName="kserve-container"
Sep 16 19:20:36.419063 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:36.418990    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fa70ab6-6d52-4039-b747-30046038d673" containerName="kserve-container"
Sep 16 19:20:36.419063 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:36.419002    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da6c72d6-5b8f-43a0-a917-9956487c11a5" containerName="storage-initializer"
Sep 16 19:20:36.419063 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:36.419008    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="da6c72d6-5b8f-43a0-a917-9956487c11a5" containerName="storage-initializer"
Sep 16 19:20:36.419063 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:36.419014    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da6c72d6-5b8f-43a0-a917-9956487c11a5" containerName="kube-rbac-proxy"
Sep 16 19:20:36.419063 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:36.419019    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="da6c72d6-5b8f-43a0-a917-9956487c11a5" containerName="kube-rbac-proxy"
Sep 16 19:20:36.419063 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:36.419025    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fa70ab6-6d52-4039-b747-30046038d673" containerName="kube-rbac-proxy"
Sep 16 19:20:36.419063 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:36.419031    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fa70ab6-6d52-4039-b747-30046038d673" containerName="kube-rbac-proxy"
Sep 16 19:20:36.419063 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:36.419044    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f998575-a92d-4cec-9fad-eafd7e481461" containerName="kube-rbac-proxy"
Sep 16 19:20:36.419063 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:36.419049    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f998575-a92d-4cec-9fad-eafd7e481461" containerName="kube-rbac-proxy"
Sep 16 19:20:36.419063 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:36.419055    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fa70ab6-6d52-4039-b747-30046038d673" containerName="storage-initializer"
Sep 16 19:20:36.419063 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:36.419061    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fa70ab6-6d52-4039-b747-30046038d673" containerName="storage-initializer"
Sep 16 19:20:36.419063 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:36.419066    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f998575-a92d-4cec-9fad-eafd7e481461" containerName="kserve-container"
Sep 16 19:20:36.419063 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:36.419071    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f998575-a92d-4cec-9fad-eafd7e481461" containerName="kserve-container"
Sep 16 19:20:36.419597 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:36.419080    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f998575-a92d-4cec-9fad-eafd7e481461" containerName="storage-initializer"
Sep 16 19:20:36.419597 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:36.419085    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f998575-a92d-4cec-9fad-eafd7e481461" containerName="storage-initializer"
Sep 16 19:20:36.419597 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:36.419151    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f998575-a92d-4cec-9fad-eafd7e481461" containerName="kube-rbac-proxy"
Sep 16 19:20:36.419597 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:36.419157    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fa70ab6-6d52-4039-b747-30046038d673" containerName="kube-rbac-proxy"
Sep 16 19:20:36.419597 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:36.419163    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f998575-a92d-4cec-9fad-eafd7e481461" containerName="kserve-container"
Sep 16 19:20:36.419597 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:36.419169    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="da6c72d6-5b8f-43a0-a917-9956487c11a5" containerName="kserve-container"
Sep 16 19:20:36.419597 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:36.419176    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fa70ab6-6d52-4039-b747-30046038d673" containerName="kserve-container"
Sep 16 19:20:36.419597 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:36.419183    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="da6c72d6-5b8f-43a0-a917-9956487c11a5" containerName="kube-rbac-proxy"
Sep 16 19:20:36.422570 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:36.422549    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-djvhk"
Sep 16 19:20:36.430492 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:36.430463    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:20:36.430626 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:36.430465    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:20:36.430626 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:36.430505    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:20:36.430626 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:36.430518    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:20:36.430626 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:36.430475    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\""
Sep 16 19:20:36.431584 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:36.431566    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-pv9ct\""
Sep 16 19:20:36.433454 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:36.433426    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
Sep 16 19:20:36.440964 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:36.440942    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-djvhk"]
Sep 16 19:20:36.489131 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:36.489093    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d382551-8624-42d3-9fda-564392d8a193-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-djvhk\" (UID: \"6d382551-8624-42d3-9fda-564392d8a193\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-djvhk"
Sep 16 19:20:36.489301 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:36.489144    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d382551-8624-42d3-9fda-564392d8a193-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-djvhk\" (UID: \"6d382551-8624-42d3-9fda-564392d8a193\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-djvhk"
Sep 16 19:20:36.489301 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:36.489209    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8jj2\" (UniqueName: \"kubernetes.io/projected/6d382551-8624-42d3-9fda-564392d8a193-kube-api-access-s8jj2\") pod \"isvc-lightgbm-predictor-54f85ddf88-djvhk\" (UID: \"6d382551-8624-42d3-9fda-564392d8a193\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-djvhk"
Sep 16 19:20:36.489301 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:36.489289    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d382551-8624-42d3-9fda-564392d8a193-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-djvhk\" (UID: \"6d382551-8624-42d3-9fda-564392d8a193\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-djvhk"
Sep 16 19:20:36.489418 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:36.489379    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d382551-8624-42d3-9fda-564392d8a193-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-djvhk\" (UID: \"6d382551-8624-42d3-9fda-564392d8a193\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-djvhk"
Sep 16 19:20:36.590644 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:36.590605    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d382551-8624-42d3-9fda-564392d8a193-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-djvhk\" (UID: \"6d382551-8624-42d3-9fda-564392d8a193\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-djvhk"
Sep 16 19:20:36.590828 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:36.590660    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d382551-8624-42d3-9fda-564392d8a193-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-djvhk\" (UID: \"6d382551-8624-42d3-9fda-564392d8a193\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-djvhk"
Sep 16 19:20:36.590828 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:36.590690    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d382551-8624-42d3-9fda-564392d8a193-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-djvhk\" (UID: \"6d382551-8624-42d3-9fda-564392d8a193\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-djvhk"
Sep 16 19:20:36.590828 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:36.590722    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8jj2\" (UniqueName: \"kubernetes.io/projected/6d382551-8624-42d3-9fda-564392d8a193-kube-api-access-s8jj2\") pod \"isvc-lightgbm-predictor-54f85ddf88-djvhk\" (UID: \"6d382551-8624-42d3-9fda-564392d8a193\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-djvhk"
Sep 16 19:20:36.590828 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:36.590750    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d382551-8624-42d3-9fda-564392d8a193-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-djvhk\" (UID: \"6d382551-8624-42d3-9fda-564392d8a193\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-djvhk"
Sep 16 19:20:36.591124 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:36.591097    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d382551-8624-42d3-9fda-564392d8a193-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-djvhk\" (UID: \"6d382551-8624-42d3-9fda-564392d8a193\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-djvhk"
Sep 16 19:20:36.591347 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:36.591327    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d382551-8624-42d3-9fda-564392d8a193-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-djvhk\" (UID: \"6d382551-8624-42d3-9fda-564392d8a193\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-djvhk"
Sep 16 19:20:36.591459 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:36.591439    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d382551-8624-42d3-9fda-564392d8a193-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-djvhk\" (UID: \"6d382551-8624-42d3-9fda-564392d8a193\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-djvhk"
Sep 16 19:20:36.593167 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:36.593146    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d382551-8624-42d3-9fda-564392d8a193-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-djvhk\" (UID: \"6d382551-8624-42d3-9fda-564392d8a193\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-djvhk"
Sep 16 19:20:36.604448 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:36.604428    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8jj2\" (UniqueName: \"kubernetes.io/projected/6d382551-8624-42d3-9fda-564392d8a193-kube-api-access-s8jj2\") pod \"isvc-lightgbm-predictor-54f85ddf88-djvhk\" (UID: \"6d382551-8624-42d3-9fda-564392d8a193\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-djvhk"
Sep 16 19:20:36.732782 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:36.732699    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-djvhk"
Sep 16 19:20:37.020589 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:37.020561    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-djvhk"]
Sep 16 19:20:37.022988 ip-10-0-129-190 kubenswrapper[2586]: W0916 19:20:37.022960    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d382551_8624_42d3_9fda_564392d8a193.slice/crio-67fca19a5587395202e73e30c894bcba9287eb7152df35173d9846f445ae4500 WatchSource:0}: Error finding container 67fca19a5587395202e73e30c894bcba9287eb7152df35173d9846f445ae4500: Status 404 returned error can't find the container with id 67fca19a5587395202e73e30c894bcba9287eb7152df35173d9846f445ae4500
Sep 16 19:20:37.472970 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:37.472935    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-djvhk" event={"ID":"6d382551-8624-42d3-9fda-564392d8a193","Type":"ContainerStarted","Data":"e0cecb1a4f50668d06f4268b24ac7eebe088c483922fde0057c59f15d8c84260"}
Sep 16 19:20:37.473473 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:37.472989    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-djvhk" event={"ID":"6d382551-8624-42d3-9fda-564392d8a193","Type":"ContainerStarted","Data":"67fca19a5587395202e73e30c894bcba9287eb7152df35173d9846f445ae4500"}
Sep 16 19:20:41.491025 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:41.490992    2586 generic.go:356] "Generic (PLEG): container finished" podID="6d382551-8624-42d3-9fda-564392d8a193" containerID="e0cecb1a4f50668d06f4268b24ac7eebe088c483922fde0057c59f15d8c84260" exitCode=0
Sep 16 19:20:41.491530 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:41.491053    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-djvhk" event={"ID":"6d382551-8624-42d3-9fda-564392d8a193","Type":"ContainerDied","Data":"e0cecb1a4f50668d06f4268b24ac7eebe088c483922fde0057c59f15d8c84260"}
Sep 16 19:20:41.492332 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:41.492313    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:20:48.528722 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:48.528684    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-djvhk" event={"ID":"6d382551-8624-42d3-9fda-564392d8a193","Type":"ContainerStarted","Data":"5c093166d9ff3d25a83b4012ef7a72c86a50542cf4320789b96582ab4cbdc94c"}
Sep 16 19:20:48.529107 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:48.528754    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-djvhk" event={"ID":"6d382551-8624-42d3-9fda-564392d8a193","Type":"ContainerStarted","Data":"c5006b7efbc8fcc5ac3487190184bc7ba12adb7411cd2278ff12479dc465fd6d"}
Sep 16 19:20:48.529107 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:48.528945    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-djvhk"
Sep 16 19:20:49.533623 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:49.533589    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-djvhk"
Sep 16 19:20:49.535002 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:49.534972    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-djvhk" podUID="6d382551-8624-42d3-9fda-564392d8a193" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 16 19:20:50.537634 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:50.537592    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-djvhk" podUID="6d382551-8624-42d3-9fda-564392d8a193" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 16 19:20:55.542155 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:55.542126    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-djvhk"
Sep 16 19:20:55.542809 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:55.542777    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-djvhk" podUID="6d382551-8624-42d3-9fda-564392d8a193" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 16 19:20:55.582641 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:20:55.582588    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-djvhk" podStartSLOduration=9.538041985 podStartE2EDuration="19.582574026s" totalImagesPullingTime="6.268700581s" totalInitContainerRuntime="3.77583146s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:20:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:20:48.56055049 +0000 UTC m=+2424.004412065" watchObservedRunningTime="2026-09-16 19:20:55.582574026 +0000 UTC m=+2431.026435602"
Sep 16 19:21:05.543483 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:21:05.543446    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-djvhk" podUID="6d382551-8624-42d3-9fda-564392d8a193" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 16 19:21:15.542990 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:21:15.542946    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-djvhk" podUID="6d382551-8624-42d3-9fda-564392d8a193" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 16 19:21:25.464923 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:21:25.464881    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-9gxq2"]
Sep 16 19:21:25.473858 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:21:25.473831    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-9gxq2"
Sep 16 19:21:25.480958 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:21:25.480932    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
Sep 16 19:21:25.481074 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:21:25.480998    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
Sep 16 19:21:25.485209 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:21:25.485188    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-9gxq2"]
Sep 16 19:21:25.519648 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:21:25.519613    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22e9c4d2-44a6-4d84-a9e8-5547802804bb-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-9gxq2\" (UID: \"22e9c4d2-44a6-4d84-a9e8-5547802804bb\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-9gxq2"
Sep 16 19:21:25.519648 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:21:25.519648    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22e9c4d2-44a6-4d84-a9e8-5547802804bb-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-9gxq2\" (UID: \"22e9c4d2-44a6-4d84-a9e8-5547802804bb\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-9gxq2"
Sep 16 19:21:25.519910 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:21:25.519702    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22e9c4d2-44a6-4d84-a9e8-5547802804bb-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-9gxq2\" (UID: \"22e9c4d2-44a6-4d84-a9e8-5547802804bb\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-9gxq2"
Sep 16 19:21:25.519910 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:21:25.519731    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8k28\" (UniqueName: \"kubernetes.io/projected/22e9c4d2-44a6-4d84-a9e8-5547802804bb-kube-api-access-b8k28\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-9gxq2\" (UID: \"22e9c4d2-44a6-4d84-a9e8-5547802804bb\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-9gxq2"
Sep 16 19:21:25.519910 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:21:25.519845    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22e9c4d2-44a6-4d84-a9e8-5547802804bb-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-9gxq2\" (UID: \"22e9c4d2-44a6-4d84-a9e8-5547802804bb\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-9gxq2"
Sep 16 19:21:25.543278 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:21:25.543217    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-djvhk" podUID="6d382551-8624-42d3-9fda-564392d8a193" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 16 19:21:25.620375 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:21:25.620333    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22e9c4d2-44a6-4d84-a9e8-5547802804bb-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-9gxq2\" (UID: \"22e9c4d2-44a6-4d84-a9e8-5547802804bb\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-9gxq2"
Sep 16 19:21:25.620545 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:21:25.620395    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22e9c4d2-44a6-4d84-a9e8-5547802804bb-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-9gxq2\" (UID: \"22e9c4d2-44a6-4d84-a9e8-5547802804bb\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-9gxq2"
Sep 16 19:21:25.620545 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:21:25.620413    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22e9c4d2-44a6-4d84-a9e8-5547802804bb-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-9gxq2\" (UID: \"22e9c4d2-44a6-4d84-a9e8-5547802804bb\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-9gxq2"
Sep 16 19:21:25.620545 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:21:25.620455    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22e9c4d2-44a6-4d84-a9e8-5547802804bb-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-9gxq2\" (UID: \"22e9c4d2-44a6-4d84-a9e8-5547802804bb\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-9gxq2"
Sep 16 19:21:25.620545 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:21:25.620490    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8k28\" (UniqueName: \"kubernetes.io/projected/22e9c4d2-44a6-4d84-a9e8-5547802804bb-kube-api-access-b8k28\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-9gxq2\" (UID: \"22e9c4d2-44a6-4d84-a9e8-5547802804bb\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-9gxq2"
Sep 16 19:21:25.620980 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:21:25.620942    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22e9c4d2-44a6-4d84-a9e8-5547802804bb-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-9gxq2\" (UID: \"22e9c4d2-44a6-4d84-a9e8-5547802804bb\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-9gxq2"
Sep 16 19:21:25.621145 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:21:25.621124    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22e9c4d2-44a6-4d84-a9e8-5547802804bb-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-9gxq2\" (UID: \"22e9c4d2-44a6-4d84-a9e8-5547802804bb\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-9gxq2"
Sep 16 19:21:25.621196 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:21:25.621184    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22e9c4d2-44a6-4d84-a9e8-5547802804bb-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-9gxq2\" (UID: \"22e9c4d2-44a6-4d84-a9e8-5547802804bb\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-9gxq2"
Sep 16 19:21:25.622814 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:21:25.622796    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22e9c4d2-44a6-4d84-a9e8-5547802804bb-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-9gxq2\" (UID: \"22e9c4d2-44a6-4d84-a9e8-5547802804bb\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-9gxq2"
Sep 16 19:21:25.633714 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:21:25.633685    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8k28\" (UniqueName: \"kubernetes.io/projected/22e9c4d2-44a6-4d84-a9e8-5547802804bb-kube-api-access-b8k28\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-9gxq2\" (UID: \"22e9c4d2-44a6-4d84-a9e8-5547802804bb\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-9gxq2"
Sep 16 19:21:25.784393 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:21:25.784306    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-9gxq2"
Sep 16 19:21:26.076037 ip-10-0-129-190 kubenswrapper[2586]: W0916 19:21:26.075997    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22e9c4d2_44a6_4d84_a9e8_5547802804bb.slice/crio-d8aa04e4059e41418b4f43d527607e6ba9ecdcc0a9dcf6cbaa9d552d0009768e WatchSource:0}: Error finding container d8aa04e4059e41418b4f43d527607e6ba9ecdcc0a9dcf6cbaa9d552d0009768e: Status 404 returned error can't find the container with id d8aa04e4059e41418b4f43d527607e6ba9ecdcc0a9dcf6cbaa9d552d0009768e
Sep 16 19:21:26.076398 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:21:26.076368    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-9gxq2"]
Sep 16 19:21:26.706273 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:21:26.706213    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-9gxq2" event={"ID":"22e9c4d2-44a6-4d84-a9e8-5547802804bb","Type":"ContainerStarted","Data":"4672b1860be2da0c01be21ab8c35676cd2f4e0e88f0bd88658c4b1b51c2c1709"}
Sep 16 19:21:26.706653 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:21:26.706308    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-9gxq2" event={"ID":"22e9c4d2-44a6-4d84-a9e8-5547802804bb","Type":"ContainerStarted","Data":"d8aa04e4059e41418b4f43d527607e6ba9ecdcc0a9dcf6cbaa9d552d0009768e"}
Sep 16 19:21:30.730414 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:21:30.730373    2586 generic.go:356] "Generic (PLEG): container finished" podID="22e9c4d2-44a6-4d84-a9e8-5547802804bb" containerID="4672b1860be2da0c01be21ab8c35676cd2f4e0e88f0bd88658c4b1b51c2c1709" exitCode=0
Sep 16 19:21:30.730414 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:21:30.730418    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-9gxq2" event={"ID":"22e9c4d2-44a6-4d84-a9e8-5547802804bb","Type":"ContainerDied","Data":"4672b1860be2da0c01be21ab8c35676cd2f4e0e88f0bd88658c4b1b51c2c1709"}
Sep 16 19:21:31.738247 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:21:31.738197    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-9gxq2" event={"ID":"22e9c4d2-44a6-4d84-a9e8-5547802804bb","Type":"ContainerStarted","Data":"7c61c3ae7d2e64c89e5d333219d2c6be9bdd4a0270473ecf6bada5fe2f1f87b1"}
Sep 16 19:21:31.738666 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:21:31.738279    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-9gxq2" event={"ID":"22e9c4d2-44a6-4d84-a9e8-5547802804bb","Type":"ContainerStarted","Data":"bf2936600a2a63e67acaba56028f1e222aa301186e9cd159711f5baa4a3df5a5"}
Sep 16 19:21:31.738666 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:21:31.738549    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-9gxq2"
Sep 16 19:21:31.738779 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:21:31.738709    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-9gxq2"
Sep 16 19:21:31.740008 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:21:31.739981    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-9gxq2" podUID="22e9c4d2-44a6-4d84-a9e8-5547802804bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 16 19:21:31.771660 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:21:31.771616    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-9gxq2" podStartSLOduration=3.174687454 podStartE2EDuration="6.771602901s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.596915447s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:21:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:21:31.76836255 +0000 UTC m=+2467.212224126" watchObservedRunningTime="2026-09-16 19:21:31.771602901 +0000 UTC m=+2467.215464477"
Sep 16 19:21:32.742835 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:21:32.742801    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-9gxq2" podUID="22e9c4d2-44a6-4d84-a9e8-5547802804bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 16 19:21:35.542765 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:21:35.542723    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-djvhk" podUID="6d382551-8624-42d3-9fda-564392d8a193" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 16 19:21:37.747128 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:21:37.747096    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-9gxq2"
Sep 16 19:21:37.748072 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:21:37.748039    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-9gxq2" podUID="22e9c4d2-44a6-4d84-a9e8-5547802804bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 16 19:21:45.543170 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:21:45.543127    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-djvhk" podUID="6d382551-8624-42d3-9fda-564392d8a193" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 16 19:21:47.748040 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:21:47.748000    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-9gxq2" podUID="22e9c4d2-44a6-4d84-a9e8-5547802804bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 16 19:21:55.543422 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:21:55.543379    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-djvhk" podUID="6d382551-8624-42d3-9fda-564392d8a193" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Sep 16 19:21:57.748201 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:21:57.748162    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-9gxq2" podUID="22e9c4d2-44a6-4d84-a9e8-5547802804bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 16 19:22:05.543458 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:05.543427    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-djvhk"
Sep 16 19:22:06.676450 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:06.676410    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-djvhk"]
Sep 16 19:22:06.676830 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:06.676660    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-djvhk" podUID="6d382551-8624-42d3-9fda-564392d8a193" containerName="kserve-container" containerID="cri-o://c5006b7efbc8fcc5ac3487190184bc7ba12adb7411cd2278ff12479dc465fd6d" gracePeriod=30
Sep 16 19:22:06.676830 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:06.676713    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-djvhk" podUID="6d382551-8624-42d3-9fda-564392d8a193" containerName="kube-rbac-proxy" containerID="cri-o://5c093166d9ff3d25a83b4012ef7a72c86a50542cf4320789b96582ab4cbdc94c" gracePeriod=30
Sep 16 19:22:06.897577 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:06.897544    2586 generic.go:356] "Generic (PLEG): container finished" podID="6d382551-8624-42d3-9fda-564392d8a193" containerID="5c093166d9ff3d25a83b4012ef7a72c86a50542cf4320789b96582ab4cbdc94c" exitCode=2
Sep 16 19:22:06.897740 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:06.897616    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-djvhk" event={"ID":"6d382551-8624-42d3-9fda-564392d8a193","Type":"ContainerDied","Data":"5c093166d9ff3d25a83b4012ef7a72c86a50542cf4320789b96582ab4cbdc94c"}
Sep 16 19:22:07.748199 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:07.748161    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-9gxq2" podUID="22e9c4d2-44a6-4d84-a9e8-5547802804bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 16 19:22:10.538004 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:10.537954    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-djvhk" podUID="6d382551-8624-42d3-9fda-564392d8a193" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.58:8643/healthz\": dial tcp 10.134.0.58:8643: connect: connection refused"
Sep 16 19:22:11.523774 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:11.523747    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-djvhk"
Sep 16 19:22:11.614666 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:11.614628    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d382551-8624-42d3-9fda-564392d8a193-cabundle-cert\") pod \"6d382551-8624-42d3-9fda-564392d8a193\" (UID: \"6d382551-8624-42d3-9fda-564392d8a193\") "
Sep 16 19:22:11.614666 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:11.614669    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d382551-8624-42d3-9fda-564392d8a193-kserve-provision-location\") pod \"6d382551-8624-42d3-9fda-564392d8a193\" (UID: \"6d382551-8624-42d3-9fda-564392d8a193\") "
Sep 16 19:22:11.615106 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:11.614696    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s8jj2\" (UniqueName: \"kubernetes.io/projected/6d382551-8624-42d3-9fda-564392d8a193-kube-api-access-s8jj2\") pod \"6d382551-8624-42d3-9fda-564392d8a193\" (UID: \"6d382551-8624-42d3-9fda-564392d8a193\") "
Sep 16 19:22:11.615106 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:11.614715    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d382551-8624-42d3-9fda-564392d8a193-kube-rbac-proxy-sar-config\") pod \"6d382551-8624-42d3-9fda-564392d8a193\" (UID: \"6d382551-8624-42d3-9fda-564392d8a193\") "
Sep 16 19:22:11.615106 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:11.614755    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d382551-8624-42d3-9fda-564392d8a193-proxy-tls\") pod \"6d382551-8624-42d3-9fda-564392d8a193\" (UID: \"6d382551-8624-42d3-9fda-564392d8a193\") "
Sep 16 19:22:11.615106 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:11.615059    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d382551-8624-42d3-9fda-564392d8a193-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6d382551-8624-42d3-9fda-564392d8a193" (UID: "6d382551-8624-42d3-9fda-564392d8a193"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:22:11.615106 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:11.615067    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d382551-8624-42d3-9fda-564392d8a193-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6d382551-8624-42d3-9fda-564392d8a193" (UID: "6d382551-8624-42d3-9fda-564392d8a193"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:22:11.615360 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:11.615148    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d382551-8624-42d3-9fda-564392d8a193-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6d382551-8624-42d3-9fda-564392d8a193" (UID: "6d382551-8624-42d3-9fda-564392d8a193"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:22:11.616797 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:11.616776    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d382551-8624-42d3-9fda-564392d8a193-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6d382551-8624-42d3-9fda-564392d8a193" (UID: "6d382551-8624-42d3-9fda-564392d8a193"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:22:11.616939 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:11.616919    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d382551-8624-42d3-9fda-564392d8a193-kube-api-access-s8jj2" (OuterVolumeSpecName: "kube-api-access-s8jj2") pod "6d382551-8624-42d3-9fda-564392d8a193" (UID: "6d382551-8624-42d3-9fda-564392d8a193"). InnerVolumeSpecName "kube-api-access-s8jj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:22:11.715586 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:11.715499    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8jj2\" (UniqueName: \"kubernetes.io/projected/6d382551-8624-42d3-9fda-564392d8a193-kube-api-access-s8jj2\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:22:11.715586 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:11.715533    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d382551-8624-42d3-9fda-564392d8a193-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:22:11.715586 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:11.715544    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d382551-8624-42d3-9fda-564392d8a193-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:22:11.715586 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:11.715555    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d382551-8624-42d3-9fda-564392d8a193-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:22:11.715586 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:11.715563    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d382551-8624-42d3-9fda-564392d8a193-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:22:11.922559 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:11.922520    2586 generic.go:356] "Generic (PLEG): container finished" podID="6d382551-8624-42d3-9fda-564392d8a193" containerID="c5006b7efbc8fcc5ac3487190184bc7ba12adb7411cd2278ff12479dc465fd6d" exitCode=0
Sep 16 19:22:11.922743 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:11.922598    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-djvhk"
Sep 16 19:22:11.922743 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:11.922612    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-djvhk" event={"ID":"6d382551-8624-42d3-9fda-564392d8a193","Type":"ContainerDied","Data":"c5006b7efbc8fcc5ac3487190184bc7ba12adb7411cd2278ff12479dc465fd6d"}
Sep 16 19:22:11.922743 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:11.922719    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-djvhk" event={"ID":"6d382551-8624-42d3-9fda-564392d8a193","Type":"ContainerDied","Data":"67fca19a5587395202e73e30c894bcba9287eb7152df35173d9846f445ae4500"}
Sep 16 19:22:11.922931 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:11.922756    2586 scope.go:117] "RemoveContainer" containerID="5c093166d9ff3d25a83b4012ef7a72c86a50542cf4320789b96582ab4cbdc94c"
Sep 16 19:22:11.932146 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:11.932125    2586 scope.go:117] "RemoveContainer" containerID="c5006b7efbc8fcc5ac3487190184bc7ba12adb7411cd2278ff12479dc465fd6d"
Sep 16 19:22:11.941414 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:11.941368    2586 scope.go:117] "RemoveContainer" containerID="e0cecb1a4f50668d06f4268b24ac7eebe088c483922fde0057c59f15d8c84260"
Sep 16 19:22:11.949365 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:11.949343    2586 scope.go:117] "RemoveContainer" containerID="5c093166d9ff3d25a83b4012ef7a72c86a50542cf4320789b96582ab4cbdc94c"
Sep 16 19:22:11.949668 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:22:11.949642    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c093166d9ff3d25a83b4012ef7a72c86a50542cf4320789b96582ab4cbdc94c\": container with ID starting with 5c093166d9ff3d25a83b4012ef7a72c86a50542cf4320789b96582ab4cbdc94c not found: ID does not exist" containerID="5c093166d9ff3d25a83b4012ef7a72c86a50542cf4320789b96582ab4cbdc94c"
Sep 16 19:22:11.949741 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:11.949678    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c093166d9ff3d25a83b4012ef7a72c86a50542cf4320789b96582ab4cbdc94c"} err="failed to get container status \"5c093166d9ff3d25a83b4012ef7a72c86a50542cf4320789b96582ab4cbdc94c\": rpc error: code = NotFound desc = could not find container \"5c093166d9ff3d25a83b4012ef7a72c86a50542cf4320789b96582ab4cbdc94c\": container with ID starting with 5c093166d9ff3d25a83b4012ef7a72c86a50542cf4320789b96582ab4cbdc94c not found: ID does not exist"
Sep 16 19:22:11.949741 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:11.949695    2586 scope.go:117] "RemoveContainer" containerID="c5006b7efbc8fcc5ac3487190184bc7ba12adb7411cd2278ff12479dc465fd6d"
Sep 16 19:22:11.949942 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:22:11.949926    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5006b7efbc8fcc5ac3487190184bc7ba12adb7411cd2278ff12479dc465fd6d\": container with ID starting with c5006b7efbc8fcc5ac3487190184bc7ba12adb7411cd2278ff12479dc465fd6d not found: ID does not exist" containerID="c5006b7efbc8fcc5ac3487190184bc7ba12adb7411cd2278ff12479dc465fd6d"
Sep 16 19:22:11.950002 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:11.949948    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5006b7efbc8fcc5ac3487190184bc7ba12adb7411cd2278ff12479dc465fd6d"} err="failed to get container status \"c5006b7efbc8fcc5ac3487190184bc7ba12adb7411cd2278ff12479dc465fd6d\": rpc error: code = NotFound desc = could not find container \"c5006b7efbc8fcc5ac3487190184bc7ba12adb7411cd2278ff12479dc465fd6d\": container with ID starting with c5006b7efbc8fcc5ac3487190184bc7ba12adb7411cd2278ff12479dc465fd6d not found: ID does not exist"
Sep 16 19:22:11.950002 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:11.949963    2586 scope.go:117] "RemoveContainer" containerID="e0cecb1a4f50668d06f4268b24ac7eebe088c483922fde0057c59f15d8c84260"
Sep 16 19:22:11.950264 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:22:11.950224    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0cecb1a4f50668d06f4268b24ac7eebe088c483922fde0057c59f15d8c84260\": container with ID starting with e0cecb1a4f50668d06f4268b24ac7eebe088c483922fde0057c59f15d8c84260 not found: ID does not exist" containerID="e0cecb1a4f50668d06f4268b24ac7eebe088c483922fde0057c59f15d8c84260"
Sep 16 19:22:11.950341 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:11.950272    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0cecb1a4f50668d06f4268b24ac7eebe088c483922fde0057c59f15d8c84260"} err="failed to get container status \"e0cecb1a4f50668d06f4268b24ac7eebe088c483922fde0057c59f15d8c84260\": rpc error: code = NotFound desc = could not find container \"e0cecb1a4f50668d06f4268b24ac7eebe088c483922fde0057c59f15d8c84260\": container with ID starting with e0cecb1a4f50668d06f4268b24ac7eebe088c483922fde0057c59f15d8c84260 not found: ID does not exist"
Sep 16 19:22:11.957847 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:11.957822    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-djvhk"]
Sep 16 19:22:11.966938 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:11.966873    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-djvhk"]
Sep 16 19:22:12.087415 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:12.087377    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8"]
Sep 16 19:22:12.087801 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:12.087782    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d382551-8624-42d3-9fda-564392d8a193" containerName="kserve-container"
Sep 16 19:22:12.087893 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:12.087804    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d382551-8624-42d3-9fda-564392d8a193" containerName="kserve-container"
Sep 16 19:22:12.087893 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:12.087817    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d382551-8624-42d3-9fda-564392d8a193" containerName="kube-rbac-proxy"
Sep 16 19:22:12.087893 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:12.087825    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d382551-8624-42d3-9fda-564392d8a193" containerName="kube-rbac-proxy"
Sep 16 19:22:12.087893 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:12.087839    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d382551-8624-42d3-9fda-564392d8a193" containerName="storage-initializer"
Sep 16 19:22:12.087893 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:12.087848    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d382551-8624-42d3-9fda-564392d8a193" containerName="storage-initializer"
Sep 16 19:22:12.088161 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:12.087961    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d382551-8624-42d3-9fda-564392d8a193" containerName="kube-rbac-proxy"
Sep 16 19:22:12.088161 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:12.087976    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d382551-8624-42d3-9fda-564392d8a193" containerName="kserve-container"
Sep 16 19:22:12.091326 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:12.091305    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8"
Sep 16 19:22:12.095669 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:12.095647    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
Sep 16 19:22:12.102829 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:12.102811    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
Sep 16 19:22:12.122385 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:12.122358    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8"]
Sep 16 19:22:12.219768 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:12.219662    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm9tv\" (UniqueName: \"kubernetes.io/projected/efbecc83-6674-4959-85e2-3056a444075a-kube-api-access-rm9tv\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8\" (UID: \"efbecc83-6674-4959-85e2-3056a444075a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8"
Sep 16 19:22:12.219768 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:12.219717    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/efbecc83-6674-4959-85e2-3056a444075a-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8\" (UID: \"efbecc83-6674-4959-85e2-3056a444075a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8"
Sep 16 19:22:12.219768 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:12.219758    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/efbecc83-6674-4959-85e2-3056a444075a-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8\" (UID: \"efbecc83-6674-4959-85e2-3056a444075a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8"
Sep 16 19:22:12.220012 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:12.219915    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/efbecc83-6674-4959-85e2-3056a444075a-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8\" (UID: \"efbecc83-6674-4959-85e2-3056a444075a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8"
Sep 16 19:22:12.220012 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:12.219959    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/efbecc83-6674-4959-85e2-3056a444075a-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8\" (UID: \"efbecc83-6674-4959-85e2-3056a444075a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8"
Sep 16 19:22:12.320743 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:12.320703    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm9tv\" (UniqueName: \"kubernetes.io/projected/efbecc83-6674-4959-85e2-3056a444075a-kube-api-access-rm9tv\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8\" (UID: \"efbecc83-6674-4959-85e2-3056a444075a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8"
Sep 16 19:22:12.320743 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:12.320745    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/efbecc83-6674-4959-85e2-3056a444075a-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8\" (UID: \"efbecc83-6674-4959-85e2-3056a444075a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8"
Sep 16 19:22:12.321015 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:12.320881    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/efbecc83-6674-4959-85e2-3056a444075a-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8\" (UID: \"efbecc83-6674-4959-85e2-3056a444075a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8"
Sep 16 19:22:12.321015 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:12.320955    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/efbecc83-6674-4959-85e2-3056a444075a-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8\" (UID: \"efbecc83-6674-4959-85e2-3056a444075a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8"
Sep 16 19:22:12.321015 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:12.320999    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/efbecc83-6674-4959-85e2-3056a444075a-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8\" (UID: \"efbecc83-6674-4959-85e2-3056a444075a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8"
Sep 16 19:22:12.321367 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:12.321344    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/efbecc83-6674-4959-85e2-3056a444075a-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8\" (UID: \"efbecc83-6674-4959-85e2-3056a444075a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8"
Sep 16 19:22:12.321610 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:12.321589    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/efbecc83-6674-4959-85e2-3056a444075a-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8\" (UID: \"efbecc83-6674-4959-85e2-3056a444075a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8"
Sep 16 19:22:12.321684 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:12.321641    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/efbecc83-6674-4959-85e2-3056a444075a-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8\" (UID: \"efbecc83-6674-4959-85e2-3056a444075a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8"
Sep 16 19:22:12.323463 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:12.323441    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/efbecc83-6674-4959-85e2-3056a444075a-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8\" (UID: \"efbecc83-6674-4959-85e2-3056a444075a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8"
Sep 16 19:22:12.341360 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:12.341329    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm9tv\" (UniqueName: \"kubernetes.io/projected/efbecc83-6674-4959-85e2-3056a444075a-kube-api-access-rm9tv\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8\" (UID: \"efbecc83-6674-4959-85e2-3056a444075a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8"
Sep 16 19:22:12.401633 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:12.401598    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8"
Sep 16 19:22:12.688406 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:12.688380    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8"]
Sep 16 19:22:12.689901 ip-10-0-129-190 kubenswrapper[2586]: W0916 19:22:12.689868    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefbecc83_6674_4959_85e2_3056a444075a.slice/crio-4a191a53c7acb2f82ce547c6b7f85ac30b016baaf469b3057b78e92cf5727d17 WatchSource:0}: Error finding container 4a191a53c7acb2f82ce547c6b7f85ac30b016baaf469b3057b78e92cf5727d17: Status 404 returned error can't find the container with id 4a191a53c7acb2f82ce547c6b7f85ac30b016baaf469b3057b78e92cf5727d17
Sep 16 19:22:12.932346 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:12.932305    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8" event={"ID":"efbecc83-6674-4959-85e2-3056a444075a","Type":"ContainerStarted","Data":"d5127150942233cd9e2171a87d19749d0f6fa775d93f3fc0551512ac309317ef"}
Sep 16 19:22:12.932555 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:12.932390    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8" event={"ID":"efbecc83-6674-4959-85e2-3056a444075a","Type":"ContainerStarted","Data":"4a191a53c7acb2f82ce547c6b7f85ac30b016baaf469b3057b78e92cf5727d17"}
Sep 16 19:22:13.095153 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:13.095069    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d382551-8624-42d3-9fda-564392d8a193" path="/var/lib/kubelet/pods/6d382551-8624-42d3-9fda-564392d8a193/volumes"
Sep 16 19:22:16.953376 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:16.953291    2586 generic.go:356] "Generic (PLEG): container finished" podID="efbecc83-6674-4959-85e2-3056a444075a" containerID="d5127150942233cd9e2171a87d19749d0f6fa775d93f3fc0551512ac309317ef" exitCode=0
Sep 16 19:22:16.953755 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:16.953371    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8" event={"ID":"efbecc83-6674-4959-85e2-3056a444075a","Type":"ContainerDied","Data":"d5127150942233cd9e2171a87d19749d0f6fa775d93f3fc0551512ac309317ef"}
Sep 16 19:22:17.748139 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:17.748093    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-9gxq2" podUID="22e9c4d2-44a6-4d84-a9e8-5547802804bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 16 19:22:17.962611 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:17.962576    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8" event={"ID":"efbecc83-6674-4959-85e2-3056a444075a","Type":"ContainerStarted","Data":"35a9396ef175a2f1908b073142313ef10a8cba7cce6bb7c04054be566f4f1683"}
Sep 16 19:22:17.963030 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:17.962669    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8" event={"ID":"efbecc83-6674-4959-85e2-3056a444075a","Type":"ContainerStarted","Data":"62865a3a9a2930dbff10d59f0b4ac5ff3b73f84931f2e24a1990f7e387e9eee5"}
Sep 16 19:22:17.963030 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:17.962863    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8"
Sep 16 19:22:18.004873 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:18.004775    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8" podStartSLOduration=2.093065149 podStartE2EDuration="6.004759133s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.911693984s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:22:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:22:18.003410732 +0000 UTC m=+2513.447272310" watchObservedRunningTime="2026-09-16 19:22:18.004759133 +0000 UTC m=+2513.448620708"
Sep 16 19:22:18.967709 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:18.967670    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8"
Sep 16 19:22:18.969064 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:18.969039    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8" podUID="efbecc83-6674-4959-85e2-3056a444075a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 16 19:22:19.972149 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:19.972113    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8" podUID="efbecc83-6674-4959-85e2-3056a444075a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 16 19:22:24.977134 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:24.977101    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8"
Sep 16 19:22:24.977744 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:24.977710    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8" podUID="efbecc83-6674-4959-85e2-3056a444075a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 16 19:22:27.748690 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:27.748647    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-9gxq2" podUID="22e9c4d2-44a6-4d84-a9e8-5547802804bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 16 19:22:34.977587 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:34.977548    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8" podUID="efbecc83-6674-4959-85e2-3056a444075a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 16 19:22:37.749065 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:37.749035    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-9gxq2"
Sep 16 19:22:44.978604 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:44.978563    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8" podUID="efbecc83-6674-4959-85e2-3056a444075a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 16 19:22:45.455558 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:45.455524    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-9gxq2"]
Sep 16 19:22:45.455816 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:45.455786    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-9gxq2" podUID="22e9c4d2-44a6-4d84-a9e8-5547802804bb" containerName="kserve-container" containerID="cri-o://bf2936600a2a63e67acaba56028f1e222aa301186e9cd159711f5baa4a3df5a5" gracePeriod=30
Sep 16 19:22:45.455908 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:45.455823    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-9gxq2" podUID="22e9c4d2-44a6-4d84-a9e8-5547802804bb" containerName="kube-rbac-proxy" containerID="cri-o://7c61c3ae7d2e64c89e5d333219d2c6be9bdd4a0270473ecf6bada5fe2f1f87b1" gracePeriod=30
Sep 16 19:22:46.092792 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:46.092753    2586 generic.go:356] "Generic (PLEG): container finished" podID="22e9c4d2-44a6-4d84-a9e8-5547802804bb" containerID="7c61c3ae7d2e64c89e5d333219d2c6be9bdd4a0270473ecf6bada5fe2f1f87b1" exitCode=2
Sep 16 19:22:46.093170 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:46.092831    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-9gxq2" event={"ID":"22e9c4d2-44a6-4d84-a9e8-5547802804bb","Type":"ContainerDied","Data":"7c61c3ae7d2e64c89e5d333219d2c6be9bdd4a0270473ecf6bada5fe2f1f87b1"}
Sep 16 19:22:47.743127 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:47.743085    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-9gxq2" podUID="22e9c4d2-44a6-4d84-a9e8-5547802804bb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.59:8643/healthz\": dial tcp 10.134.0.59:8643: connect: connection refused"
Sep 16 19:22:47.748377 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:47.748348    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-9gxq2" podUID="22e9c4d2-44a6-4d84-a9e8-5547802804bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 16 19:22:49.108467 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:49.108430    2586 generic.go:356] "Generic (PLEG): container finished" podID="22e9c4d2-44a6-4d84-a9e8-5547802804bb" containerID="bf2936600a2a63e67acaba56028f1e222aa301186e9cd159711f5baa4a3df5a5" exitCode=0
Sep 16 19:22:49.108847 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:49.108471    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-9gxq2" event={"ID":"22e9c4d2-44a6-4d84-a9e8-5547802804bb","Type":"ContainerDied","Data":"bf2936600a2a63e67acaba56028f1e222aa301186e9cd159711f5baa4a3df5a5"}
Sep 16 19:22:49.202753 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:49.202731    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-9gxq2"
Sep 16 19:22:49.232014 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:49.231982    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22e9c4d2-44a6-4d84-a9e8-5547802804bb-kserve-provision-location\") pod \"22e9c4d2-44a6-4d84-a9e8-5547802804bb\" (UID: \"22e9c4d2-44a6-4d84-a9e8-5547802804bb\") "
Sep 16 19:22:49.232193 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:49.232043    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8k28\" (UniqueName: \"kubernetes.io/projected/22e9c4d2-44a6-4d84-a9e8-5547802804bb-kube-api-access-b8k28\") pod \"22e9c4d2-44a6-4d84-a9e8-5547802804bb\" (UID: \"22e9c4d2-44a6-4d84-a9e8-5547802804bb\") "
Sep 16 19:22:49.232193 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:49.232115    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22e9c4d2-44a6-4d84-a9e8-5547802804bb-proxy-tls\") pod \"22e9c4d2-44a6-4d84-a9e8-5547802804bb\" (UID: \"22e9c4d2-44a6-4d84-a9e8-5547802804bb\") "
Sep 16 19:22:49.232193 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:49.232142    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22e9c4d2-44a6-4d84-a9e8-5547802804bb-cabundle-cert\") pod \"22e9c4d2-44a6-4d84-a9e8-5547802804bb\" (UID: \"22e9c4d2-44a6-4d84-a9e8-5547802804bb\") "
Sep 16 19:22:49.232193 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:49.232177    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22e9c4d2-44a6-4d84-a9e8-5547802804bb-kube-rbac-proxy-sar-config\") pod \"22e9c4d2-44a6-4d84-a9e8-5547802804bb\" (UID: \"22e9c4d2-44a6-4d84-a9e8-5547802804bb\") "
Sep 16 19:22:49.232462 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:49.232432    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22e9c4d2-44a6-4d84-a9e8-5547802804bb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "22e9c4d2-44a6-4d84-a9e8-5547802804bb" (UID: "22e9c4d2-44a6-4d84-a9e8-5547802804bb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:22:49.232575 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:49.232550    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22e9c4d2-44a6-4d84-a9e8-5547802804bb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "22e9c4d2-44a6-4d84-a9e8-5547802804bb" (UID: "22e9c4d2-44a6-4d84-a9e8-5547802804bb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:22:49.232789 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:49.232739    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22e9c4d2-44a6-4d84-a9e8-5547802804bb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "22e9c4d2-44a6-4d84-a9e8-5547802804bb" (UID: "22e9c4d2-44a6-4d84-a9e8-5547802804bb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:22:49.234552 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:49.234528    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22e9c4d2-44a6-4d84-a9e8-5547802804bb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "22e9c4d2-44a6-4d84-a9e8-5547802804bb" (UID: "22e9c4d2-44a6-4d84-a9e8-5547802804bb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:22:49.234661 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:49.234598    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22e9c4d2-44a6-4d84-a9e8-5547802804bb-kube-api-access-b8k28" (OuterVolumeSpecName: "kube-api-access-b8k28") pod "22e9c4d2-44a6-4d84-a9e8-5547802804bb" (UID: "22e9c4d2-44a6-4d84-a9e8-5547802804bb"). InnerVolumeSpecName "kube-api-access-b8k28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:22:49.333426 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:49.333343    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22e9c4d2-44a6-4d84-a9e8-5547802804bb-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:22:49.333426 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:49.333373    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22e9c4d2-44a6-4d84-a9e8-5547802804bb-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:22:49.333426 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:49.333383    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22e9c4d2-44a6-4d84-a9e8-5547802804bb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:22:49.333426 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:49.333391    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22e9c4d2-44a6-4d84-a9e8-5547802804bb-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:22:49.333426 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:49.333401    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8k28\" (UniqueName: \"kubernetes.io/projected/22e9c4d2-44a6-4d84-a9e8-5547802804bb-kube-api-access-b8k28\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:22:50.114933 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:50.114897    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-9gxq2" event={"ID":"22e9c4d2-44a6-4d84-a9e8-5547802804bb","Type":"ContainerDied","Data":"d8aa04e4059e41418b4f43d527607e6ba9ecdcc0a9dcf6cbaa9d552d0009768e"}
Sep 16 19:22:50.115363 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:50.114966    2586 scope.go:117] "RemoveContainer" containerID="7c61c3ae7d2e64c89e5d333219d2c6be9bdd4a0270473ecf6bada5fe2f1f87b1"
Sep 16 19:22:50.115363 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:50.114908    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-9gxq2"
Sep 16 19:22:50.123680 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:50.123661    2586 scope.go:117] "RemoveContainer" containerID="bf2936600a2a63e67acaba56028f1e222aa301186e9cd159711f5baa4a3df5a5"
Sep 16 19:22:50.131178 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:50.131162    2586 scope.go:117] "RemoveContainer" containerID="4672b1860be2da0c01be21ab8c35676cd2f4e0e88f0bd88658c4b1b51c2c1709"
Sep 16 19:22:50.151002 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:50.150975    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-9gxq2"]
Sep 16 19:22:50.155509 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:50.155486    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-9gxq2"]
Sep 16 19:22:51.093005 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:51.092969    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22e9c4d2-44a6-4d84-a9e8-5547802804bb" path="/var/lib/kubelet/pods/22e9c4d2-44a6-4d84-a9e8-5547802804bb/volumes"
Sep 16 19:22:54.977865 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:22:54.977823    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8" podUID="efbecc83-6674-4959-85e2-3056a444075a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 16 19:23:04.978461 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:04.978422    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8" podUID="efbecc83-6674-4959-85e2-3056a444075a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 16 19:23:14.978004 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:14.977960    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8" podUID="efbecc83-6674-4959-85e2-3056a444075a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 16 19:23:24.978531 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:24.978473    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8" podUID="efbecc83-6674-4959-85e2-3056a444075a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 16 19:23:34.978385 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:34.978351    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8"
Sep 16 19:23:42.747165 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:42.747129    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8"]
Sep 16 19:23:42.747732 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:42.747391    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8" podUID="efbecc83-6674-4959-85e2-3056a444075a" containerName="kserve-container" containerID="cri-o://62865a3a9a2930dbff10d59f0b4ac5ff3b73f84931f2e24a1990f7e387e9eee5" gracePeriod=30
Sep 16 19:23:42.747732 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:42.747443    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8" podUID="efbecc83-6674-4959-85e2-3056a444075a" containerName="kube-rbac-proxy" containerID="cri-o://35a9396ef175a2f1908b073142313ef10a8cba7cce6bb7c04054be566f4f1683" gracePeriod=30
Sep 16 19:23:43.360991 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:43.360953    2586 generic.go:356] "Generic (PLEG): container finished" podID="efbecc83-6674-4959-85e2-3056a444075a" containerID="35a9396ef175a2f1908b073142313ef10a8cba7cce6bb7c04054be566f4f1683" exitCode=2
Sep 16 19:23:43.361156 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:43.361027    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8" event={"ID":"efbecc83-6674-4959-85e2-3056a444075a","Type":"ContainerDied","Data":"35a9396ef175a2f1908b073142313ef10a8cba7cce6bb7c04054be566f4f1683"}
Sep 16 19:23:44.973461 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:44.973417    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8" podUID="efbecc83-6674-4959-85e2-3056a444075a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.60:8643/healthz\": dial tcp 10.134.0.60:8643: connect: connection refused"
Sep 16 19:23:44.977785 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:44.977756    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8" podUID="efbecc83-6674-4959-85e2-3056a444075a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 16 19:23:47.193417 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:47.193388    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8"
Sep 16 19:23:47.230384 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:47.230355    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/efbecc83-6674-4959-85e2-3056a444075a-proxy-tls\") pod \"efbecc83-6674-4959-85e2-3056a444075a\" (UID: \"efbecc83-6674-4959-85e2-3056a444075a\") "
Sep 16 19:23:47.230512 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:47.230403    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/efbecc83-6674-4959-85e2-3056a444075a-kserve-provision-location\") pod \"efbecc83-6674-4959-85e2-3056a444075a\" (UID: \"efbecc83-6674-4959-85e2-3056a444075a\") "
Sep 16 19:23:47.230512 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:47.230435    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/efbecc83-6674-4959-85e2-3056a444075a-cabundle-cert\") pod \"efbecc83-6674-4959-85e2-3056a444075a\" (UID: \"efbecc83-6674-4959-85e2-3056a444075a\") "
Sep 16 19:23:47.230512 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:47.230474    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rm9tv\" (UniqueName: \"kubernetes.io/projected/efbecc83-6674-4959-85e2-3056a444075a-kube-api-access-rm9tv\") pod \"efbecc83-6674-4959-85e2-3056a444075a\" (UID: \"efbecc83-6674-4959-85e2-3056a444075a\") "
Sep 16 19:23:47.230622 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:47.230528    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/efbecc83-6674-4959-85e2-3056a444075a-kube-rbac-proxy-sar-config\") pod \"efbecc83-6674-4959-85e2-3056a444075a\" (UID: \"efbecc83-6674-4959-85e2-3056a444075a\") "
Sep 16 19:23:47.230822 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:47.230791    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efbecc83-6674-4959-85e2-3056a444075a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "efbecc83-6674-4959-85e2-3056a444075a" (UID: "efbecc83-6674-4959-85e2-3056a444075a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:23:47.230963 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:47.230823    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efbecc83-6674-4959-85e2-3056a444075a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "efbecc83-6674-4959-85e2-3056a444075a" (UID: "efbecc83-6674-4959-85e2-3056a444075a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:23:47.231011 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:47.230971    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efbecc83-6674-4959-85e2-3056a444075a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "efbecc83-6674-4959-85e2-3056a444075a" (UID: "efbecc83-6674-4959-85e2-3056a444075a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:23:47.232647 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:47.232620    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efbecc83-6674-4959-85e2-3056a444075a-kube-api-access-rm9tv" (OuterVolumeSpecName: "kube-api-access-rm9tv") pod "efbecc83-6674-4959-85e2-3056a444075a" (UID: "efbecc83-6674-4959-85e2-3056a444075a"). InnerVolumeSpecName "kube-api-access-rm9tv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:23:47.232736 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:47.232649    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efbecc83-6674-4959-85e2-3056a444075a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "efbecc83-6674-4959-85e2-3056a444075a" (UID: "efbecc83-6674-4959-85e2-3056a444075a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:23:47.331613 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:47.331502    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/efbecc83-6674-4959-85e2-3056a444075a-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:23:47.331613 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:47.331534    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/efbecc83-6674-4959-85e2-3056a444075a-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:23:47.331613 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:47.331545    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/efbecc83-6674-4959-85e2-3056a444075a-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:23:47.331613 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:47.331554    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm9tv\" (UniqueName: \"kubernetes.io/projected/efbecc83-6674-4959-85e2-3056a444075a-kube-api-access-rm9tv\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:23:47.331613 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:47.331563    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/efbecc83-6674-4959-85e2-3056a444075a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:23:47.379782 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:47.379746    2586 generic.go:356] "Generic (PLEG): container finished" podID="efbecc83-6674-4959-85e2-3056a444075a" containerID="62865a3a9a2930dbff10d59f0b4ac5ff3b73f84931f2e24a1990f7e387e9eee5" exitCode=0
Sep 16 19:23:47.379949 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:47.379818    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8" event={"ID":"efbecc83-6674-4959-85e2-3056a444075a","Type":"ContainerDied","Data":"62865a3a9a2930dbff10d59f0b4ac5ff3b73f84931f2e24a1990f7e387e9eee5"}
Sep 16 19:23:47.379949 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:47.379831    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8"
Sep 16 19:23:47.379949 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:47.379880    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8" event={"ID":"efbecc83-6674-4959-85e2-3056a444075a","Type":"ContainerDied","Data":"4a191a53c7acb2f82ce547c6b7f85ac30b016baaf469b3057b78e92cf5727d17"}
Sep 16 19:23:47.379949 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:47.379917    2586 scope.go:117] "RemoveContainer" containerID="35a9396ef175a2f1908b073142313ef10a8cba7cce6bb7c04054be566f4f1683"
Sep 16 19:23:47.389297 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:47.389268    2586 scope.go:117] "RemoveContainer" containerID="62865a3a9a2930dbff10d59f0b4ac5ff3b73f84931f2e24a1990f7e387e9eee5"
Sep 16 19:23:47.396780 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:47.396760    2586 scope.go:117] "RemoveContainer" containerID="d5127150942233cd9e2171a87d19749d0f6fa775d93f3fc0551512ac309317ef"
Sep 16 19:23:47.404145 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:47.404126    2586 scope.go:117] "RemoveContainer" containerID="35a9396ef175a2f1908b073142313ef10a8cba7cce6bb7c04054be566f4f1683"
Sep 16 19:23:47.404422 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:23:47.404402    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35a9396ef175a2f1908b073142313ef10a8cba7cce6bb7c04054be566f4f1683\": container with ID starting with 35a9396ef175a2f1908b073142313ef10a8cba7cce6bb7c04054be566f4f1683 not found: ID does not exist" containerID="35a9396ef175a2f1908b073142313ef10a8cba7cce6bb7c04054be566f4f1683"
Sep 16 19:23:47.404471 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:47.404430    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35a9396ef175a2f1908b073142313ef10a8cba7cce6bb7c04054be566f4f1683"} err="failed to get container status \"35a9396ef175a2f1908b073142313ef10a8cba7cce6bb7c04054be566f4f1683\": rpc error: code = NotFound desc = could not find container \"35a9396ef175a2f1908b073142313ef10a8cba7cce6bb7c04054be566f4f1683\": container with ID starting with 35a9396ef175a2f1908b073142313ef10a8cba7cce6bb7c04054be566f4f1683 not found: ID does not exist"
Sep 16 19:23:47.404471 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:47.404447    2586 scope.go:117] "RemoveContainer" containerID="62865a3a9a2930dbff10d59f0b4ac5ff3b73f84931f2e24a1990f7e387e9eee5"
Sep 16 19:23:47.404725 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:23:47.404707    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62865a3a9a2930dbff10d59f0b4ac5ff3b73f84931f2e24a1990f7e387e9eee5\": container with ID starting with 62865a3a9a2930dbff10d59f0b4ac5ff3b73f84931f2e24a1990f7e387e9eee5 not found: ID does not exist" containerID="62865a3a9a2930dbff10d59f0b4ac5ff3b73f84931f2e24a1990f7e387e9eee5"
Sep 16 19:23:47.404764 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:47.404732    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62865a3a9a2930dbff10d59f0b4ac5ff3b73f84931f2e24a1990f7e387e9eee5"} err="failed to get container status \"62865a3a9a2930dbff10d59f0b4ac5ff3b73f84931f2e24a1990f7e387e9eee5\": rpc error: code = NotFound desc = could not find container \"62865a3a9a2930dbff10d59f0b4ac5ff3b73f84931f2e24a1990f7e387e9eee5\": container with ID starting with 62865a3a9a2930dbff10d59f0b4ac5ff3b73f84931f2e24a1990f7e387e9eee5 not found: ID does not exist"
Sep 16 19:23:47.404764 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:47.404748    2586 scope.go:117] "RemoveContainer" containerID="d5127150942233cd9e2171a87d19749d0f6fa775d93f3fc0551512ac309317ef"
Sep 16 19:23:47.404985 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:23:47.404967    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5127150942233cd9e2171a87d19749d0f6fa775d93f3fc0551512ac309317ef\": container with ID starting with d5127150942233cd9e2171a87d19749d0f6fa775d93f3fc0551512ac309317ef not found: ID does not exist" containerID="d5127150942233cd9e2171a87d19749d0f6fa775d93f3fc0551512ac309317ef"
Sep 16 19:23:47.405023 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:47.404992    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5127150942233cd9e2171a87d19749d0f6fa775d93f3fc0551512ac309317ef"} err="failed to get container status \"d5127150942233cd9e2171a87d19749d0f6fa775d93f3fc0551512ac309317ef\": rpc error: code = NotFound desc = could not find container \"d5127150942233cd9e2171a87d19749d0f6fa775d93f3fc0551512ac309317ef\": container with ID starting with d5127150942233cd9e2171a87d19749d0f6fa775d93f3fc0551512ac309317ef not found: ID does not exist"
Sep 16 19:23:47.419263 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:47.419221    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8"]
Sep 16 19:23:47.420807 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:47.420782    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-ljdm8"]
Sep 16 19:23:49.092169 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:49.092129    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efbecc83-6674-4959-85e2-3056a444075a" path="/var/lib/kubelet/pods/efbecc83-6674-4959-85e2-3056a444075a/volumes"
Sep 16 19:23:56.068843 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:56.068805    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w5xtm"]
Sep 16 19:23:56.069221 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:56.069118    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22e9c4d2-44a6-4d84-a9e8-5547802804bb" containerName="storage-initializer"
Sep 16 19:23:56.069221 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:56.069130    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="22e9c4d2-44a6-4d84-a9e8-5547802804bb" containerName="storage-initializer"
Sep 16 19:23:56.069221 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:56.069140    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efbecc83-6674-4959-85e2-3056a444075a" containerName="storage-initializer"
Sep 16 19:23:56.069221 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:56.069145    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="efbecc83-6674-4959-85e2-3056a444075a" containerName="storage-initializer"
Sep 16 19:23:56.069221 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:56.069163    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efbecc83-6674-4959-85e2-3056a444075a" containerName="kube-rbac-proxy"
Sep 16 19:23:56.069221 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:56.069169    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="efbecc83-6674-4959-85e2-3056a444075a" containerName="kube-rbac-proxy"
Sep 16 19:23:56.069221 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:56.069175    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22e9c4d2-44a6-4d84-a9e8-5547802804bb" containerName="kube-rbac-proxy"
Sep 16 19:23:56.069221 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:56.069180    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="22e9c4d2-44a6-4d84-a9e8-5547802804bb" containerName="kube-rbac-proxy"
Sep 16 19:23:56.069221 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:56.069187    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efbecc83-6674-4959-85e2-3056a444075a" containerName="kserve-container"
Sep 16 19:23:56.069221 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:56.069192    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="efbecc83-6674-4959-85e2-3056a444075a" containerName="kserve-container"
Sep 16 19:23:56.069221 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:56.069201    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22e9c4d2-44a6-4d84-a9e8-5547802804bb" containerName="kserve-container"
Sep 16 19:23:56.069221 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:56.069206    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="22e9c4d2-44a6-4d84-a9e8-5547802804bb" containerName="kserve-container"
Sep 16 19:23:56.069613 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:56.069285    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="22e9c4d2-44a6-4d84-a9e8-5547802804bb" containerName="kserve-container"
Sep 16 19:23:56.069613 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:56.069292    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="22e9c4d2-44a6-4d84-a9e8-5547802804bb" containerName="kube-rbac-proxy"
Sep 16 19:23:56.069613 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:56.069299    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="efbecc83-6674-4959-85e2-3056a444075a" containerName="kube-rbac-proxy"
Sep 16 19:23:56.069613 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:56.069306    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="efbecc83-6674-4959-85e2-3056a444075a" containerName="kserve-container"
Sep 16 19:23:56.072516 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:56.072499    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w5xtm"
Sep 16 19:23:56.076896 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:56.076871    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
Sep 16 19:23:56.077040 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:56.076876    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:23:56.077360 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:56.077343    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:23:56.077633 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:56.077610    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-pv9ct\""
Sep 16 19:23:56.077712 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:56.077610    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:23:56.077797 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:56.077781    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:23:56.077874 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:56.077861    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-predictor-serving-cert\""
Sep 16 19:23:56.092911 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:56.092885    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w5xtm"]
Sep 16 19:23:56.214494 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:56.214455    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w5xtm\" (UID: \"bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w5xtm"
Sep 16 19:23:56.214671 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:56.214521    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w5xtm\" (UID: \"bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w5xtm"
Sep 16 19:23:56.214671 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:56.214560    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w5xtm\" (UID: \"bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w5xtm"
Sep 16 19:23:56.214671 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:56.214578    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7zz2\" (UniqueName: \"kubernetes.io/projected/bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98-kube-api-access-z7zz2\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w5xtm\" (UID: \"bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w5xtm"
Sep 16 19:23:56.214778 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:56.214659    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w5xtm\" (UID: \"bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w5xtm"
Sep 16 19:23:56.315661 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:56.315625    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w5xtm\" (UID: \"bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w5xtm"
Sep 16 19:23:56.315661 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:56.315663    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7zz2\" (UniqueName: \"kubernetes.io/projected/bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98-kube-api-access-z7zz2\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w5xtm\" (UID: \"bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w5xtm"
Sep 16 19:23:56.315908 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:56.315705    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w5xtm\" (UID: \"bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w5xtm"
Sep 16 19:23:56.315908 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:56.315742    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w5xtm\" (UID: \"bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w5xtm"
Sep 16 19:23:56.315908 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:56.315814    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w5xtm\" (UID: \"bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w5xtm"
Sep 16 19:23:56.316294 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:56.316272    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w5xtm\" (UID: \"bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w5xtm"
Sep 16 19:23:56.316512 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:56.316492    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w5xtm\" (UID: \"bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w5xtm"
Sep 16 19:23:56.316552 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:56.316521    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w5xtm\" (UID: \"bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w5xtm"
Sep 16 19:23:56.318160 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:56.318143    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w5xtm\" (UID: \"bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w5xtm"
Sep 16 19:23:56.330542 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:56.330484    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7zz2\" (UniqueName: \"kubernetes.io/projected/bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98-kube-api-access-z7zz2\") pod \"isvc-xgboost-v2-predictor-fcc668bc-w5xtm\" (UID: \"bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w5xtm"
Sep 16 19:23:56.384277 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:56.384250    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w5xtm"
Sep 16 19:23:56.678138 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:56.678102    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w5xtm"]
Sep 16 19:23:56.683918 ip-10-0-129-190 kubenswrapper[2586]: W0916 19:23:56.683887    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb8d8bf8_1a1d_4b4c_a2be_1e310029fa98.slice/crio-64b28b1169b99ac0cf2a35f4c187ced547e706e0f4e67eeeb88ead9c3d79d060 WatchSource:0}: Error finding container 64b28b1169b99ac0cf2a35f4c187ced547e706e0f4e67eeeb88ead9c3d79d060: Status 404 returned error can't find the container with id 64b28b1169b99ac0cf2a35f4c187ced547e706e0f4e67eeeb88ead9c3d79d060
Sep 16 19:23:57.425962 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:57.425928    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w5xtm" event={"ID":"bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98","Type":"ContainerStarted","Data":"cf883a169341f314bb9fec1842639f016f062bd5955f8d075aca521d06d0165c"}
Sep 16 19:23:57.426353 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:23:57.425991    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w5xtm" event={"ID":"bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98","Type":"ContainerStarted","Data":"64b28b1169b99ac0cf2a35f4c187ced547e706e0f4e67eeeb88ead9c3d79d060"}
Sep 16 19:24:01.443736 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:24:01.443704    2586 generic.go:356] "Generic (PLEG): container finished" podID="bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98" containerID="cf883a169341f314bb9fec1842639f016f062bd5955f8d075aca521d06d0165c" exitCode=0
Sep 16 19:24:01.444144 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:24:01.443758    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w5xtm" event={"ID":"bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98","Type":"ContainerDied","Data":"cf883a169341f314bb9fec1842639f016f062bd5955f8d075aca521d06d0165c"}
Sep 16 19:24:02.451881 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:24:02.451846    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w5xtm" event={"ID":"bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98","Type":"ContainerStarted","Data":"93ead68248656a7bf4077cc07be91d54d2a1678661a010d9c55e1e33a1e3ee22"}
Sep 16 19:24:02.452283 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:24:02.451935    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w5xtm" event={"ID":"bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98","Type":"ContainerStarted","Data":"a72993c923bfc3b8b2d3a9c4e478852aec93ce4134f8d3b535da6102ffdfb226"}
Sep 16 19:24:02.452283 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:24:02.452132    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w5xtm"
Sep 16 19:24:02.480447 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:24:02.480393    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w5xtm" podStartSLOduration=2.580999826 podStartE2EDuration="6.480376625s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.899376799s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:23:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:24:02.476877445 +0000 UTC m=+2617.920739020" watchObservedRunningTime="2026-09-16 19:24:02.480376625 +0000 UTC m=+2617.924238275"
Sep 16 19:24:03.456524 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:24:03.456494    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w5xtm"
Sep 16 19:24:03.457920 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:24:03.457895    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w5xtm" podUID="bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Sep 16 19:24:04.460702 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:24:04.460667    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w5xtm" podUID="bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Sep 16 19:24:09.466071 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:24:09.466040    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w5xtm"
Sep 16 19:24:09.466740 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:24:09.466710    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w5xtm" podUID="bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Sep 16 19:24:19.467345 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:24:19.467305    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w5xtm" podUID="bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Sep 16 19:24:29.467082 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:24:29.467041    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w5xtm" podUID="bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Sep 16 19:24:39.466898 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:24:39.466852    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w5xtm" podUID="bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Sep 16 19:24:43.226920 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:24:43.226879    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r"]
Sep 16 19:24:43.237892 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:24:43.237852    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r"
Sep 16 19:24:43.240369 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:24:43.240345    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
Sep 16 19:24:43.240508 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:24:43.240352    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 16 19:24:43.250991 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:24:43.250968    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r"]
Sep 16 19:24:43.326845 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:24:43.326811    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r\" (UID: \"5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r"
Sep 16 19:24:43.327017 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:24:43.326859    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r\" (UID: \"5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r"
Sep 16 19:24:43.327017 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:24:43.326911    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r\" (UID: \"5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r"
Sep 16 19:24:43.327017 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:24:43.326930    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r\" (UID: \"5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r"
Sep 16 19:24:43.327017 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:24:43.326957    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw6jj\" (UniqueName: \"kubernetes.io/projected/5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0-kube-api-access-qw6jj\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r\" (UID: \"5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r"
Sep 16 19:24:43.427621 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:24:43.427580    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r\" (UID: \"5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r"
Sep 16 19:24:43.427621 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:24:43.427622    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r\" (UID: \"5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r"
Sep 16 19:24:43.427895 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:24:43.427655    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw6jj\" (UniqueName: \"kubernetes.io/projected/5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0-kube-api-access-qw6jj\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r\" (UID: \"5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r"
Sep 16 19:24:43.427895 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:24:43.427787    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r\" (UID: \"5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r"
Sep 16 19:24:43.427895 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:24:43.427820    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r\" (UID: \"5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r"
Sep 16 19:24:43.428123 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:24:43.428096    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r\" (UID: \"5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r"
Sep 16 19:24:43.428425 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:24:43.428394    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r\" (UID: \"5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r"
Sep 16 19:24:43.428425 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:24:43.428407    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r\" (UID: \"5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r"
Sep 16 19:24:43.430224 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:24:43.430200    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r\" (UID: \"5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r"
Sep 16 19:24:43.438637 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:24:43.438608    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw6jj\" (UniqueName: \"kubernetes.io/projected/5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0-kube-api-access-qw6jj\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r\" (UID: \"5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r"
Sep 16 19:24:43.548721 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:24:43.548639    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r"
Sep 16 19:24:43.828631 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:24:43.828600    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r"]
Sep 16 19:24:43.830443 ip-10-0-129-190 kubenswrapper[2586]: W0916 19:24:43.830410    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cf1b25c_5a0f_4b37_85a9_64c9d7f6dbf0.slice/crio-e3aa3567c2800c78dcb86aa3267c58d64727ab397c05fef4c0afdee729006b0d WatchSource:0}: Error finding container e3aa3567c2800c78dcb86aa3267c58d64727ab397c05fef4c0afdee729006b0d: Status 404 returned error can't find the container with id e3aa3567c2800c78dcb86aa3267c58d64727ab397c05fef4c0afdee729006b0d
Sep 16 19:24:44.646300 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:24:44.646259    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r" event={"ID":"5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0","Type":"ContainerStarted","Data":"bc4779529114df56e943a85a456d8c2444d6df91392707c3a230c43a10239d37"}
Sep 16 19:24:44.646711 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:24:44.646328    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r" event={"ID":"5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0","Type":"ContainerStarted","Data":"e3aa3567c2800c78dcb86aa3267c58d64727ab397c05fef4c0afdee729006b0d"}
Sep 16 19:24:48.665340 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:24:48.665297    2586 generic.go:356] "Generic (PLEG): container finished" podID="5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0" containerID="bc4779529114df56e943a85a456d8c2444d6df91392707c3a230c43a10239d37" exitCode=0
Sep 16 19:24:48.665751 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:24:48.665391    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r" event={"ID":"5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0","Type":"ContainerDied","Data":"bc4779529114df56e943a85a456d8c2444d6df91392707c3a230c43a10239d37"}
Sep 16 19:24:49.467054 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:24:49.467009    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w5xtm" podUID="bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Sep 16 19:24:49.673330 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:24:49.673295    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r" event={"ID":"5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0","Type":"ContainerStarted","Data":"54e7480b21d41891ffd1afc3e564c074446584292783f9a2984e36461d518dd5"}
Sep 16 19:24:49.673798 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:24:49.673366    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r" event={"ID":"5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0","Type":"ContainerStarted","Data":"3116033b222ddb66660bf807772afda114a95bf314e79d5f20136e07963ed223"}
Sep 16 19:24:49.673798 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:24:49.673579    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r"
Sep 16 19:24:49.699599 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:24:49.699547    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r" podStartSLOduration=2.703966423 podStartE2EDuration="6.699534745s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.995568322s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:24:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:24:49.69818178 +0000 UTC m=+2665.142043356" watchObservedRunningTime="2026-09-16 19:24:49.699534745 +0000 UTC m=+2665.143396324"
Sep 16 19:24:50.678514 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:24:50.678477    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r"
Sep 16 19:24:50.679725 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:24:50.679681    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r" podUID="5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 16 19:24:51.683854 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:24:51.683814    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r" podUID="5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 16 19:24:56.688407 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:24:56.688377    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r"
Sep 16 19:24:56.689383 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:24:56.689361    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r"
Sep 16 19:24:59.467447 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:24:59.467412    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w5xtm" podUID="bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Sep 16 19:25:03.269672 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:03.269638    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r"]
Sep 16 19:25:03.270098 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:03.269869    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r" podUID="5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0" containerName="kserve-container" containerID="cri-o://3116033b222ddb66660bf807772afda114a95bf314e79d5f20136e07963ed223" gracePeriod=30
Sep 16 19:25:03.270098 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:03.269910    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r" podUID="5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0" containerName="kube-rbac-proxy" containerID="cri-o://54e7480b21d41891ffd1afc3e564c074446584292783f9a2984e36461d518dd5" gracePeriod=30
Sep 16 19:25:03.744117 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:03.744087    2586 generic.go:356] "Generic (PLEG): container finished" podID="5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0" containerID="54e7480b21d41891ffd1afc3e564c074446584292783f9a2984e36461d518dd5" exitCode=2
Sep 16 19:25:03.744280 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:03.744160    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r" event={"ID":"5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0","Type":"ContainerDied","Data":"54e7480b21d41891ffd1afc3e564c074446584292783f9a2984e36461d518dd5"}
Sep 16 19:25:03.916717 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:03.916687    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r"
Sep 16 19:25:04.019965 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:04.019872    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qw6jj\" (UniqueName: \"kubernetes.io/projected/5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0-kube-api-access-qw6jj\") pod \"5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0\" (UID: \"5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0\") "
Sep 16 19:25:04.019965 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:04.019933    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0-kserve-provision-location\") pod \"5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0\" (UID: \"5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0\") "
Sep 16 19:25:04.020199 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:04.019984    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0-cabundle-cert\") pod \"5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0\" (UID: \"5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0\") "
Sep 16 19:25:04.020199 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:04.020026    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0-kube-rbac-proxy-sar-config\") pod \"5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0\" (UID: \"5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0\") "
Sep 16 19:25:04.020199 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:04.020060    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0-proxy-tls\") pod \"5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0\" (UID: \"5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0\") "
Sep 16 19:25:04.020404 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:04.020343    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0" (UID: "5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:25:04.020469 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:04.020430    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0" (UID: "5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:25:04.020469 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:04.020453    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0" (UID: "5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:25:04.022101 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:04.022077    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0-kube-api-access-qw6jj" (OuterVolumeSpecName: "kube-api-access-qw6jj") pod "5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0" (UID: "5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0"). InnerVolumeSpecName "kube-api-access-qw6jj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:25:04.022162 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:04.022113    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0" (UID: "5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:25:04.120634 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:04.120599    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:25:04.120634 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:04.120626    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:25:04.120634 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:04.120636    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qw6jj\" (UniqueName: \"kubernetes.io/projected/5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0-kube-api-access-qw6jj\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:25:04.120634 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:04.120646    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:25:04.120913 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:04.120656    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:25:04.750969 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:04.750938    2586 generic.go:356] "Generic (PLEG): container finished" podID="5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0" containerID="3116033b222ddb66660bf807772afda114a95bf314e79d5f20136e07963ed223" exitCode=0
Sep 16 19:25:04.751461 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:04.750996    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r"
Sep 16 19:25:04.751461 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:04.751007    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r" event={"ID":"5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0","Type":"ContainerDied","Data":"3116033b222ddb66660bf807772afda114a95bf314e79d5f20136e07963ed223"}
Sep 16 19:25:04.751461 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:04.751073    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r" event={"ID":"5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0","Type":"ContainerDied","Data":"e3aa3567c2800c78dcb86aa3267c58d64727ab397c05fef4c0afdee729006b0d"}
Sep 16 19:25:04.751461 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:04.751116    2586 scope.go:117] "RemoveContainer" containerID="54e7480b21d41891ffd1afc3e564c074446584292783f9a2984e36461d518dd5"
Sep 16 19:25:04.760522 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:04.760506    2586 scope.go:117] "RemoveContainer" containerID="3116033b222ddb66660bf807772afda114a95bf314e79d5f20136e07963ed223"
Sep 16 19:25:04.767968 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:04.767953    2586 scope.go:117] "RemoveContainer" containerID="bc4779529114df56e943a85a456d8c2444d6df91392707c3a230c43a10239d37"
Sep 16 19:25:04.775251 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:04.775210    2586 scope.go:117] "RemoveContainer" containerID="54e7480b21d41891ffd1afc3e564c074446584292783f9a2984e36461d518dd5"
Sep 16 19:25:04.775581 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:25:04.775558    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54e7480b21d41891ffd1afc3e564c074446584292783f9a2984e36461d518dd5\": container with ID starting with 54e7480b21d41891ffd1afc3e564c074446584292783f9a2984e36461d518dd5 not found: ID does not exist" containerID="54e7480b21d41891ffd1afc3e564c074446584292783f9a2984e36461d518dd5"
Sep 16 19:25:04.775663 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:04.775590    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54e7480b21d41891ffd1afc3e564c074446584292783f9a2984e36461d518dd5"} err="failed to get container status \"54e7480b21d41891ffd1afc3e564c074446584292783f9a2984e36461d518dd5\": rpc error: code = NotFound desc = could not find container \"54e7480b21d41891ffd1afc3e564c074446584292783f9a2984e36461d518dd5\": container with ID starting with 54e7480b21d41891ffd1afc3e564c074446584292783f9a2984e36461d518dd5 not found: ID does not exist"
Sep 16 19:25:04.775663 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:04.775606    2586 scope.go:117] "RemoveContainer" containerID="3116033b222ddb66660bf807772afda114a95bf314e79d5f20136e07963ed223"
Sep 16 19:25:04.775929 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:25:04.775905    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3116033b222ddb66660bf807772afda114a95bf314e79d5f20136e07963ed223\": container with ID starting with 3116033b222ddb66660bf807772afda114a95bf314e79d5f20136e07963ed223 not found: ID does not exist" containerID="3116033b222ddb66660bf807772afda114a95bf314e79d5f20136e07963ed223"
Sep 16 19:25:04.775979 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:04.775939    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3116033b222ddb66660bf807772afda114a95bf314e79d5f20136e07963ed223"} err="failed to get container status \"3116033b222ddb66660bf807772afda114a95bf314e79d5f20136e07963ed223\": rpc error: code = NotFound desc = could not find container \"3116033b222ddb66660bf807772afda114a95bf314e79d5f20136e07963ed223\": container with ID starting with 3116033b222ddb66660bf807772afda114a95bf314e79d5f20136e07963ed223 not found: ID does not exist"
Sep 16 19:25:04.775979 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:04.775961    2586 scope.go:117] "RemoveContainer" containerID="bc4779529114df56e943a85a456d8c2444d6df91392707c3a230c43a10239d37"
Sep 16 19:25:04.776271 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:25:04.776221    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc4779529114df56e943a85a456d8c2444d6df91392707c3a230c43a10239d37\": container with ID starting with bc4779529114df56e943a85a456d8c2444d6df91392707c3a230c43a10239d37 not found: ID does not exist" containerID="bc4779529114df56e943a85a456d8c2444d6df91392707c3a230c43a10239d37"
Sep 16 19:25:04.776394 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:04.776275    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc4779529114df56e943a85a456d8c2444d6df91392707c3a230c43a10239d37"} err="failed to get container status \"bc4779529114df56e943a85a456d8c2444d6df91392707c3a230c43a10239d37\": rpc error: code = NotFound desc = could not find container \"bc4779529114df56e943a85a456d8c2444d6df91392707c3a230c43a10239d37\": container with ID starting with bc4779529114df56e943a85a456d8c2444d6df91392707c3a230c43a10239d37 not found: ID does not exist"
Sep 16 19:25:04.776843 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:04.776826    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r"]
Sep 16 19:25:04.779483 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:04.779461    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-w765r"]
Sep 16 19:25:05.094791 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:05.094683    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0" path="/var/lib/kubelet/pods/5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0/volumes"
Sep 16 19:25:09.468011 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:09.467982    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w5xtm"
Sep 16 19:25:16.097302 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:16.097265    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w5xtm"]
Sep 16 19:25:16.097749 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:16.097506    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w5xtm" podUID="bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98" containerName="kserve-container" containerID="cri-o://a72993c923bfc3b8b2d3a9c4e478852aec93ce4134f8d3b535da6102ffdfb226" gracePeriod=30
Sep 16 19:25:16.097749 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:16.097537    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w5xtm" podUID="bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98" containerName="kube-rbac-proxy" containerID="cri-o://93ead68248656a7bf4077cc07be91d54d2a1678661a010d9c55e1e33a1e3ee22" gracePeriod=30
Sep 16 19:25:16.806024 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:16.805982    2586 generic.go:356] "Generic (PLEG): container finished" podID="bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98" containerID="93ead68248656a7bf4077cc07be91d54d2a1678661a010d9c55e1e33a1e3ee22" exitCode=2
Sep 16 19:25:16.806203 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:16.806033    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w5xtm" event={"ID":"bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98","Type":"ContainerDied","Data":"93ead68248656a7bf4077cc07be91d54d2a1678661a010d9c55e1e33a1e3ee22"}
Sep 16 19:25:19.461739 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:19.461658    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w5xtm" podUID="bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.61:8643/healthz\": dial tcp 10.134.0.61:8643: connect: connection refused"
Sep 16 19:25:19.467198 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:19.467168    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w5xtm" podUID="bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Sep 16 19:25:19.823908 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:19.823872    2586 generic.go:356] "Generic (PLEG): container finished" podID="bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98" containerID="a72993c923bfc3b8b2d3a9c4e478852aec93ce4134f8d3b535da6102ffdfb226" exitCode=0
Sep 16 19:25:19.824072 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:19.823937    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w5xtm" event={"ID":"bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98","Type":"ContainerDied","Data":"a72993c923bfc3b8b2d3a9c4e478852aec93ce4134f8d3b535da6102ffdfb226"}
Sep 16 19:25:19.855859 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:19.855833    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w5xtm"
Sep 16 19:25:19.949521 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:19.949488    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98-kserve-provision-location\") pod \"bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98\" (UID: \"bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98\") "
Sep 16 19:25:19.949708 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:19.949564    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98-proxy-tls\") pod \"bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98\" (UID: \"bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98\") "
Sep 16 19:25:19.949708 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:19.949589    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98-cabundle-cert\") pod \"bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98\" (UID: \"bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98\") "
Sep 16 19:25:19.949708 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:19.949691    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7zz2\" (UniqueName: \"kubernetes.io/projected/bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98-kube-api-access-z7zz2\") pod \"bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98\" (UID: \"bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98\") "
Sep 16 19:25:19.949852 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:19.949732    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98-kube-rbac-proxy-sar-config\") pod \"bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98\" (UID: \"bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98\") "
Sep 16 19:25:19.949921 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:19.949899    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98" (UID: "bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:25:19.949992 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:19.949921    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98" (UID: "bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:25:19.950046 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:19.950024    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:25:19.950098 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:19.950045    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:25:19.950147 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:19.950101    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98" (UID: "bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:25:19.951738 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:19.951714    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98-kube-api-access-z7zz2" (OuterVolumeSpecName: "kube-api-access-z7zz2") pod "bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98" (UID: "bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98"). InnerVolumeSpecName "kube-api-access-z7zz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:25:19.951738 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:19.951716    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98" (UID: "bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:25:20.051379 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:20.051282    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:25:20.051379 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:20.051324    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7zz2\" (UniqueName: \"kubernetes.io/projected/bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98-kube-api-access-z7zz2\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:25:20.051379 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:20.051335    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:25:20.831371 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:20.831338    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w5xtm" event={"ID":"bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98","Type":"ContainerDied","Data":"64b28b1169b99ac0cf2a35f4c187ced547e706e0f4e67eeeb88ead9c3d79d060"}
Sep 16 19:25:20.831371 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:20.831363    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w5xtm"
Sep 16 19:25:20.831908 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:20.831404    2586 scope.go:117] "RemoveContainer" containerID="93ead68248656a7bf4077cc07be91d54d2a1678661a010d9c55e1e33a1e3ee22"
Sep 16 19:25:20.843698 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:20.843668    2586 scope.go:117] "RemoveContainer" containerID="a72993c923bfc3b8b2d3a9c4e478852aec93ce4134f8d3b535da6102ffdfb226"
Sep 16 19:25:20.851676 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:20.851648    2586 scope.go:117] "RemoveContainer" containerID="cf883a169341f314bb9fec1842639f016f062bd5955f8d075aca521d06d0165c"
Sep 16 19:25:20.864917 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:20.864886    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w5xtm"]
Sep 16 19:25:20.872711 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:20.872683    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-w5xtm"]
Sep 16 19:25:21.092038 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:21.091965    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98" path="/var/lib/kubelet/pods/bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98/volumes"
Sep 16 19:25:21.320346 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:21.320308    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6786d494d6-9vddp"]
Sep 16 19:25:21.320702 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:21.320685    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98" containerName="kserve-container"
Sep 16 19:25:21.320778 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:21.320703    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98" containerName="kserve-container"
Sep 16 19:25:21.320778 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:21.320711    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0" containerName="kserve-container"
Sep 16 19:25:21.320778 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:21.320717    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0" containerName="kserve-container"
Sep 16 19:25:21.320778 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:21.320729    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0" containerName="storage-initializer"
Sep 16 19:25:21.320778 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:21.320736    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0" containerName="storage-initializer"
Sep 16 19:25:21.320778 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:21.320750    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98" containerName="kube-rbac-proxy"
Sep 16 19:25:21.320778 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:21.320755    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98" containerName="kube-rbac-proxy"
Sep 16 19:25:21.320778 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:21.320765    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98" containerName="storage-initializer"
Sep 16 19:25:21.320778 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:21.320770    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98" containerName="storage-initializer"
Sep 16 19:25:21.320778 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:21.320777    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0" containerName="kube-rbac-proxy"
Sep 16 19:25:21.320778 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:21.320782    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0" containerName="kube-rbac-proxy"
Sep 16 19:25:21.321364 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:21.320830    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98" containerName="kserve-container"
Sep 16 19:25:21.321364 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:21.320839    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0" containerName="kube-rbac-proxy"
Sep 16 19:25:21.321364 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:21.320846    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb8d8bf8-1a1d-4b4c-a2be-1e310029fa98" containerName="kube-rbac-proxy"
Sep 16 19:25:21.321364 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:21.320852    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cf1b25c-5a0f-4b37-85a9-64c9d7f6dbf0" containerName="kserve-container"
Sep 16 19:25:21.324083 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:21.324066    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6786d494d6-9vddp"
Sep 16 19:25:21.329652 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:21.329630    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 16 19:25:21.330034 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:21.330013    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\""
Sep 16 19:25:21.330127 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:21.330109    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:25:21.330405 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:21.330388    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-pv9ct\""
Sep 16 19:25:21.330718 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:21.330704    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:25:21.331914 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:21.331893    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
Sep 16 19:25:21.342564 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:21.342493    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6786d494d6-9vddp"]
Sep 16 19:25:21.361000 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:21.360966    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/611be502-44e4-4c3c-bfc6-02ac46e40350-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-6786d494d6-9vddp\" (UID: \"611be502-44e4-4c3c-bfc6-02ac46e40350\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6786d494d6-9vddp"
Sep 16 19:25:21.361171 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:21.361056    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/611be502-44e4-4c3c-bfc6-02ac46e40350-proxy-tls\") pod \"isvc-sklearn-s3-predictor-6786d494d6-9vddp\" (UID: \"611be502-44e4-4c3c-bfc6-02ac46e40350\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6786d494d6-9vddp"
Sep 16 19:25:21.361171 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:21.361126    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gb4f\" (UniqueName: \"kubernetes.io/projected/611be502-44e4-4c3c-bfc6-02ac46e40350-kube-api-access-6gb4f\") pod \"isvc-sklearn-s3-predictor-6786d494d6-9vddp\" (UID: \"611be502-44e4-4c3c-bfc6-02ac46e40350\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6786d494d6-9vddp"
Sep 16 19:25:21.361308 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:21.361173    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/611be502-44e4-4c3c-bfc6-02ac46e40350-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-6786d494d6-9vddp\" (UID: \"611be502-44e4-4c3c-bfc6-02ac46e40350\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6786d494d6-9vddp"
Sep 16 19:25:21.461986 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:21.461939    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/611be502-44e4-4c3c-bfc6-02ac46e40350-proxy-tls\") pod \"isvc-sklearn-s3-predictor-6786d494d6-9vddp\" (UID: \"611be502-44e4-4c3c-bfc6-02ac46e40350\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6786d494d6-9vddp"
Sep 16 19:25:21.462211 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:21.462038    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gb4f\" (UniqueName: \"kubernetes.io/projected/611be502-44e4-4c3c-bfc6-02ac46e40350-kube-api-access-6gb4f\") pod \"isvc-sklearn-s3-predictor-6786d494d6-9vddp\" (UID: \"611be502-44e4-4c3c-bfc6-02ac46e40350\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6786d494d6-9vddp"
Sep 16 19:25:21.462211 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:21.462074    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/611be502-44e4-4c3c-bfc6-02ac46e40350-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-6786d494d6-9vddp\" (UID: \"611be502-44e4-4c3c-bfc6-02ac46e40350\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6786d494d6-9vddp"
Sep 16 19:25:21.462211 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:25:21.462108    2586 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-predictor-serving-cert: secret "isvc-sklearn-s3-predictor-serving-cert" not found
Sep 16 19:25:21.462211 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:21.462118    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/611be502-44e4-4c3c-bfc6-02ac46e40350-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-6786d494d6-9vddp\" (UID: \"611be502-44e4-4c3c-bfc6-02ac46e40350\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6786d494d6-9vddp"
Sep 16 19:25:21.462446 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:25:21.462217    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/611be502-44e4-4c3c-bfc6-02ac46e40350-proxy-tls podName:611be502-44e4-4c3c-bfc6-02ac46e40350 nodeName:}" failed. No retries permitted until 2026-09-16 19:25:21.962193469 +0000 UTC m=+2697.406055024 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/611be502-44e4-4c3c-bfc6-02ac46e40350-proxy-tls") pod "isvc-sklearn-s3-predictor-6786d494d6-9vddp" (UID: "611be502-44e4-4c3c-bfc6-02ac46e40350") : secret "isvc-sklearn-s3-predictor-serving-cert" not found
Sep 16 19:25:21.462614 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:21.462591    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/611be502-44e4-4c3c-bfc6-02ac46e40350-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-6786d494d6-9vddp\" (UID: \"611be502-44e4-4c3c-bfc6-02ac46e40350\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6786d494d6-9vddp"
Sep 16 19:25:21.462817 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:21.462800    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/611be502-44e4-4c3c-bfc6-02ac46e40350-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-6786d494d6-9vddp\" (UID: \"611be502-44e4-4c3c-bfc6-02ac46e40350\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6786d494d6-9vddp"
Sep 16 19:25:21.475143 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:21.475111    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gb4f\" (UniqueName: \"kubernetes.io/projected/611be502-44e4-4c3c-bfc6-02ac46e40350-kube-api-access-6gb4f\") pod \"isvc-sklearn-s3-predictor-6786d494d6-9vddp\" (UID: \"611be502-44e4-4c3c-bfc6-02ac46e40350\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6786d494d6-9vddp"
Sep 16 19:25:21.966173 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:21.966133    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/611be502-44e4-4c3c-bfc6-02ac46e40350-proxy-tls\") pod \"isvc-sklearn-s3-predictor-6786d494d6-9vddp\" (UID: \"611be502-44e4-4c3c-bfc6-02ac46e40350\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6786d494d6-9vddp"
Sep 16 19:25:21.968722 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:21.968696    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/611be502-44e4-4c3c-bfc6-02ac46e40350-proxy-tls\") pod \"isvc-sklearn-s3-predictor-6786d494d6-9vddp\" (UID: \"611be502-44e4-4c3c-bfc6-02ac46e40350\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6786d494d6-9vddp"
Sep 16 19:25:22.234349 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:22.234272    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6786d494d6-9vddp"
Sep 16 19:25:22.525882 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:22.525855    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6786d494d6-9vddp"]
Sep 16 19:25:22.528281 ip-10-0-129-190 kubenswrapper[2586]: W0916 19:25:22.528255    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod611be502_44e4_4c3c_bfc6_02ac46e40350.slice/crio-59dcac136e53b460fe550dfb48dcd1436e2abc0e0737385baf55f75cc2abe3e9 WatchSource:0}: Error finding container 59dcac136e53b460fe550dfb48dcd1436e2abc0e0737385baf55f75cc2abe3e9: Status 404 returned error can't find the container with id 59dcac136e53b460fe550dfb48dcd1436e2abc0e0737385baf55f75cc2abe3e9
Sep 16 19:25:22.844504 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:22.844407    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6786d494d6-9vddp" event={"ID":"611be502-44e4-4c3c-bfc6-02ac46e40350","Type":"ContainerStarted","Data":"93949c4b96c1b37c6b7d31d522dbe42b5727674bd559bac6bf0af1350c43b1af"}
Sep 16 19:25:22.844504 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:22.844476    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6786d494d6-9vddp" event={"ID":"611be502-44e4-4c3c-bfc6-02ac46e40350","Type":"ContainerStarted","Data":"59dcac136e53b460fe550dfb48dcd1436e2abc0e0737385baf55f75cc2abe3e9"}
Sep 16 19:25:23.850967 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:23.850934    2586 generic.go:356] "Generic (PLEG): container finished" podID="611be502-44e4-4c3c-bfc6-02ac46e40350" containerID="93949c4b96c1b37c6b7d31d522dbe42b5727674bd559bac6bf0af1350c43b1af" exitCode=0
Sep 16 19:25:23.851371 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:23.851015    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6786d494d6-9vddp" event={"ID":"611be502-44e4-4c3c-bfc6-02ac46e40350","Type":"ContainerDied","Data":"93949c4b96c1b37c6b7d31d522dbe42b5727674bd559bac6bf0af1350c43b1af"}
Sep 16 19:25:24.857929 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:24.857891    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6786d494d6-9vddp" event={"ID":"611be502-44e4-4c3c-bfc6-02ac46e40350","Type":"ContainerStarted","Data":"ce8320311b668c53a8a677f5f473bf0c69de060a4896874400cc5cb9395ced6a"}
Sep 16 19:25:24.858326 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:24.857964    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6786d494d6-9vddp" event={"ID":"611be502-44e4-4c3c-bfc6-02ac46e40350","Type":"ContainerStarted","Data":"c364653a79d5e61ae738082be23e5abe8dd8eb91d8db2be7de997ebe6119389e"}
Sep 16 19:25:24.858326 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:24.857999    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6786d494d6-9vddp"
Sep 16 19:25:24.858326 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:24.858146    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6786d494d6-9vddp"
Sep 16 19:25:24.859417 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:24.859392    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6786d494d6-9vddp" podUID="611be502-44e4-4c3c-bfc6-02ac46e40350" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 16 19:25:24.912066 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:24.912021    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6786d494d6-9vddp" podStartSLOduration=2.728511794 podStartE2EDuration="3.91200519s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.183493396s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:25:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:25:24.911303697 +0000 UTC m=+2700.355165273" watchObservedRunningTime="2026-09-16 19:25:24.91200519 +0000 UTC m=+2700.355866766"
Sep 16 19:25:25.862435 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:25.862396    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6786d494d6-9vddp" podUID="611be502-44e4-4c3c-bfc6-02ac46e40350" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 16 19:25:30.867566 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:30.867529    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6786d494d6-9vddp"
Sep 16 19:25:30.868177 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:30.868146    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6786d494d6-9vddp" podUID="611be502-44e4-4c3c-bfc6-02ac46e40350" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 16 19:25:40.868707 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:40.868667    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6786d494d6-9vddp" podUID="611be502-44e4-4c3c-bfc6-02ac46e40350" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 16 19:25:50.868088 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:25:50.868049    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6786d494d6-9vddp" podUID="611be502-44e4-4c3c-bfc6-02ac46e40350" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 16 19:26:00.868468 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:00.868430    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6786d494d6-9vddp" podUID="611be502-44e4-4c3c-bfc6-02ac46e40350" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 16 19:26:03.892985 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:03.892951    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv"]
Sep 16 19:26:03.896555 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:03.896534    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv"
Sep 16 19:26:03.901917 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:03.901893    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:26:03.902067 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:03.901962    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:26:03.902067 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:03.901963    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
Sep 16 19:26:03.902067 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:03.901903    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
Sep 16 19:26:03.915338 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:03.915312    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv"]
Sep 16 19:26:03.927509 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:03.927482    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/497855df-2858-4915-9e8e-3504b3a6c276-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-76d4c69869-4jtdv\" (UID: \"497855df-2858-4915-9e8e-3504b3a6c276\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv"
Sep 16 19:26:03.927655 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:03.927527    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/497855df-2858-4915-9e8e-3504b3a6c276-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-76d4c69869-4jtdv\" (UID: \"497855df-2858-4915-9e8e-3504b3a6c276\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv"
Sep 16 19:26:03.927655 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:03.927570    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/497855df-2858-4915-9e8e-3504b3a6c276-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-76d4c69869-4jtdv\" (UID: \"497855df-2858-4915-9e8e-3504b3a6c276\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv"
Sep 16 19:26:03.927655 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:03.927650    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/497855df-2858-4915-9e8e-3504b3a6c276-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-76d4c69869-4jtdv\" (UID: \"497855df-2858-4915-9e8e-3504b3a6c276\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv"
Sep 16 19:26:03.927844 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:03.927742    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clcxn\" (UniqueName: \"kubernetes.io/projected/497855df-2858-4915-9e8e-3504b3a6c276-kube-api-access-clcxn\") pod \"isvc-sklearn-mcp-predictor-76d4c69869-4jtdv\" (UID: \"497855df-2858-4915-9e8e-3504b3a6c276\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv"
Sep 16 19:26:04.028438 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:04.028406    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/497855df-2858-4915-9e8e-3504b3a6c276-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-76d4c69869-4jtdv\" (UID: \"497855df-2858-4915-9e8e-3504b3a6c276\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv"
Sep 16 19:26:04.028438 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:04.028441    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/497855df-2858-4915-9e8e-3504b3a6c276-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-76d4c69869-4jtdv\" (UID: \"497855df-2858-4915-9e8e-3504b3a6c276\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv"
Sep 16 19:26:04.028693 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:04.028467    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/497855df-2858-4915-9e8e-3504b3a6c276-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-76d4c69869-4jtdv\" (UID: \"497855df-2858-4915-9e8e-3504b3a6c276\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv"
Sep 16 19:26:04.028693 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:04.028521    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/497855df-2858-4915-9e8e-3504b3a6c276-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-76d4c69869-4jtdv\" (UID: \"497855df-2858-4915-9e8e-3504b3a6c276\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv"
Sep 16 19:26:04.028693 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:04.028554    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clcxn\" (UniqueName: \"kubernetes.io/projected/497855df-2858-4915-9e8e-3504b3a6c276-kube-api-access-clcxn\") pod \"isvc-sklearn-mcp-predictor-76d4c69869-4jtdv\" (UID: \"497855df-2858-4915-9e8e-3504b3a6c276\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv"
Sep 16 19:26:04.028693 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:26:04.028574    2586 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-serving-cert: secret "isvc-sklearn-mcp-predictor-serving-cert" not found
Sep 16 19:26:04.028693 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:26:04.028659    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/497855df-2858-4915-9e8e-3504b3a6c276-proxy-tls podName:497855df-2858-4915-9e8e-3504b3a6c276 nodeName:}" failed. No retries permitted until 2026-09-16 19:26:04.528635077 +0000 UTC m=+2739.972496634 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/497855df-2858-4915-9e8e-3504b3a6c276-proxy-tls") pod "isvc-sklearn-mcp-predictor-76d4c69869-4jtdv" (UID: "497855df-2858-4915-9e8e-3504b3a6c276") : secret "isvc-sklearn-mcp-predictor-serving-cert" not found
Sep 16 19:26:04.029014 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:04.028840    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/497855df-2858-4915-9e8e-3504b3a6c276-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-76d4c69869-4jtdv\" (UID: \"497855df-2858-4915-9e8e-3504b3a6c276\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv"
Sep 16 19:26:04.029225 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:04.029204    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/497855df-2858-4915-9e8e-3504b3a6c276-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-76d4c69869-4jtdv\" (UID: \"497855df-2858-4915-9e8e-3504b3a6c276\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv"
Sep 16 19:26:04.029324 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:04.029254    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/497855df-2858-4915-9e8e-3504b3a6c276-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-76d4c69869-4jtdv\" (UID: \"497855df-2858-4915-9e8e-3504b3a6c276\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv"
Sep 16 19:26:04.041041 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:04.041016    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clcxn\" (UniqueName: \"kubernetes.io/projected/497855df-2858-4915-9e8e-3504b3a6c276-kube-api-access-clcxn\") pod \"isvc-sklearn-mcp-predictor-76d4c69869-4jtdv\" (UID: \"497855df-2858-4915-9e8e-3504b3a6c276\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv"
Sep 16 19:26:04.533242 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:04.533202    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/497855df-2858-4915-9e8e-3504b3a6c276-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-76d4c69869-4jtdv\" (UID: \"497855df-2858-4915-9e8e-3504b3a6c276\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv"
Sep 16 19:26:04.535695 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:04.535657    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/497855df-2858-4915-9e8e-3504b3a6c276-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-76d4c69869-4jtdv\" (UID: \"497855df-2858-4915-9e8e-3504b3a6c276\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv"
Sep 16 19:26:04.806982 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:04.806894    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv"
Sep 16 19:26:05.093071 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:05.093049    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv"]
Sep 16 19:26:05.095093 ip-10-0-129-190 kubenswrapper[2586]: W0916 19:26:05.095067    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod497855df_2858_4915_9e8e_3504b3a6c276.slice/crio-89dc12dfa961f4fef7dede2b20314beb930e93f1f6a5bb214efe2d6270b5473b WatchSource:0}: Error finding container 89dc12dfa961f4fef7dede2b20314beb930e93f1f6a5bb214efe2d6270b5473b: Status 404 returned error can't find the container with id 89dc12dfa961f4fef7dede2b20314beb930e93f1f6a5bb214efe2d6270b5473b
Sep 16 19:26:06.056495 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:06.056454    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv" event={"ID":"497855df-2858-4915-9e8e-3504b3a6c276","Type":"ContainerStarted","Data":"1482db372dfad57caccffae74041f811b5d36026f55982681f806c610fd6124c"}
Sep 16 19:26:06.056676 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:06.056540    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv" event={"ID":"497855df-2858-4915-9e8e-3504b3a6c276","Type":"ContainerStarted","Data":"89dc12dfa961f4fef7dede2b20314beb930e93f1f6a5bb214efe2d6270b5473b"}
Sep 16 19:26:09.071048 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:09.071013    2586 generic.go:356] "Generic (PLEG): container finished" podID="497855df-2858-4915-9e8e-3504b3a6c276" containerID="1482db372dfad57caccffae74041f811b5d36026f55982681f806c610fd6124c" exitCode=0
Sep 16 19:26:09.071406 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:09.071064    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv" event={"ID":"497855df-2858-4915-9e8e-3504b3a6c276","Type":"ContainerDied","Data":"1482db372dfad57caccffae74041f811b5d36026f55982681f806c610fd6124c"}
Sep 16 19:26:09.134998 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:09.134980    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:26:10.078909 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:10.078875    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv" event={"ID":"497855df-2858-4915-9e8e-3504b3a6c276","Type":"ContainerStarted","Data":"d4f406f81f018ef0bbaf64927e675747966660f85b3f0b0557d5a338a0f0cccc"}
Sep 16 19:26:10.868756 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:10.868709    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6786d494d6-9vddp" podUID="611be502-44e4-4c3c-bfc6-02ac46e40350" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 16 19:26:12.091356 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:12.091258    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv" event={"ID":"497855df-2858-4915-9e8e-3504b3a6c276","Type":"ContainerStarted","Data":"fefc2d3cd74c8d76c0819b89924f5bfc63914b9727aa20215dbc466a490c0cd1"}
Sep 16 19:26:12.091356 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:12.091331    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv" event={"ID":"497855df-2858-4915-9e8e-3504b3a6c276","Type":"ContainerStarted","Data":"0a132ae38b2efa6aa7adc6876f2454b842b8cfd939e0b347ba9e29e4a5e7bc5f"}
Sep 16 19:26:12.091754 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:12.091371    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv"
Sep 16 19:26:12.091754 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:12.091412    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv"
Sep 16 19:26:12.133368 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:12.133294    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv" podStartSLOduration=2.706352923 podStartE2EDuration="9.133271928s" totalImagesPullingTime="2.592417926s" totalInitContainerRuntime="3.834501079s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:26:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:26:12.131099004 +0000 UTC m=+2747.574960579" watchObservedRunningTime="2026-09-16 19:26:12.133271928 +0000 UTC m=+2747.577133504"
Sep 16 19:26:13.095565 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:13.095531    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv"
Sep 16 19:26:19.105223 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:19.105194    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv"
Sep 16 19:26:20.868495 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:20.868453    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6786d494d6-9vddp" podUID="611be502-44e4-4c3c-bfc6-02ac46e40350" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 16 19:26:30.868096 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:30.868048    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6786d494d6-9vddp" podUID="611be502-44e4-4c3c-bfc6-02ac46e40350" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 16 19:26:39.106889 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:39.106846    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv" podUID="497855df-2858-4915-9e8e-3504b3a6c276" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.64:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 16 19:26:40.869303 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:40.869272    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6786d494d6-9vddp"
Sep 16 19:26:41.423394 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:41.423359    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6786d494d6-9vddp"]
Sep 16 19:26:41.423657 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:41.423601    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6786d494d6-9vddp" podUID="611be502-44e4-4c3c-bfc6-02ac46e40350" containerName="kserve-container" containerID="cri-o://c364653a79d5e61ae738082be23e5abe8dd8eb91d8db2be7de997ebe6119389e" gracePeriod=30
Sep 16 19:26:41.423783 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:41.423629    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6786d494d6-9vddp" podUID="611be502-44e4-4c3c-bfc6-02ac46e40350" containerName="kube-rbac-proxy" containerID="cri-o://ce8320311b668c53a8a677f5f473bf0c69de060a4896874400cc5cb9395ced6a" gracePeriod=30
Sep 16 19:26:42.240893 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:42.240858    2586 generic.go:356] "Generic (PLEG): container finished" podID="611be502-44e4-4c3c-bfc6-02ac46e40350" containerID="ce8320311b668c53a8a677f5f473bf0c69de060a4896874400cc5cb9395ced6a" exitCode=2
Sep 16 19:26:42.241397 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:42.240931    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6786d494d6-9vddp" event={"ID":"611be502-44e4-4c3c-bfc6-02ac46e40350","Type":"ContainerDied","Data":"ce8320311b668c53a8a677f5f473bf0c69de060a4896874400cc5cb9395ced6a"}
Sep 16 19:26:44.807697 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:44.807642    2586 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv" podUID="497855df-2858-4915-9e8e-3504b3a6c276" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.64:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 16 19:26:45.862868 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:45.862817    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6786d494d6-9vddp" podUID="611be502-44e4-4c3c-bfc6-02ac46e40350" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.63:8643/healthz\": dial tcp 10.134.0.63:8643: connect: connection refused"
Sep 16 19:26:46.261511 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:46.261483    2586 generic.go:356] "Generic (PLEG): container finished" podID="611be502-44e4-4c3c-bfc6-02ac46e40350" containerID="c364653a79d5e61ae738082be23e5abe8dd8eb91d8db2be7de997ebe6119389e" exitCode=0
Sep 16 19:26:46.261651 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:46.261558    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6786d494d6-9vddp" event={"ID":"611be502-44e4-4c3c-bfc6-02ac46e40350","Type":"ContainerDied","Data":"c364653a79d5e61ae738082be23e5abe8dd8eb91d8db2be7de997ebe6119389e"}
Sep 16 19:26:46.261651 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:46.261641    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6786d494d6-9vddp" event={"ID":"611be502-44e4-4c3c-bfc6-02ac46e40350","Type":"ContainerDied","Data":"59dcac136e53b460fe550dfb48dcd1436e2abc0e0737385baf55f75cc2abe3e9"}
Sep 16 19:26:46.261726 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:46.261678    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59dcac136e53b460fe550dfb48dcd1436e2abc0e0737385baf55f75cc2abe3e9"
Sep 16 19:26:46.271426 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:46.271407    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6786d494d6-9vddp"
Sep 16 19:26:46.408304 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:46.408197    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/611be502-44e4-4c3c-bfc6-02ac46e40350-kube-rbac-proxy-sar-config\") pod \"611be502-44e4-4c3c-bfc6-02ac46e40350\" (UID: \"611be502-44e4-4c3c-bfc6-02ac46e40350\") "
Sep 16 19:26:46.408304 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:46.408271    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/611be502-44e4-4c3c-bfc6-02ac46e40350-kserve-provision-location\") pod \"611be502-44e4-4c3c-bfc6-02ac46e40350\" (UID: \"611be502-44e4-4c3c-bfc6-02ac46e40350\") "
Sep 16 19:26:46.408537 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:46.408325    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6gb4f\" (UniqueName: \"kubernetes.io/projected/611be502-44e4-4c3c-bfc6-02ac46e40350-kube-api-access-6gb4f\") pod \"611be502-44e4-4c3c-bfc6-02ac46e40350\" (UID: \"611be502-44e4-4c3c-bfc6-02ac46e40350\") "
Sep 16 19:26:46.408537 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:46.408354    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/611be502-44e4-4c3c-bfc6-02ac46e40350-proxy-tls\") pod \"611be502-44e4-4c3c-bfc6-02ac46e40350\" (UID: \"611be502-44e4-4c3c-bfc6-02ac46e40350\") "
Sep 16 19:26:46.408637 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:46.408595    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/611be502-44e4-4c3c-bfc6-02ac46e40350-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "611be502-44e4-4c3c-bfc6-02ac46e40350" (UID: "611be502-44e4-4c3c-bfc6-02ac46e40350"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:26:46.408637 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:46.408619    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/611be502-44e4-4c3c-bfc6-02ac46e40350-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "611be502-44e4-4c3c-bfc6-02ac46e40350" (UID: "611be502-44e4-4c3c-bfc6-02ac46e40350"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:26:46.408738 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:46.408671    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/611be502-44e4-4c3c-bfc6-02ac46e40350-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:26:46.408738 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:46.408683    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/611be502-44e4-4c3c-bfc6-02ac46e40350-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:26:46.410487 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:46.410463    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/611be502-44e4-4c3c-bfc6-02ac46e40350-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "611be502-44e4-4c3c-bfc6-02ac46e40350" (UID: "611be502-44e4-4c3c-bfc6-02ac46e40350"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:26:46.410564 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:46.410501    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/611be502-44e4-4c3c-bfc6-02ac46e40350-kube-api-access-6gb4f" (OuterVolumeSpecName: "kube-api-access-6gb4f") pod "611be502-44e4-4c3c-bfc6-02ac46e40350" (UID: "611be502-44e4-4c3c-bfc6-02ac46e40350"). InnerVolumeSpecName "kube-api-access-6gb4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:26:46.509540 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:46.509503    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gb4f\" (UniqueName: \"kubernetes.io/projected/611be502-44e4-4c3c-bfc6-02ac46e40350-kube-api-access-6gb4f\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:26:46.509540 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:46.509533    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/611be502-44e4-4c3c-bfc6-02ac46e40350-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:26:47.266238 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:47.266157    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6786d494d6-9vddp"
Sep 16 19:26:47.292290 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:47.292253    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6786d494d6-9vddp"]
Sep 16 19:26:47.299630 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:47.299604    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6786d494d6-9vddp"]
Sep 16 19:26:49.092552 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:49.092517    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="611be502-44e4-4c3c-bfc6-02ac46e40350" path="/var/lib/kubelet/pods/611be502-44e4-4c3c-bfc6-02ac46e40350/volumes"
Sep 16 19:26:49.109137 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:26:49.109111    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv"
Sep 16 19:27:19.110312 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:27:19.110283    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv"
Sep 16 19:27:23.939785 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:27:23.939746    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv"]
Sep 16 19:27:23.940277 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:27:23.940047    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv" podUID="497855df-2858-4915-9e8e-3504b3a6c276" containerName="kserve-container" containerID="cri-o://d4f406f81f018ef0bbaf64927e675747966660f85b3f0b0557d5a338a0f0cccc" gracePeriod=30
Sep 16 19:27:23.940277 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:27:23.940095    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv" podUID="497855df-2858-4915-9e8e-3504b3a6c276" containerName="kserve-agent" containerID="cri-o://0a132ae38b2efa6aa7adc6876f2454b842b8cfd939e0b347ba9e29e4a5e7bc5f" gracePeriod=30
Sep 16 19:27:23.940277 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:27:23.940104    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv" podUID="497855df-2858-4915-9e8e-3504b3a6c276" containerName="kube-rbac-proxy" containerID="cri-o://fefc2d3cd74c8d76c0819b89924f5bfc63914b9727aa20215dbc466a490c0cd1" gracePeriod=30
Sep 16 19:27:24.100615 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:27:24.100577    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv" podUID="497855df-2858-4915-9e8e-3504b3a6c276" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.64:8643/healthz\": dial tcp 10.134.0.64:8643: connect: connection refused"
Sep 16 19:27:24.435887 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:27:24.435854    2586 generic.go:356] "Generic (PLEG): container finished" podID="497855df-2858-4915-9e8e-3504b3a6c276" containerID="fefc2d3cd74c8d76c0819b89924f5bfc63914b9727aa20215dbc466a490c0cd1" exitCode=2
Sep 16 19:27:24.436068 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:27:24.435905    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv" event={"ID":"497855df-2858-4915-9e8e-3504b3a6c276","Type":"ContainerDied","Data":"fefc2d3cd74c8d76c0819b89924f5bfc63914b9727aa20215dbc466a490c0cd1"}
Sep 16 19:27:26.454133 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:27:26.454042    2586 generic.go:356] "Generic (PLEG): container finished" podID="497855df-2858-4915-9e8e-3504b3a6c276" containerID="d4f406f81f018ef0bbaf64927e675747966660f85b3f0b0557d5a338a0f0cccc" exitCode=0
Sep 16 19:27:26.454133 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:27:26.454112    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv" event={"ID":"497855df-2858-4915-9e8e-3504b3a6c276","Type":"ContainerDied","Data":"d4f406f81f018ef0bbaf64927e675747966660f85b3f0b0557d5a338a0f0cccc"}
Sep 16 19:27:29.100476 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:27:29.100435    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv" podUID="497855df-2858-4915-9e8e-3504b3a6c276" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.64:8643/healthz\": dial tcp 10.134.0.64:8643: connect: connection refused"
Sep 16 19:27:29.105980 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:27:29.105941    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv" podUID="497855df-2858-4915-9e8e-3504b3a6c276" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.64:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 16 19:27:34.100595 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:27:34.100545    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv" podUID="497855df-2858-4915-9e8e-3504b3a6c276" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.64:8643/healthz\": dial tcp 10.134.0.64:8643: connect: connection refused"
Sep 16 19:27:34.101033 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:27:34.100686    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv"
Sep 16 19:27:39.100385 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:27:39.100345    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv" podUID="497855df-2858-4915-9e8e-3504b3a6c276" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.64:8643/healthz\": dial tcp 10.134.0.64:8643: connect: connection refused"
Sep 16 19:27:39.105768 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:27:39.105740    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv" podUID="497855df-2858-4915-9e8e-3504b3a6c276" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.64:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 16 19:27:44.100138 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:27:44.100094    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv" podUID="497855df-2858-4915-9e8e-3504b3a6c276" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.64:8643/healthz\": dial tcp 10.134.0.64:8643: connect: connection refused"
Sep 16 19:27:49.100654 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:27:49.100607    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv" podUID="497855df-2858-4915-9e8e-3504b3a6c276" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.64:8643/healthz\": dial tcp 10.134.0.64:8643: connect: connection refused"
Sep 16 19:27:49.106104 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:27:49.106074    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv" podUID="497855df-2858-4915-9e8e-3504b3a6c276" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.64:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 16 19:27:49.106224 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:27:49.106187    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv"
Sep 16 19:27:54.126370 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:27:54.126348    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv"
Sep 16 19:27:54.271668 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:27:54.271578    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/497855df-2858-4915-9e8e-3504b3a6c276-kserve-provision-location\") pod \"497855df-2858-4915-9e8e-3504b3a6c276\" (UID: \"497855df-2858-4915-9e8e-3504b3a6c276\") "
Sep 16 19:27:54.271668 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:27:54.271630    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/497855df-2858-4915-9e8e-3504b3a6c276-proxy-tls\") pod \"497855df-2858-4915-9e8e-3504b3a6c276\" (UID: \"497855df-2858-4915-9e8e-3504b3a6c276\") "
Sep 16 19:27:54.271855 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:27:54.271690    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/497855df-2858-4915-9e8e-3504b3a6c276-kube-rbac-proxy-sar-config\") pod \"497855df-2858-4915-9e8e-3504b3a6c276\" (UID: \"497855df-2858-4915-9e8e-3504b3a6c276\") "
Sep 16 19:27:54.271855 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:27:54.271744    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-clcxn\" (UniqueName: \"kubernetes.io/projected/497855df-2858-4915-9e8e-3504b3a6c276-kube-api-access-clcxn\") pod \"497855df-2858-4915-9e8e-3504b3a6c276\" (UID: \"497855df-2858-4915-9e8e-3504b3a6c276\") "
Sep 16 19:27:54.271855 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:27:54.271778    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/497855df-2858-4915-9e8e-3504b3a6c276-cabundle-cert\") pod \"497855df-2858-4915-9e8e-3504b3a6c276\" (UID: \"497855df-2858-4915-9e8e-3504b3a6c276\") "
Sep 16 19:27:54.271976 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:27:54.271868    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/497855df-2858-4915-9e8e-3504b3a6c276-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "497855df-2858-4915-9e8e-3504b3a6c276" (UID: "497855df-2858-4915-9e8e-3504b3a6c276"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:27:54.272048 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:27:54.272029    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/497855df-2858-4915-9e8e-3504b3a6c276-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:27:54.272167 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:27:54.272141    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/497855df-2858-4915-9e8e-3504b3a6c276-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "497855df-2858-4915-9e8e-3504b3a6c276" (UID: "497855df-2858-4915-9e8e-3504b3a6c276"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:27:54.272247 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:27:54.272188    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/497855df-2858-4915-9e8e-3504b3a6c276-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "497855df-2858-4915-9e8e-3504b3a6c276" (UID: "497855df-2858-4915-9e8e-3504b3a6c276"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:27:54.273665 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:27:54.273649    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/497855df-2858-4915-9e8e-3504b3a6c276-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "497855df-2858-4915-9e8e-3504b3a6c276" (UID: "497855df-2858-4915-9e8e-3504b3a6c276"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:27:54.273985 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:27:54.273954    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/497855df-2858-4915-9e8e-3504b3a6c276-kube-api-access-clcxn" (OuterVolumeSpecName: "kube-api-access-clcxn") pod "497855df-2858-4915-9e8e-3504b3a6c276" (UID: "497855df-2858-4915-9e8e-3504b3a6c276"). InnerVolumeSpecName "kube-api-access-clcxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:27:54.372788 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:27:54.372738    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clcxn\" (UniqueName: \"kubernetes.io/projected/497855df-2858-4915-9e8e-3504b3a6c276-kube-api-access-clcxn\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:27:54.372788 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:27:54.372782    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/497855df-2858-4915-9e8e-3504b3a6c276-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:27:54.372788 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:27:54.372794    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/497855df-2858-4915-9e8e-3504b3a6c276-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:27:54.372788 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:27:54.372802    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/497855df-2858-4915-9e8e-3504b3a6c276-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:27:54.585450 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:27:54.585361    2586 generic.go:356] "Generic (PLEG): container finished" podID="497855df-2858-4915-9e8e-3504b3a6c276" containerID="0a132ae38b2efa6aa7adc6876f2454b842b8cfd939e0b347ba9e29e4a5e7bc5f" exitCode=137
Sep 16 19:27:54.585450 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:27:54.585422    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv"
Sep 16 19:27:54.585621 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:27:54.585448    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv" event={"ID":"497855df-2858-4915-9e8e-3504b3a6c276","Type":"ContainerDied","Data":"0a132ae38b2efa6aa7adc6876f2454b842b8cfd939e0b347ba9e29e4a5e7bc5f"}
Sep 16 19:27:54.585621 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:27:54.585514    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv" event={"ID":"497855df-2858-4915-9e8e-3504b3a6c276","Type":"ContainerDied","Data":"89dc12dfa961f4fef7dede2b20314beb930e93f1f6a5bb214efe2d6270b5473b"}
Sep 16 19:27:54.585621 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:27:54.585547    2586 scope.go:117] "RemoveContainer" containerID="fefc2d3cd74c8d76c0819b89924f5bfc63914b9727aa20215dbc466a490c0cd1"
Sep 16 19:27:54.598481 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:27:54.598464    2586 scope.go:117] "RemoveContainer" containerID="0a132ae38b2efa6aa7adc6876f2454b842b8cfd939e0b347ba9e29e4a5e7bc5f"
Sep 16 19:27:54.606013 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:27:54.605998    2586 scope.go:117] "RemoveContainer" containerID="d4f406f81f018ef0bbaf64927e675747966660f85b3f0b0557d5a338a0f0cccc"
Sep 16 19:27:54.613377 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:27:54.613359    2586 scope.go:117] "RemoveContainer" containerID="1482db372dfad57caccffae74041f811b5d36026f55982681f806c610fd6124c"
Sep 16 19:27:54.620808 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:27:54.620791    2586 scope.go:117] "RemoveContainer" containerID="fefc2d3cd74c8d76c0819b89924f5bfc63914b9727aa20215dbc466a490c0cd1"
Sep 16 19:27:54.621073 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:27:54.621052    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fefc2d3cd74c8d76c0819b89924f5bfc63914b9727aa20215dbc466a490c0cd1\": container with ID starting with fefc2d3cd74c8d76c0819b89924f5bfc63914b9727aa20215dbc466a490c0cd1 not found: ID does not exist" containerID="fefc2d3cd74c8d76c0819b89924f5bfc63914b9727aa20215dbc466a490c0cd1"
Sep 16 19:27:54.621160 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:27:54.621085    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fefc2d3cd74c8d76c0819b89924f5bfc63914b9727aa20215dbc466a490c0cd1"} err="failed to get container status \"fefc2d3cd74c8d76c0819b89924f5bfc63914b9727aa20215dbc466a490c0cd1\": rpc error: code = NotFound desc = could not find container \"fefc2d3cd74c8d76c0819b89924f5bfc63914b9727aa20215dbc466a490c0cd1\": container with ID starting with fefc2d3cd74c8d76c0819b89924f5bfc63914b9727aa20215dbc466a490c0cd1 not found: ID does not exist"
Sep 16 19:27:54.621160 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:27:54.621111    2586 scope.go:117] "RemoveContainer" containerID="0a132ae38b2efa6aa7adc6876f2454b842b8cfd939e0b347ba9e29e4a5e7bc5f"
Sep 16 19:27:54.621458 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:27:54.621430    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a132ae38b2efa6aa7adc6876f2454b842b8cfd939e0b347ba9e29e4a5e7bc5f\": container with ID starting with 0a132ae38b2efa6aa7adc6876f2454b842b8cfd939e0b347ba9e29e4a5e7bc5f not found: ID does not exist" containerID="0a132ae38b2efa6aa7adc6876f2454b842b8cfd939e0b347ba9e29e4a5e7bc5f"
Sep 16 19:27:54.621604 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:27:54.621464    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a132ae38b2efa6aa7adc6876f2454b842b8cfd939e0b347ba9e29e4a5e7bc5f"} err="failed to get container status \"0a132ae38b2efa6aa7adc6876f2454b842b8cfd939e0b347ba9e29e4a5e7bc5f\": rpc error: code = NotFound desc = could not find container \"0a132ae38b2efa6aa7adc6876f2454b842b8cfd939e0b347ba9e29e4a5e7bc5f\": container with ID starting with 0a132ae38b2efa6aa7adc6876f2454b842b8cfd939e0b347ba9e29e4a5e7bc5f not found: ID does not exist"
Sep 16 19:27:54.621604 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:27:54.621487    2586 scope.go:117] "RemoveContainer" containerID="d4f406f81f018ef0bbaf64927e675747966660f85b3f0b0557d5a338a0f0cccc"
Sep 16 19:27:54.621879 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:27:54.621850    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4f406f81f018ef0bbaf64927e675747966660f85b3f0b0557d5a338a0f0cccc\": container with ID starting with d4f406f81f018ef0bbaf64927e675747966660f85b3f0b0557d5a338a0f0cccc not found: ID does not exist" containerID="d4f406f81f018ef0bbaf64927e675747966660f85b3f0b0557d5a338a0f0cccc"
Sep 16 19:27:54.621975 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:27:54.621884    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4f406f81f018ef0bbaf64927e675747966660f85b3f0b0557d5a338a0f0cccc"} err="failed to get container status \"d4f406f81f018ef0bbaf64927e675747966660f85b3f0b0557d5a338a0f0cccc\": rpc error: code = NotFound desc = could not find container \"d4f406f81f018ef0bbaf64927e675747966660f85b3f0b0557d5a338a0f0cccc\": container with ID starting with d4f406f81f018ef0bbaf64927e675747966660f85b3f0b0557d5a338a0f0cccc not found: ID does not exist"
Sep 16 19:27:54.621975 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:27:54.621899    2586 scope.go:117] "RemoveContainer" containerID="1482db372dfad57caccffae74041f811b5d36026f55982681f806c610fd6124c"
Sep 16 19:27:54.622190 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:27:54.622172    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1482db372dfad57caccffae74041f811b5d36026f55982681f806c610fd6124c\": container with ID starting with 1482db372dfad57caccffae74041f811b5d36026f55982681f806c610fd6124c not found: ID does not exist" containerID="1482db372dfad57caccffae74041f811b5d36026f55982681f806c610fd6124c"
Sep 16 19:27:54.622337 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:27:54.622195    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1482db372dfad57caccffae74041f811b5d36026f55982681f806c610fd6124c"} err="failed to get container status \"1482db372dfad57caccffae74041f811b5d36026f55982681f806c610fd6124c\": rpc error: code = NotFound desc = could not find container \"1482db372dfad57caccffae74041f811b5d36026f55982681f806c610fd6124c\": container with ID starting with 1482db372dfad57caccffae74041f811b5d36026f55982681f806c610fd6124c not found: ID does not exist"
Sep 16 19:27:54.623508 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:27:54.623487    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv"]
Sep 16 19:27:54.626407 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:27:54.626386    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv"]
Sep 16 19:27:55.092772 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:27:55.092731    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="497855df-2858-4915-9e8e-3504b3a6c276" path="/var/lib/kubelet/pods/497855df-2858-4915-9e8e-3504b3a6c276/volumes"
Sep 16 19:27:55.100915 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:27:55.100881    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-76d4c69869-4jtdv" podUID="497855df-2858-4915-9e8e-3504b3a6c276" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.64:8643/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 16 19:28:02.788955 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:02.788916    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br"]
Sep 16 19:28:02.789376 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:02.789283    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="497855df-2858-4915-9e8e-3504b3a6c276" containerName="storage-initializer"
Sep 16 19:28:02.789376 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:02.789298    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="497855df-2858-4915-9e8e-3504b3a6c276" containerName="storage-initializer"
Sep 16 19:28:02.789376 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:02.789306    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="611be502-44e4-4c3c-bfc6-02ac46e40350" containerName="storage-initializer"
Sep 16 19:28:02.789376 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:02.789313    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="611be502-44e4-4c3c-bfc6-02ac46e40350" containerName="storage-initializer"
Sep 16 19:28:02.789376 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:02.789320    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="497855df-2858-4915-9e8e-3504b3a6c276" containerName="kserve-container"
Sep 16 19:28:02.789376 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:02.789326    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="497855df-2858-4915-9e8e-3504b3a6c276" containerName="kserve-container"
Sep 16 19:28:02.789376 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:02.789335    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="497855df-2858-4915-9e8e-3504b3a6c276" containerName="kserve-agent"
Sep 16 19:28:02.789376 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:02.789341    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="497855df-2858-4915-9e8e-3504b3a6c276" containerName="kserve-agent"
Sep 16 19:28:02.789376 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:02.789348    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="611be502-44e4-4c3c-bfc6-02ac46e40350" containerName="kserve-container"
Sep 16 19:28:02.789376 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:02.789352    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="611be502-44e4-4c3c-bfc6-02ac46e40350" containerName="kserve-container"
Sep 16 19:28:02.789376 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:02.789360    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="611be502-44e4-4c3c-bfc6-02ac46e40350" containerName="kube-rbac-proxy"
Sep 16 19:28:02.789376 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:02.789365    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="611be502-44e4-4c3c-bfc6-02ac46e40350" containerName="kube-rbac-proxy"
Sep 16 19:28:02.789376 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:02.789379    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="497855df-2858-4915-9e8e-3504b3a6c276" containerName="kube-rbac-proxy"
Sep 16 19:28:02.789376 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:02.789384    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="497855df-2858-4915-9e8e-3504b3a6c276" containerName="kube-rbac-proxy"
Sep 16 19:28:02.789808 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:02.789439    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="611be502-44e4-4c3c-bfc6-02ac46e40350" containerName="kube-rbac-proxy"
Sep 16 19:28:02.789808 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:02.789449    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="497855df-2858-4915-9e8e-3504b3a6c276" containerName="kserve-agent"
Sep 16 19:28:02.789808 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:02.789455    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="611be502-44e4-4c3c-bfc6-02ac46e40350" containerName="kserve-container"
Sep 16 19:28:02.789808 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:02.789462    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="497855df-2858-4915-9e8e-3504b3a6c276" containerName="kserve-container"
Sep 16 19:28:02.789808 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:02.789467    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="497855df-2858-4915-9e8e-3504b3a6c276" containerName="kube-rbac-proxy"
Sep 16 19:28:02.794255 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:02.794217    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br"
Sep 16 19:28:02.797843 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:02.797817    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
Sep 16 19:28:02.798064 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:02.798034    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 16 19:28:02.798759 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:02.798744    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
Sep 16 19:28:02.799434 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:02.799411    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:28:02.802112 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:02.802092    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-pv9ct\""
Sep 16 19:28:02.804927 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:02.804911    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:28:02.824016 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:02.823984    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br"]
Sep 16 19:28:02.844790 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:02.844760    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9011415f-d141-449b-b542-10629dfd62ba-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br\" (UID: \"9011415f-d141-449b-b542-10629dfd62ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br"
Sep 16 19:28:02.844933 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:02.844795    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t28cr\" (UniqueName: \"kubernetes.io/projected/9011415f-d141-449b-b542-10629dfd62ba-kube-api-access-t28cr\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br\" (UID: \"9011415f-d141-449b-b542-10629dfd62ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br"
Sep 16 19:28:02.844933 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:02.844822    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9011415f-d141-449b-b542-10629dfd62ba-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br\" (UID: \"9011415f-d141-449b-b542-10629dfd62ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br"
Sep 16 19:28:02.844933 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:02.844894    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9011415f-d141-449b-b542-10629dfd62ba-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br\" (UID: \"9011415f-d141-449b-b542-10629dfd62ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br"
Sep 16 19:28:02.945345 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:02.945304    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9011415f-d141-449b-b542-10629dfd62ba-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br\" (UID: \"9011415f-d141-449b-b542-10629dfd62ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br"
Sep 16 19:28:02.945518 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:02.945354    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t28cr\" (UniqueName: \"kubernetes.io/projected/9011415f-d141-449b-b542-10629dfd62ba-kube-api-access-t28cr\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br\" (UID: \"9011415f-d141-449b-b542-10629dfd62ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br"
Sep 16 19:28:02.945518 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:02.945390    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9011415f-d141-449b-b542-10629dfd62ba-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br\" (UID: \"9011415f-d141-449b-b542-10629dfd62ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br"
Sep 16 19:28:02.945518 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:02.945429    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9011415f-d141-449b-b542-10629dfd62ba-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br\" (UID: \"9011415f-d141-449b-b542-10629dfd62ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br"
Sep 16 19:28:02.945886 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:02.945850    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9011415f-d141-449b-b542-10629dfd62ba-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br\" (UID: \"9011415f-d141-449b-b542-10629dfd62ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br"
Sep 16 19:28:02.946114 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:02.946093    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9011415f-d141-449b-b542-10629dfd62ba-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br\" (UID: \"9011415f-d141-449b-b542-10629dfd62ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br"
Sep 16 19:28:02.948212 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:02.948189    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9011415f-d141-449b-b542-10629dfd62ba-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br\" (UID: \"9011415f-d141-449b-b542-10629dfd62ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br"
Sep 16 19:28:02.958426 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:02.958398    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t28cr\" (UniqueName: \"kubernetes.io/projected/9011415f-d141-449b-b542-10629dfd62ba-kube-api-access-t28cr\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br\" (UID: \"9011415f-d141-449b-b542-10629dfd62ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br"
Sep 16 19:28:03.104835 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:03.104803    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br"
Sep 16 19:28:03.383831 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:03.383732    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br"]
Sep 16 19:28:03.388022 ip-10-0-129-190 kubenswrapper[2586]: W0916 19:28:03.387996    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9011415f_d141_449b_b542_10629dfd62ba.slice/crio-89c0e4976a5f3d7888ca895149ddf032139da5614a4c5bac5b5c2f2eea288e01 WatchSource:0}: Error finding container 89c0e4976a5f3d7888ca895149ddf032139da5614a4c5bac5b5c2f2eea288e01: Status 404 returned error can't find the container with id 89c0e4976a5f3d7888ca895149ddf032139da5614a4c5bac5b5c2f2eea288e01
Sep 16 19:28:03.627974 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:03.627934    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br" event={"ID":"9011415f-d141-449b-b542-10629dfd62ba","Type":"ContainerStarted","Data":"1ec9054d793afb260937c2d1020cfd5a2e74efabc568e64548eb0435fb8b9ae9"}
Sep 16 19:28:03.628140 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:03.628009    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br" event={"ID":"9011415f-d141-449b-b542-10629dfd62ba","Type":"ContainerStarted","Data":"89c0e4976a5f3d7888ca895149ddf032139da5614a4c5bac5b5c2f2eea288e01"}
Sep 16 19:28:07.647509 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:07.647477    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br_9011415f-d141-449b-b542-10629dfd62ba/storage-initializer/0.log"
Sep 16 19:28:07.647894 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:07.647516    2586 generic.go:356] "Generic (PLEG): container finished" podID="9011415f-d141-449b-b542-10629dfd62ba" containerID="1ec9054d793afb260937c2d1020cfd5a2e74efabc568e64548eb0435fb8b9ae9" exitCode=1
Sep 16 19:28:07.647894 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:07.647583    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br" event={"ID":"9011415f-d141-449b-b542-10629dfd62ba","Type":"ContainerDied","Data":"1ec9054d793afb260937c2d1020cfd5a2e74efabc568e64548eb0435fb8b9ae9"}
Sep 16 19:28:08.653823 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:08.653795    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br_9011415f-d141-449b-b542-10629dfd62ba/storage-initializer/0.log"
Sep 16 19:28:08.654210 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:08.653905    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br" event={"ID":"9011415f-d141-449b-b542-10629dfd62ba","Type":"ContainerStarted","Data":"c4dec1a758f99239b9949463dc6fd5df1039bb6df39981b29ee912093c5e47a0"}
Sep 16 19:28:10.663818 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:10.663787    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br_9011415f-d141-449b-b542-10629dfd62ba/storage-initializer/1.log"
Sep 16 19:28:10.664207 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:10.664190    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br_9011415f-d141-449b-b542-10629dfd62ba/storage-initializer/0.log"
Sep 16 19:28:10.664280 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:10.664223    2586 generic.go:356] "Generic (PLEG): container finished" podID="9011415f-d141-449b-b542-10629dfd62ba" containerID="c4dec1a758f99239b9949463dc6fd5df1039bb6df39981b29ee912093c5e47a0" exitCode=1
Sep 16 19:28:10.664321 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:10.664266    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br" event={"ID":"9011415f-d141-449b-b542-10629dfd62ba","Type":"ContainerDied","Data":"c4dec1a758f99239b9949463dc6fd5df1039bb6df39981b29ee912093c5e47a0"}
Sep 16 19:28:10.664365 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:10.664336    2586 scope.go:117] "RemoveContainer" containerID="1ec9054d793afb260937c2d1020cfd5a2e74efabc568e64548eb0435fb8b9ae9"
Sep 16 19:28:10.664724 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:10.664700    2586 scope.go:117] "RemoveContainer" containerID="1ec9054d793afb260937c2d1020cfd5a2e74efabc568e64548eb0435fb8b9ae9"
Sep 16 19:28:10.675831 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:28:10.675802    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br_kserve-ci-e2e-test(9011415f-d141-449b-b542-10629dfd62ba)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br" podUID="9011415f-d141-449b-b542-10629dfd62ba"
Sep 16 19:28:10.675831 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:28:10.675800    2586 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br_kserve-ci-e2e-test_9011415f-d141-449b-b542-10629dfd62ba_0 in pod sandbox 89c0e4976a5f3d7888ca895149ddf032139da5614a4c5bac5b5c2f2eea288e01 from index: no such id: '1ec9054d793afb260937c2d1020cfd5a2e74efabc568e64548eb0435fb8b9ae9'" containerID="1ec9054d793afb260937c2d1020cfd5a2e74efabc568e64548eb0435fb8b9ae9"
Sep 16 19:28:10.676022 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:10.675845    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ec9054d793afb260937c2d1020cfd5a2e74efabc568e64548eb0435fb8b9ae9"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br_kserve-ci-e2e-test_9011415f-d141-449b-b542-10629dfd62ba_0 in pod sandbox 89c0e4976a5f3d7888ca895149ddf032139da5614a4c5bac5b5c2f2eea288e01 from index: no such id: '1ec9054d793afb260937c2d1020cfd5a2e74efabc568e64548eb0435fb8b9ae9'"
Sep 16 19:28:11.670558 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:11.670533    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br_9011415f-d141-449b-b542-10629dfd62ba/storage-initializer/1.log"
Sep 16 19:28:12.753355 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:12.753321    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br"]
Sep 16 19:28:12.891075 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:12.891052    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br_9011415f-d141-449b-b542-10629dfd62ba/storage-initializer/1.log"
Sep 16 19:28:12.894487 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:12.894469    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br"
Sep 16 19:28:12.924813 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:12.924783    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9011415f-d141-449b-b542-10629dfd62ba-proxy-tls\") pod \"9011415f-d141-449b-b542-10629dfd62ba\" (UID: \"9011415f-d141-449b-b542-10629dfd62ba\") "
Sep 16 19:28:12.924988 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:12.924834    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9011415f-d141-449b-b542-10629dfd62ba-kserve-provision-location\") pod \"9011415f-d141-449b-b542-10629dfd62ba\" (UID: \"9011415f-d141-449b-b542-10629dfd62ba\") "
Sep 16 19:28:12.924988 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:12.924862    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9011415f-d141-449b-b542-10629dfd62ba-kube-rbac-proxy-sar-config\") pod \"9011415f-d141-449b-b542-10629dfd62ba\" (UID: \"9011415f-d141-449b-b542-10629dfd62ba\") "
Sep 16 19:28:12.924988 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:12.924881    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t28cr\" (UniqueName: \"kubernetes.io/projected/9011415f-d141-449b-b542-10629dfd62ba-kube-api-access-t28cr\") pod \"9011415f-d141-449b-b542-10629dfd62ba\" (UID: \"9011415f-d141-449b-b542-10629dfd62ba\") "
Sep 16 19:28:12.925172 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:12.925144    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9011415f-d141-449b-b542-10629dfd62ba-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9011415f-d141-449b-b542-10629dfd62ba" (UID: "9011415f-d141-449b-b542-10629dfd62ba"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:28:12.925261 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:12.925244    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9011415f-d141-449b-b542-10629dfd62ba-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9011415f-d141-449b-b542-10629dfd62ba" (UID: "9011415f-d141-449b-b542-10629dfd62ba"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:28:12.926996 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:12.926967    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9011415f-d141-449b-b542-10629dfd62ba-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9011415f-d141-449b-b542-10629dfd62ba" (UID: "9011415f-d141-449b-b542-10629dfd62ba"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:28:12.927099 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:12.927016    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9011415f-d141-449b-b542-10629dfd62ba-kube-api-access-t28cr" (OuterVolumeSpecName: "kube-api-access-t28cr") pod "9011415f-d141-449b-b542-10629dfd62ba" (UID: "9011415f-d141-449b-b542-10629dfd62ba"). InnerVolumeSpecName "kube-api-access-t28cr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:28:13.025496 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:13.025405    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9011415f-d141-449b-b542-10629dfd62ba-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:28:13.025496 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:13.025438    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t28cr\" (UniqueName: \"kubernetes.io/projected/9011415f-d141-449b-b542-10629dfd62ba-kube-api-access-t28cr\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:28:13.025496 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:13.025449    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9011415f-d141-449b-b542-10629dfd62ba-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:28:13.025496 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:13.025458    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9011415f-d141-449b-b542-10629dfd62ba-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:28:13.685048 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:13.685005    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br_9011415f-d141-449b-b542-10629dfd62ba/storage-initializer/1.log"
Sep 16 19:28:13.685248 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:13.685113    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br"
Sep 16 19:28:13.685248 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:13.685135    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br" event={"ID":"9011415f-d141-449b-b542-10629dfd62ba","Type":"ContainerDied","Data":"89c0e4976a5f3d7888ca895149ddf032139da5614a4c5bac5b5c2f2eea288e01"}
Sep 16 19:28:13.685248 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:13.685196    2586 scope.go:117] "RemoveContainer" containerID="c4dec1a758f99239b9949463dc6fd5df1039bb6df39981b29ee912093c5e47a0"
Sep 16 19:28:13.747294 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:13.744568    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br"]
Sep 16 19:28:13.752411 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:13.752384    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7bbb8cdc9d-zt8br"]
Sep 16 19:28:13.984210 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:13.984128    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c"]
Sep 16 19:28:13.984567 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:13.984500    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9011415f-d141-449b-b542-10629dfd62ba" containerName="storage-initializer"
Sep 16 19:28:13.984567 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:13.984512    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="9011415f-d141-449b-b542-10629dfd62ba" containerName="storage-initializer"
Sep 16 19:28:13.984567 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:13.984521    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9011415f-d141-449b-b542-10629dfd62ba" containerName="storage-initializer"
Sep 16 19:28:13.984567 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:13.984527    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="9011415f-d141-449b-b542-10629dfd62ba" containerName="storage-initializer"
Sep 16 19:28:13.984711 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:13.984574    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="9011415f-d141-449b-b542-10629dfd62ba" containerName="storage-initializer"
Sep 16 19:28:13.984711 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:13.984582    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="9011415f-d141-449b-b542-10629dfd62ba" containerName="storage-initializer"
Sep 16 19:28:13.989060 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:13.988901    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c"
Sep 16 19:28:13.994008 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:13.993981    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-pv9ct\""
Sep 16 19:28:13.994136 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:13.994020    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
Sep 16 19:28:13.994136 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:13.994054    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:28:13.994136 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:13.994070    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:28:13.994136 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:13.994021    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:28:13.994136 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:13.993981    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
Sep 16 19:28:13.994136 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:13.994027    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 16 19:28:14.008420 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:14.008395    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c"]
Sep 16 19:28:14.034583 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:14.034557    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39d1f358-7083-4b19-800e-ca0bbce8e754-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c\" (UID: \"39d1f358-7083-4b19-800e-ca0bbce8e754\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c"
Sep 16 19:28:14.034702 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:14.034602    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39d1f358-7083-4b19-800e-ca0bbce8e754-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c\" (UID: \"39d1f358-7083-4b19-800e-ca0bbce8e754\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c"
Sep 16 19:28:14.034702 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:14.034633    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39d1f358-7083-4b19-800e-ca0bbce8e754-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c\" (UID: \"39d1f358-7083-4b19-800e-ca0bbce8e754\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c"
Sep 16 19:28:14.034827 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:14.034700    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39d1f358-7083-4b19-800e-ca0bbce8e754-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c\" (UID: \"39d1f358-7083-4b19-800e-ca0bbce8e754\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c"
Sep 16 19:28:14.034827 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:14.034730    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph64m\" (UniqueName: \"kubernetes.io/projected/39d1f358-7083-4b19-800e-ca0bbce8e754-kube-api-access-ph64m\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c\" (UID: \"39d1f358-7083-4b19-800e-ca0bbce8e754\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c"
Sep 16 19:28:14.135508 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:14.135475    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39d1f358-7083-4b19-800e-ca0bbce8e754-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c\" (UID: \"39d1f358-7083-4b19-800e-ca0bbce8e754\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c"
Sep 16 19:28:14.135719 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:14.135534    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39d1f358-7083-4b19-800e-ca0bbce8e754-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c\" (UID: \"39d1f358-7083-4b19-800e-ca0bbce8e754\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c"
Sep 16 19:28:14.135719 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:14.135572    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39d1f358-7083-4b19-800e-ca0bbce8e754-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c\" (UID: \"39d1f358-7083-4b19-800e-ca0bbce8e754\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c"
Sep 16 19:28:14.135719 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:14.135637    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39d1f358-7083-4b19-800e-ca0bbce8e754-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c\" (UID: \"39d1f358-7083-4b19-800e-ca0bbce8e754\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c"
Sep 16 19:28:14.135719 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:14.135679    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph64m\" (UniqueName: \"kubernetes.io/projected/39d1f358-7083-4b19-800e-ca0bbce8e754-kube-api-access-ph64m\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c\" (UID: \"39d1f358-7083-4b19-800e-ca0bbce8e754\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c"
Sep 16 19:28:14.135988 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:28:14.135747    2586 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found
Sep 16 19:28:14.135988 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:28:14.135842    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39d1f358-7083-4b19-800e-ca0bbce8e754-proxy-tls podName:39d1f358-7083-4b19-800e-ca0bbce8e754 nodeName:}" failed. No retries permitted until 2026-09-16 19:28:14.635818055 +0000 UTC m=+2870.079679610 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/39d1f358-7083-4b19-800e-ca0bbce8e754-proxy-tls") pod "isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c" (UID: "39d1f358-7083-4b19-800e-ca0bbce8e754") : secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found
Sep 16 19:28:14.136155 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:14.136026    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39d1f358-7083-4b19-800e-ca0bbce8e754-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c\" (UID: \"39d1f358-7083-4b19-800e-ca0bbce8e754\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c"
Sep 16 19:28:14.136276 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:14.136255    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39d1f358-7083-4b19-800e-ca0bbce8e754-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c\" (UID: \"39d1f358-7083-4b19-800e-ca0bbce8e754\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c"
Sep 16 19:28:14.136383 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:14.136363    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39d1f358-7083-4b19-800e-ca0bbce8e754-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c\" (UID: \"39d1f358-7083-4b19-800e-ca0bbce8e754\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c"
Sep 16 19:28:14.147262 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:14.147218    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph64m\" (UniqueName: \"kubernetes.io/projected/39d1f358-7083-4b19-800e-ca0bbce8e754-kube-api-access-ph64m\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c\" (UID: \"39d1f358-7083-4b19-800e-ca0bbce8e754\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c"
Sep 16 19:28:14.640247 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:14.640195    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39d1f358-7083-4b19-800e-ca0bbce8e754-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c\" (UID: \"39d1f358-7083-4b19-800e-ca0bbce8e754\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c"
Sep 16 19:28:14.642624 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:14.642604    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39d1f358-7083-4b19-800e-ca0bbce8e754-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c\" (UID: \"39d1f358-7083-4b19-800e-ca0bbce8e754\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c"
Sep 16 19:28:14.899690 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:14.899597    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c"
Sep 16 19:28:15.092986 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:15.092949    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9011415f-d141-449b-b542-10629dfd62ba" path="/var/lib/kubelet/pods/9011415f-d141-449b-b542-10629dfd62ba/volumes"
Sep 16 19:28:15.178657 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:15.178633    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c"]
Sep 16 19:28:15.180500 ip-10-0-129-190 kubenswrapper[2586]: W0916 19:28:15.180469    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39d1f358_7083_4b19_800e_ca0bbce8e754.slice/crio-b442bf30879881a927e437e425c699c3426db9076f880f4f5fe703d27f0bf8ea WatchSource:0}: Error finding container b442bf30879881a927e437e425c699c3426db9076f880f4f5fe703d27f0bf8ea: Status 404 returned error can't find the container with id b442bf30879881a927e437e425c699c3426db9076f880f4f5fe703d27f0bf8ea
Sep 16 19:28:15.697255 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:15.697193    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c" event={"ID":"39d1f358-7083-4b19-800e-ca0bbce8e754","Type":"ContainerStarted","Data":"fa923b29ff74939e71bc03394f1ede583839ad51f924633ff5f1fcf10de67009"}
Sep 16 19:28:15.697456 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:15.697275    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c" event={"ID":"39d1f358-7083-4b19-800e-ca0bbce8e754","Type":"ContainerStarted","Data":"b442bf30879881a927e437e425c699c3426db9076f880f4f5fe703d27f0bf8ea"}
Sep 16 19:28:16.703275 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:16.703241    2586 generic.go:356] "Generic (PLEG): container finished" podID="39d1f358-7083-4b19-800e-ca0bbce8e754" containerID="fa923b29ff74939e71bc03394f1ede583839ad51f924633ff5f1fcf10de67009" exitCode=0
Sep 16 19:28:16.703690 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:16.703326    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c" event={"ID":"39d1f358-7083-4b19-800e-ca0bbce8e754","Type":"ContainerDied","Data":"fa923b29ff74939e71bc03394f1ede583839ad51f924633ff5f1fcf10de67009"}
Sep 16 19:28:17.710479 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:17.710384    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c" event={"ID":"39d1f358-7083-4b19-800e-ca0bbce8e754","Type":"ContainerStarted","Data":"6e503402955af45602a8f5712e45edb7a8fa5cc991ded32f61b94197158dfeeb"}
Sep 16 19:28:17.710479 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:17.710448    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c" event={"ID":"39d1f358-7083-4b19-800e-ca0bbce8e754","Type":"ContainerStarted","Data":"3b10ae1b09b6156ce9dfa186f3b8e9f2bdecfe1334deb2dd57212fbef833e8af"}
Sep 16 19:28:17.710907 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:17.710583    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c"
Sep 16 19:28:18.715223 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:18.715185    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c"
Sep 16 19:28:18.716668 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:18.716639    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c" podUID="39d1f358-7083-4b19-800e-ca0bbce8e754" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 16 19:28:19.720293 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:19.720251    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c" podUID="39d1f358-7083-4b19-800e-ca0bbce8e754" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 16 19:28:24.725222 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:24.725190    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c"
Sep 16 19:28:24.725872 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:24.725845    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c" podUID="39d1f358-7083-4b19-800e-ca0bbce8e754" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 16 19:28:24.748534 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:24.748481    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c" podStartSLOduration=10.701559236 podStartE2EDuration="11.748465169s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.046905933s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:28:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:28:17.741919491 +0000 UTC m=+2873.185781067" watchObservedRunningTime="2026-09-16 19:28:24.748465169 +0000 UTC m=+2880.192326815"
Sep 16 19:28:34.725778 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:34.725738    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c" podUID="39d1f358-7083-4b19-800e-ca0bbce8e754" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 16 19:28:44.726270 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:44.726212    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c" podUID="39d1f358-7083-4b19-800e-ca0bbce8e754" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 16 19:28:45.413813 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:45.413773    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-m4465"]
Sep 16 19:28:45.417215 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:45.417197    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-m4465"
Sep 16 19:28:45.421111 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:45.421085    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:28:45.421313 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:45.421114    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
Sep 16 19:28:45.421428 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:45.421408    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\""
Sep 16 19:28:45.430206 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:45.430182    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-m4465"]
Sep 16 19:28:45.497454 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:45.497418    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62b1e971-1993-48a8-a34c-e9af455fbd1b-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-m4465\" (UID: \"62b1e971-1993-48a8-a34c-e9af455fbd1b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-m4465"
Sep 16 19:28:45.497650 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:45.497481    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62b1e971-1993-48a8-a34c-e9af455fbd1b-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-m4465\" (UID: \"62b1e971-1993-48a8-a34c-e9af455fbd1b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-m4465"
Sep 16 19:28:45.497650 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:45.497509    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62b1e971-1993-48a8-a34c-e9af455fbd1b-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-m4465\" (UID: \"62b1e971-1993-48a8-a34c-e9af455fbd1b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-m4465"
Sep 16 19:28:45.497650 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:45.497532    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62b1e971-1993-48a8-a34c-e9af455fbd1b-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-m4465\" (UID: \"62b1e971-1993-48a8-a34c-e9af455fbd1b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-m4465"
Sep 16 19:28:45.497650 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:45.497547    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9tfm\" (UniqueName: \"kubernetes.io/projected/62b1e971-1993-48a8-a34c-e9af455fbd1b-kube-api-access-d9tfm\") pod \"isvc-paddle-runtime-predictor-58b99fff76-m4465\" (UID: \"62b1e971-1993-48a8-a34c-e9af455fbd1b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-m4465"
Sep 16 19:28:45.598929 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:45.598888    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62b1e971-1993-48a8-a34c-e9af455fbd1b-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-m4465\" (UID: \"62b1e971-1993-48a8-a34c-e9af455fbd1b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-m4465"
Sep 16 19:28:45.599122 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:45.598945    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62b1e971-1993-48a8-a34c-e9af455fbd1b-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-m4465\" (UID: \"62b1e971-1993-48a8-a34c-e9af455fbd1b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-m4465"
Sep 16 19:28:45.599122 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:45.598980    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62b1e971-1993-48a8-a34c-e9af455fbd1b-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-m4465\" (UID: \"62b1e971-1993-48a8-a34c-e9af455fbd1b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-m4465"
Sep 16 19:28:45.599122 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:45.599004    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9tfm\" (UniqueName: \"kubernetes.io/projected/62b1e971-1993-48a8-a34c-e9af455fbd1b-kube-api-access-d9tfm\") pod \"isvc-paddle-runtime-predictor-58b99fff76-m4465\" (UID: \"62b1e971-1993-48a8-a34c-e9af455fbd1b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-m4465"
Sep 16 19:28:45.599122 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:45.599068    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62b1e971-1993-48a8-a34c-e9af455fbd1b-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-m4465\" (UID: \"62b1e971-1993-48a8-a34c-e9af455fbd1b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-m4465"
Sep 16 19:28:45.599394 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:28:45.599137    2586 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-runtime-predictor-serving-cert: secret "isvc-paddle-runtime-predictor-serving-cert" not found
Sep 16 19:28:45.599394 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:28:45.599218    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62b1e971-1993-48a8-a34c-e9af455fbd1b-proxy-tls podName:62b1e971-1993-48a8-a34c-e9af455fbd1b nodeName:}" failed. No retries permitted until 2026-09-16 19:28:46.099197519 +0000 UTC m=+2901.543059087 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/62b1e971-1993-48a8-a34c-e9af455fbd1b-proxy-tls") pod "isvc-paddle-runtime-predictor-58b99fff76-m4465" (UID: "62b1e971-1993-48a8-a34c-e9af455fbd1b") : secret "isvc-paddle-runtime-predictor-serving-cert" not found
Sep 16 19:28:45.599502 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:45.599484    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62b1e971-1993-48a8-a34c-e9af455fbd1b-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-m4465\" (UID: \"62b1e971-1993-48a8-a34c-e9af455fbd1b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-m4465"
Sep 16 19:28:45.599721 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:45.599703    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62b1e971-1993-48a8-a34c-e9af455fbd1b-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-m4465\" (UID: \"62b1e971-1993-48a8-a34c-e9af455fbd1b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-m4465"
Sep 16 19:28:45.599785 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:45.599703    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62b1e971-1993-48a8-a34c-e9af455fbd1b-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-m4465\" (UID: \"62b1e971-1993-48a8-a34c-e9af455fbd1b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-m4465"
Sep 16 19:28:45.612784 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:45.612756    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9tfm\" (UniqueName: \"kubernetes.io/projected/62b1e971-1993-48a8-a34c-e9af455fbd1b-kube-api-access-d9tfm\") pod \"isvc-paddle-runtime-predictor-58b99fff76-m4465\" (UID: \"62b1e971-1993-48a8-a34c-e9af455fbd1b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-m4465"
Sep 16 19:28:46.103520 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:46.103488    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62b1e971-1993-48a8-a34c-e9af455fbd1b-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-m4465\" (UID: \"62b1e971-1993-48a8-a34c-e9af455fbd1b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-m4465"
Sep 16 19:28:46.105980 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:46.105955    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62b1e971-1993-48a8-a34c-e9af455fbd1b-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-m4465\" (UID: \"62b1e971-1993-48a8-a34c-e9af455fbd1b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-m4465"
Sep 16 19:28:46.327464 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:46.327427    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-m4465"
Sep 16 19:28:46.611340 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:46.611305    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-m4465"]
Sep 16 19:28:46.616461 ip-10-0-129-190 kubenswrapper[2586]: W0916 19:28:46.616429    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62b1e971_1993_48a8_a34c_e9af455fbd1b.slice/crio-b85b9698fa59e90a529728674c381f07f513a05abd70bf5eb39d378355a83722 WatchSource:0}: Error finding container b85b9698fa59e90a529728674c381f07f513a05abd70bf5eb39d378355a83722: Status 404 returned error can't find the container with id b85b9698fa59e90a529728674c381f07f513a05abd70bf5eb39d378355a83722
Sep 16 19:28:46.850985 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:46.850946    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-m4465" event={"ID":"62b1e971-1993-48a8-a34c-e9af455fbd1b","Type":"ContainerStarted","Data":"729b4bd7c6a4610e3927362cd8198045782617415d212e1c4e782f3184d74fa1"}
Sep 16 19:28:46.851151 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:46.851018    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-m4465" event={"ID":"62b1e971-1993-48a8-a34c-e9af455fbd1b","Type":"ContainerStarted","Data":"b85b9698fa59e90a529728674c381f07f513a05abd70bf5eb39d378355a83722"}
Sep 16 19:28:51.876888 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:51.876852    2586 generic.go:356] "Generic (PLEG): container finished" podID="62b1e971-1993-48a8-a34c-e9af455fbd1b" containerID="729b4bd7c6a4610e3927362cd8198045782617415d212e1c4e782f3184d74fa1" exitCode=0
Sep 16 19:28:51.877296 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:51.876926    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-m4465" event={"ID":"62b1e971-1993-48a8-a34c-e9af455fbd1b","Type":"ContainerDied","Data":"729b4bd7c6a4610e3927362cd8198045782617415d212e1c4e782f3184d74fa1"}
Sep 16 19:28:54.726111 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:28:54.726068    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c" podUID="39d1f358-7083-4b19-800e-ca0bbce8e754" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 16 19:29:03.946936 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:03.946898    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-m4465" event={"ID":"62b1e971-1993-48a8-a34c-e9af455fbd1b","Type":"ContainerStarted","Data":"fb884196d68e8aa6e782b445bca4989eec7a29ff54ee48f8c7056570a32d0c85"}
Sep 16 19:29:03.947383 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:03.946992    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-m4465" event={"ID":"62b1e971-1993-48a8-a34c-e9af455fbd1b","Type":"ContainerStarted","Data":"d8a6bac3430c6543812fd003e7d8973c790843d265471ec855dddeffbcfbdd90"}
Sep 16 19:29:03.947383 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:03.947189    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-m4465"
Sep 16 19:29:03.970298 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:03.970245    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-m4465" podStartSLOduration=3.096402321 podStartE2EDuration="18.97021014s" totalImagesPullingTime="11.483518326s" totalInitContainerRuntime="4.390289493s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:28:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:29:03.968631301 +0000 UTC m=+2919.412492890" watchObservedRunningTime="2026-09-16 19:29:03.97021014 +0000 UTC m=+2919.414071716"
Sep 16 19:29:04.726050 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:04.726012    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c" podUID="39d1f358-7083-4b19-800e-ca0bbce8e754" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 16 19:29:04.953679 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:04.953644    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-m4465"
Sep 16 19:29:04.954862 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:04.954833    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-m4465" podUID="62b1e971-1993-48a8-a34c-e9af455fbd1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Sep 16 19:29:05.958017 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:05.957980    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-m4465" podUID="62b1e971-1993-48a8-a34c-e9af455fbd1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Sep 16 19:29:10.962178 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:10.962148    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-m4465"
Sep 16 19:29:10.962692 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:10.962664    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-m4465" podUID="62b1e971-1993-48a8-a34c-e9af455fbd1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Sep 16 19:29:14.726403 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:14.726360    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c" podUID="39d1f358-7083-4b19-800e-ca0bbce8e754" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 16 19:29:20.962649 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:20.962601    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-m4465" podUID="62b1e971-1993-48a8-a34c-e9af455fbd1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Sep 16 19:29:24.726608 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:24.726563    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c" podUID="39d1f358-7083-4b19-800e-ca0bbce8e754" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 16 19:29:30.962842 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:30.962795    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-m4465" podUID="62b1e971-1993-48a8-a34c-e9af455fbd1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Sep 16 19:29:34.726936 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:34.726905    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c"
Sep 16 19:29:40.963395 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:40.963354    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-m4465" podUID="62b1e971-1993-48a8-a34c-e9af455fbd1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Sep 16 19:29:43.975632 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:43.975596    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c"]
Sep 16 19:29:43.976061 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:43.975871    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c" podUID="39d1f358-7083-4b19-800e-ca0bbce8e754" containerName="kserve-container" containerID="cri-o://3b10ae1b09b6156ce9dfa186f3b8e9f2bdecfe1334deb2dd57212fbef833e8af" gracePeriod=30
Sep 16 19:29:43.976061 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:43.975939    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c" podUID="39d1f358-7083-4b19-800e-ca0bbce8e754" containerName="kube-rbac-proxy" containerID="cri-o://6e503402955af45602a8f5712e45edb7a8fa5cc991ded32f61b94197158dfeeb" gracePeriod=30
Sep 16 19:29:44.137348 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:44.137314    2586 generic.go:356] "Generic (PLEG): container finished" podID="39d1f358-7083-4b19-800e-ca0bbce8e754" containerID="6e503402955af45602a8f5712e45edb7a8fa5cc991ded32f61b94197158dfeeb" exitCode=2
Sep 16 19:29:44.137508 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:44.137404    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c" event={"ID":"39d1f358-7083-4b19-800e-ca0bbce8e754","Type":"ContainerDied","Data":"6e503402955af45602a8f5712e45edb7a8fa5cc991ded32f61b94197158dfeeb"}
Sep 16 19:29:44.720742 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:44.720699    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c" podUID="39d1f358-7083-4b19-800e-ca0bbce8e754" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.66:8643/healthz\": dial tcp 10.134.0.66:8643: connect: connection refused"
Sep 16 19:29:44.726041 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:44.726016    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c" podUID="39d1f358-7083-4b19-800e-ca0bbce8e754" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 16 19:29:48.827399 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:48.827373    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c"
Sep 16 19:29:48.945300 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:48.945182    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39d1f358-7083-4b19-800e-ca0bbce8e754-kserve-provision-location\") pod \"39d1f358-7083-4b19-800e-ca0bbce8e754\" (UID: \"39d1f358-7083-4b19-800e-ca0bbce8e754\") "
Sep 16 19:29:48.945300 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:48.945225    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39d1f358-7083-4b19-800e-ca0bbce8e754-cabundle-cert\") pod \"39d1f358-7083-4b19-800e-ca0bbce8e754\" (UID: \"39d1f358-7083-4b19-800e-ca0bbce8e754\") "
Sep 16 19:29:48.945300 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:48.945301    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39d1f358-7083-4b19-800e-ca0bbce8e754-proxy-tls\") pod \"39d1f358-7083-4b19-800e-ca0bbce8e754\" (UID: \"39d1f358-7083-4b19-800e-ca0bbce8e754\") "
Sep 16 19:29:48.945579 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:48.945338    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ph64m\" (UniqueName: \"kubernetes.io/projected/39d1f358-7083-4b19-800e-ca0bbce8e754-kube-api-access-ph64m\") pod \"39d1f358-7083-4b19-800e-ca0bbce8e754\" (UID: \"39d1f358-7083-4b19-800e-ca0bbce8e754\") "
Sep 16 19:29:48.945579 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:48.945396    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39d1f358-7083-4b19-800e-ca0bbce8e754-kube-rbac-proxy-sar-config\") pod \"39d1f358-7083-4b19-800e-ca0bbce8e754\" (UID: \"39d1f358-7083-4b19-800e-ca0bbce8e754\") "
Sep 16 19:29:48.945579 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:48.945534    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39d1f358-7083-4b19-800e-ca0bbce8e754-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "39d1f358-7083-4b19-800e-ca0bbce8e754" (UID: "39d1f358-7083-4b19-800e-ca0bbce8e754"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:29:48.945736 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:48.945710    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39d1f358-7083-4b19-800e-ca0bbce8e754-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:29:48.945820 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:48.945740    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39d1f358-7083-4b19-800e-ca0bbce8e754-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "39d1f358-7083-4b19-800e-ca0bbce8e754" (UID: "39d1f358-7083-4b19-800e-ca0bbce8e754"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:29:48.945874 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:48.945825    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39d1f358-7083-4b19-800e-ca0bbce8e754-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "39d1f358-7083-4b19-800e-ca0bbce8e754" (UID: "39d1f358-7083-4b19-800e-ca0bbce8e754"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:29:48.947470 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:48.947439    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39d1f358-7083-4b19-800e-ca0bbce8e754-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "39d1f358-7083-4b19-800e-ca0bbce8e754" (UID: "39d1f358-7083-4b19-800e-ca0bbce8e754"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:29:48.947470 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:48.947459    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39d1f358-7083-4b19-800e-ca0bbce8e754-kube-api-access-ph64m" (OuterVolumeSpecName: "kube-api-access-ph64m") pod "39d1f358-7083-4b19-800e-ca0bbce8e754" (UID: "39d1f358-7083-4b19-800e-ca0bbce8e754"). InnerVolumeSpecName "kube-api-access-ph64m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:29:49.046244 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:49.046201    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph64m\" (UniqueName: \"kubernetes.io/projected/39d1f358-7083-4b19-800e-ca0bbce8e754-kube-api-access-ph64m\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:29:49.046414 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:49.046260    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39d1f358-7083-4b19-800e-ca0bbce8e754-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:29:49.046414 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:49.046279    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39d1f358-7083-4b19-800e-ca0bbce8e754-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:29:49.046414 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:49.046291    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39d1f358-7083-4b19-800e-ca0bbce8e754-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:29:49.161728 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:49.161692    2586 generic.go:356] "Generic (PLEG): container finished" podID="39d1f358-7083-4b19-800e-ca0bbce8e754" containerID="3b10ae1b09b6156ce9dfa186f3b8e9f2bdecfe1334deb2dd57212fbef833e8af" exitCode=0
Sep 16 19:29:49.161889 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:49.161750    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c"
Sep 16 19:29:49.161889 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:49.161770    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c" event={"ID":"39d1f358-7083-4b19-800e-ca0bbce8e754","Type":"ContainerDied","Data":"3b10ae1b09b6156ce9dfa186f3b8e9f2bdecfe1334deb2dd57212fbef833e8af"}
Sep 16 19:29:49.161889 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:49.161871    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c" event={"ID":"39d1f358-7083-4b19-800e-ca0bbce8e754","Type":"ContainerDied","Data":"b442bf30879881a927e437e425c699c3426db9076f880f4f5fe703d27f0bf8ea"}
Sep 16 19:29:49.162003 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:49.161910    2586 scope.go:117] "RemoveContainer" containerID="6e503402955af45602a8f5712e45edb7a8fa5cc991ded32f61b94197158dfeeb"
Sep 16 19:29:49.170091 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:49.170071    2586 scope.go:117] "RemoveContainer" containerID="3b10ae1b09b6156ce9dfa186f3b8e9f2bdecfe1334deb2dd57212fbef833e8af"
Sep 16 19:29:49.177589 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:49.177573    2586 scope.go:117] "RemoveContainer" containerID="fa923b29ff74939e71bc03394f1ede583839ad51f924633ff5f1fcf10de67009"
Sep 16 19:29:49.184767 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:49.184749    2586 scope.go:117] "RemoveContainer" containerID="6e503402955af45602a8f5712e45edb7a8fa5cc991ded32f61b94197158dfeeb"
Sep 16 19:29:49.185044 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:29:49.185023    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e503402955af45602a8f5712e45edb7a8fa5cc991ded32f61b94197158dfeeb\": container with ID starting with 6e503402955af45602a8f5712e45edb7a8fa5cc991ded32f61b94197158dfeeb not found: ID does not exist" containerID="6e503402955af45602a8f5712e45edb7a8fa5cc991ded32f61b94197158dfeeb"
Sep 16 19:29:49.185130 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:49.185051    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e503402955af45602a8f5712e45edb7a8fa5cc991ded32f61b94197158dfeeb"} err="failed to get container status \"6e503402955af45602a8f5712e45edb7a8fa5cc991ded32f61b94197158dfeeb\": rpc error: code = NotFound desc = could not find container \"6e503402955af45602a8f5712e45edb7a8fa5cc991ded32f61b94197158dfeeb\": container with ID starting with 6e503402955af45602a8f5712e45edb7a8fa5cc991ded32f61b94197158dfeeb not found: ID does not exist"
Sep 16 19:29:49.185130 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:49.185067    2586 scope.go:117] "RemoveContainer" containerID="3b10ae1b09b6156ce9dfa186f3b8e9f2bdecfe1334deb2dd57212fbef833e8af"
Sep 16 19:29:49.185336 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:29:49.185317    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b10ae1b09b6156ce9dfa186f3b8e9f2bdecfe1334deb2dd57212fbef833e8af\": container with ID starting with 3b10ae1b09b6156ce9dfa186f3b8e9f2bdecfe1334deb2dd57212fbef833e8af not found: ID does not exist" containerID="3b10ae1b09b6156ce9dfa186f3b8e9f2bdecfe1334deb2dd57212fbef833e8af"
Sep 16 19:29:49.185394 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:49.185342    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b10ae1b09b6156ce9dfa186f3b8e9f2bdecfe1334deb2dd57212fbef833e8af"} err="failed to get container status \"3b10ae1b09b6156ce9dfa186f3b8e9f2bdecfe1334deb2dd57212fbef833e8af\": rpc error: code = NotFound desc = could not find container \"3b10ae1b09b6156ce9dfa186f3b8e9f2bdecfe1334deb2dd57212fbef833e8af\": container with ID starting with 3b10ae1b09b6156ce9dfa186f3b8e9f2bdecfe1334deb2dd57212fbef833e8af not found: ID does not exist"
Sep 16 19:29:49.185394 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:49.185360    2586 scope.go:117] "RemoveContainer" containerID="fa923b29ff74939e71bc03394f1ede583839ad51f924633ff5f1fcf10de67009"
Sep 16 19:29:49.185597 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:29:49.185580    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa923b29ff74939e71bc03394f1ede583839ad51f924633ff5f1fcf10de67009\": container with ID starting with fa923b29ff74939e71bc03394f1ede583839ad51f924633ff5f1fcf10de67009 not found: ID does not exist" containerID="fa923b29ff74939e71bc03394f1ede583839ad51f924633ff5f1fcf10de67009"
Sep 16 19:29:49.185641 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:49.185600    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa923b29ff74939e71bc03394f1ede583839ad51f924633ff5f1fcf10de67009"} err="failed to get container status \"fa923b29ff74939e71bc03394f1ede583839ad51f924633ff5f1fcf10de67009\": rpc error: code = NotFound desc = could not find container \"fa923b29ff74939e71bc03394f1ede583839ad51f924633ff5f1fcf10de67009\": container with ID starting with fa923b29ff74939e71bc03394f1ede583839ad51f924633ff5f1fcf10de67009 not found: ID does not exist"
Sep 16 19:29:49.194968 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:49.194941    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c"]
Sep 16 19:29:49.197953 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:49.197904    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-5b8fbfc5d-f6s7c"]
Sep 16 19:29:50.963921 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:50.963886    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-m4465"
Sep 16 19:29:51.091954 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:51.091917    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39d1f358-7083-4b19-800e-ca0bbce8e754" path="/var/lib/kubelet/pods/39d1f358-7083-4b19-800e-ca0bbce8e754/volumes"
Sep 16 19:29:56.246543 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:56.246509    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t"]
Sep 16 19:29:56.246959 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:56.246895    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39d1f358-7083-4b19-800e-ca0bbce8e754" containerName="kserve-container"
Sep 16 19:29:56.246959 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:56.246911    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="39d1f358-7083-4b19-800e-ca0bbce8e754" containerName="kserve-container"
Sep 16 19:29:56.246959 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:56.246930    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39d1f358-7083-4b19-800e-ca0bbce8e754" containerName="kube-rbac-proxy"
Sep 16 19:29:56.246959 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:56.246935    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="39d1f358-7083-4b19-800e-ca0bbce8e754" containerName="kube-rbac-proxy"
Sep 16 19:29:56.246959 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:56.246949    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39d1f358-7083-4b19-800e-ca0bbce8e754" containerName="storage-initializer"
Sep 16 19:29:56.246959 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:56.246956    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="39d1f358-7083-4b19-800e-ca0bbce8e754" containerName="storage-initializer"
Sep 16 19:29:56.247216 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:56.247015    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="39d1f358-7083-4b19-800e-ca0bbce8e754" containerName="kserve-container"
Sep 16 19:29:56.247216 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:56.247030    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="39d1f358-7083-4b19-800e-ca0bbce8e754" containerName="kube-rbac-proxy"
Sep 16 19:29:56.250258 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:56.250241    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t"
Sep 16 19:29:56.252705 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:56.252684    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Sep 16 19:29:56.252888 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:56.252863    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Sep 16 19:29:56.253145 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:56.253127    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 16 19:29:56.268883 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:56.268859    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t"]
Sep 16 19:29:56.407277 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:56.407216    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/82f0b30c-a8bf-4e2d-b96a-d83462e31716-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t\" (UID: \"82f0b30c-a8bf-4e2d-b96a-d83462e31716\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t"
Sep 16 19:29:56.407277 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:56.407277    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82f0b30c-a8bf-4e2d-b96a-d83462e31716-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t\" (UID: \"82f0b30c-a8bf-4e2d-b96a-d83462e31716\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t"
Sep 16 19:29:56.407534 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:56.407366    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb5bq\" (UniqueName: \"kubernetes.io/projected/82f0b30c-a8bf-4e2d-b96a-d83462e31716-kube-api-access-hb5bq\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t\" (UID: \"82f0b30c-a8bf-4e2d-b96a-d83462e31716\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t"
Sep 16 19:29:56.407534 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:56.407419    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82f0b30c-a8bf-4e2d-b96a-d83462e31716-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t\" (UID: \"82f0b30c-a8bf-4e2d-b96a-d83462e31716\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t"
Sep 16 19:29:56.407534 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:56.407465    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82f0b30c-a8bf-4e2d-b96a-d83462e31716-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t\" (UID: \"82f0b30c-a8bf-4e2d-b96a-d83462e31716\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t"
Sep 16 19:29:56.508306 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:56.508184    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82f0b30c-a8bf-4e2d-b96a-d83462e31716-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t\" (UID: \"82f0b30c-a8bf-4e2d-b96a-d83462e31716\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t"
Sep 16 19:29:56.508474 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:56.508334    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/82f0b30c-a8bf-4e2d-b96a-d83462e31716-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t\" (UID: \"82f0b30c-a8bf-4e2d-b96a-d83462e31716\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t"
Sep 16 19:29:56.508474 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:56.508358    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82f0b30c-a8bf-4e2d-b96a-d83462e31716-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t\" (UID: \"82f0b30c-a8bf-4e2d-b96a-d83462e31716\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t"
Sep 16 19:29:56.508474 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:56.508381    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb5bq\" (UniqueName: \"kubernetes.io/projected/82f0b30c-a8bf-4e2d-b96a-d83462e31716-kube-api-access-hb5bq\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t\" (UID: \"82f0b30c-a8bf-4e2d-b96a-d83462e31716\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t"
Sep 16 19:29:56.508474 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:56.508405    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82f0b30c-a8bf-4e2d-b96a-d83462e31716-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t\" (UID: \"82f0b30c-a8bf-4e2d-b96a-d83462e31716\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t"
Sep 16 19:29:56.508829 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:56.508798    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82f0b30c-a8bf-4e2d-b96a-d83462e31716-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t\" (UID: \"82f0b30c-a8bf-4e2d-b96a-d83462e31716\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t"
Sep 16 19:29:56.509101 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:56.509079    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/82f0b30c-a8bf-4e2d-b96a-d83462e31716-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t\" (UID: \"82f0b30c-a8bf-4e2d-b96a-d83462e31716\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t"
Sep 16 19:29:56.509224 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:56.509199    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82f0b30c-a8bf-4e2d-b96a-d83462e31716-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t\" (UID: \"82f0b30c-a8bf-4e2d-b96a-d83462e31716\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t"
Sep 16 19:29:56.510864 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:56.510832    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82f0b30c-a8bf-4e2d-b96a-d83462e31716-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t\" (UID: \"82f0b30c-a8bf-4e2d-b96a-d83462e31716\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t"
Sep 16 19:29:56.522614 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:56.522584    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb5bq\" (UniqueName: \"kubernetes.io/projected/82f0b30c-a8bf-4e2d-b96a-d83462e31716-kube-api-access-hb5bq\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t\" (UID: \"82f0b30c-a8bf-4e2d-b96a-d83462e31716\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t"
Sep 16 19:29:56.560613 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:56.560586    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t"
Sep 16 19:29:56.617121 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:56.617089    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-m4465"]
Sep 16 19:29:56.617463 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:56.617433    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-m4465" podUID="62b1e971-1993-48a8-a34c-e9af455fbd1b" containerName="kserve-container" containerID="cri-o://d8a6bac3430c6543812fd003e7d8973c790843d265471ec855dddeffbcfbdd90" gracePeriod=30
Sep 16 19:29:56.617533 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:56.617503    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-m4465" podUID="62b1e971-1993-48a8-a34c-e9af455fbd1b" containerName="kube-rbac-proxy" containerID="cri-o://fb884196d68e8aa6e782b445bca4989eec7a29ff54ee48f8c7056570a32d0c85" gracePeriod=30
Sep 16 19:29:56.842722 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:56.842691    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t"]
Sep 16 19:29:56.844489 ip-10-0-129-190 kubenswrapper[2586]: W0916 19:29:56.844451    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82f0b30c_a8bf_4e2d_b96a_d83462e31716.slice/crio-a978a76c8789ddb9ee9dacb77d79a3e5923a2909f197b84cb6983b7ceb9f1168 WatchSource:0}: Error finding container a978a76c8789ddb9ee9dacb77d79a3e5923a2909f197b84cb6983b7ceb9f1168: Status 404 returned error can't find the container with id a978a76c8789ddb9ee9dacb77d79a3e5923a2909f197b84cb6983b7ceb9f1168
Sep 16 19:29:57.201279 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:57.201224    2586 generic.go:356] "Generic (PLEG): container finished" podID="62b1e971-1993-48a8-a34c-e9af455fbd1b" containerID="fb884196d68e8aa6e782b445bca4989eec7a29ff54ee48f8c7056570a32d0c85" exitCode=2
Sep 16 19:29:57.201279 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:57.201266    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-m4465" event={"ID":"62b1e971-1993-48a8-a34c-e9af455fbd1b","Type":"ContainerDied","Data":"fb884196d68e8aa6e782b445bca4989eec7a29ff54ee48f8c7056570a32d0c85"}
Sep 16 19:29:57.203038 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:57.203005    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t" event={"ID":"82f0b30c-a8bf-4e2d-b96a-d83462e31716","Type":"ContainerStarted","Data":"a800416b5a7c272a1f7cedbf4458df4eaa189303a5767462ece4a2bfb3640059"}
Sep 16 19:29:57.203173 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:57.203073    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t" event={"ID":"82f0b30c-a8bf-4e2d-b96a-d83462e31716","Type":"ContainerStarted","Data":"a978a76c8789ddb9ee9dacb77d79a3e5923a2909f197b84cb6983b7ceb9f1168"}
Sep 16 19:29:58.209296 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:58.209202    2586 generic.go:356] "Generic (PLEG): container finished" podID="82f0b30c-a8bf-4e2d-b96a-d83462e31716" containerID="a800416b5a7c272a1f7cedbf4458df4eaa189303a5767462ece4a2bfb3640059" exitCode=0
Sep 16 19:29:58.209660 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:58.209287    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t" event={"ID":"82f0b30c-a8bf-4e2d-b96a-d83462e31716","Type":"ContainerDied","Data":"a800416b5a7c272a1f7cedbf4458df4eaa189303a5767462ece4a2bfb3640059"}
Sep 16 19:29:59.216114 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:59.216078    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t" event={"ID":"82f0b30c-a8bf-4e2d-b96a-d83462e31716","Type":"ContainerStarted","Data":"6e18467128715ea4b3a2709271cb993cdeaeb77f7b078409f4d8bd62d14aed09"}
Sep 16 19:29:59.216540 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:59.216142    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t" event={"ID":"82f0b30c-a8bf-4e2d-b96a-d83462e31716","Type":"ContainerStarted","Data":"b4461bb0851d2869d4eacac93657fba9f83fba3812c644d595f9efc096226562"}
Sep 16 19:29:59.216540 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:59.216176    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t"
Sep 16 19:29:59.243733 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:59.243670    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t" podStartSLOduration=2.190462928 podStartE2EDuration="3.243651682s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.053188754s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:29:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:29:59.241619321 +0000 UTC m=+2974.685480898" watchObservedRunningTime="2026-09-16 19:29:59.243651682 +0000 UTC m=+2974.687513258"
Sep 16 19:29:59.571285 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:59.571262    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-m4465"
Sep 16 19:29:59.738161 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:59.738076    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62b1e971-1993-48a8-a34c-e9af455fbd1b-kube-rbac-proxy-sar-config\") pod \"62b1e971-1993-48a8-a34c-e9af455fbd1b\" (UID: \"62b1e971-1993-48a8-a34c-e9af455fbd1b\") "
Sep 16 19:29:59.738161 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:59.738109    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9tfm\" (UniqueName: \"kubernetes.io/projected/62b1e971-1993-48a8-a34c-e9af455fbd1b-kube-api-access-d9tfm\") pod \"62b1e971-1993-48a8-a34c-e9af455fbd1b\" (UID: \"62b1e971-1993-48a8-a34c-e9af455fbd1b\") "
Sep 16 19:29:59.738161 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:59.738158    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62b1e971-1993-48a8-a34c-e9af455fbd1b-proxy-tls\") pod \"62b1e971-1993-48a8-a34c-e9af455fbd1b\" (UID: \"62b1e971-1993-48a8-a34c-e9af455fbd1b\") "
Sep 16 19:29:59.738477 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:59.738187    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62b1e971-1993-48a8-a34c-e9af455fbd1b-kserve-provision-location\") pod \"62b1e971-1993-48a8-a34c-e9af455fbd1b\" (UID: \"62b1e971-1993-48a8-a34c-e9af455fbd1b\") "
Sep 16 19:29:59.738477 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:59.738264    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62b1e971-1993-48a8-a34c-e9af455fbd1b-cabundle-cert\") pod \"62b1e971-1993-48a8-a34c-e9af455fbd1b\" (UID: \"62b1e971-1993-48a8-a34c-e9af455fbd1b\") "
Sep 16 19:29:59.738638 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:59.738535    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62b1e971-1993-48a8-a34c-e9af455fbd1b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "62b1e971-1993-48a8-a34c-e9af455fbd1b" (UID: "62b1e971-1993-48a8-a34c-e9af455fbd1b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:29:59.738773 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:59.738713    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62b1e971-1993-48a8-a34c-e9af455fbd1b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "62b1e971-1993-48a8-a34c-e9af455fbd1b" (UID: "62b1e971-1993-48a8-a34c-e9af455fbd1b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:29:59.740354 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:59.740326    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62b1e971-1993-48a8-a34c-e9af455fbd1b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "62b1e971-1993-48a8-a34c-e9af455fbd1b" (UID: "62b1e971-1993-48a8-a34c-e9af455fbd1b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:29:59.740438 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:59.740380    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62b1e971-1993-48a8-a34c-e9af455fbd1b-kube-api-access-d9tfm" (OuterVolumeSpecName: "kube-api-access-d9tfm") pod "62b1e971-1993-48a8-a34c-e9af455fbd1b" (UID: "62b1e971-1993-48a8-a34c-e9af455fbd1b"). InnerVolumeSpecName "kube-api-access-d9tfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:29:59.748075 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:59.748052    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62b1e971-1993-48a8-a34c-e9af455fbd1b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "62b1e971-1993-48a8-a34c-e9af455fbd1b" (UID: "62b1e971-1993-48a8-a34c-e9af455fbd1b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:29:59.839563 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:59.839527    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62b1e971-1993-48a8-a34c-e9af455fbd1b-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:29:59.839563 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:59.839557    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62b1e971-1993-48a8-a34c-e9af455fbd1b-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:29:59.839750 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:59.839570    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62b1e971-1993-48a8-a34c-e9af455fbd1b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:29:59.839750 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:59.839584    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9tfm\" (UniqueName: \"kubernetes.io/projected/62b1e971-1993-48a8-a34c-e9af455fbd1b-kube-api-access-d9tfm\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:29:59.839750 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:29:59.839597    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62b1e971-1993-48a8-a34c-e9af455fbd1b-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:30:00.223173 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:00.223134    2586 generic.go:356] "Generic (PLEG): container finished" podID="62b1e971-1993-48a8-a34c-e9af455fbd1b" containerID="d8a6bac3430c6543812fd003e7d8973c790843d265471ec855dddeffbcfbdd90" exitCode=0
Sep 16 19:30:00.223634 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:00.223187    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-m4465"
Sep 16 19:30:00.223634 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:00.223218    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-m4465" event={"ID":"62b1e971-1993-48a8-a34c-e9af455fbd1b","Type":"ContainerDied","Data":"d8a6bac3430c6543812fd003e7d8973c790843d265471ec855dddeffbcfbdd90"}
Sep 16 19:30:00.223634 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:00.223296    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-m4465" event={"ID":"62b1e971-1993-48a8-a34c-e9af455fbd1b","Type":"ContainerDied","Data":"b85b9698fa59e90a529728674c381f07f513a05abd70bf5eb39d378355a83722"}
Sep 16 19:30:00.223634 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:00.223324    2586 scope.go:117] "RemoveContainer" containerID="fb884196d68e8aa6e782b445bca4989eec7a29ff54ee48f8c7056570a32d0c85"
Sep 16 19:30:00.223862 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:00.223827    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t"
Sep 16 19:30:00.224960 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:00.224931    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t" podUID="82f0b30c-a8bf-4e2d-b96a-d83462e31716" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Sep 16 19:30:00.233395 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:00.233378    2586 scope.go:117] "RemoveContainer" containerID="d8a6bac3430c6543812fd003e7d8973c790843d265471ec855dddeffbcfbdd90"
Sep 16 19:30:00.241398 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:00.241380    2586 scope.go:117] "RemoveContainer" containerID="729b4bd7c6a4610e3927362cd8198045782617415d212e1c4e782f3184d74fa1"
Sep 16 19:30:00.249213 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:00.249191    2586 scope.go:117] "RemoveContainer" containerID="fb884196d68e8aa6e782b445bca4989eec7a29ff54ee48f8c7056570a32d0c85"
Sep 16 19:30:00.249535 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:30:00.249516    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb884196d68e8aa6e782b445bca4989eec7a29ff54ee48f8c7056570a32d0c85\": container with ID starting with fb884196d68e8aa6e782b445bca4989eec7a29ff54ee48f8c7056570a32d0c85 not found: ID does not exist" containerID="fb884196d68e8aa6e782b445bca4989eec7a29ff54ee48f8c7056570a32d0c85"
Sep 16 19:30:00.249593 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:00.249543    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb884196d68e8aa6e782b445bca4989eec7a29ff54ee48f8c7056570a32d0c85"} err="failed to get container status \"fb884196d68e8aa6e782b445bca4989eec7a29ff54ee48f8c7056570a32d0c85\": rpc error: code = NotFound desc = could not find container \"fb884196d68e8aa6e782b445bca4989eec7a29ff54ee48f8c7056570a32d0c85\": container with ID starting with fb884196d68e8aa6e782b445bca4989eec7a29ff54ee48f8c7056570a32d0c85 not found: ID does not exist"
Sep 16 19:30:00.249593 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:00.249561    2586 scope.go:117] "RemoveContainer" containerID="d8a6bac3430c6543812fd003e7d8973c790843d265471ec855dddeffbcfbdd90"
Sep 16 19:30:00.249829 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:30:00.249811    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8a6bac3430c6543812fd003e7d8973c790843d265471ec855dddeffbcfbdd90\": container with ID starting with d8a6bac3430c6543812fd003e7d8973c790843d265471ec855dddeffbcfbdd90 not found: ID does not exist" containerID="d8a6bac3430c6543812fd003e7d8973c790843d265471ec855dddeffbcfbdd90"
Sep 16 19:30:00.249880 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:00.249834    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8a6bac3430c6543812fd003e7d8973c790843d265471ec855dddeffbcfbdd90"} err="failed to get container status \"d8a6bac3430c6543812fd003e7d8973c790843d265471ec855dddeffbcfbdd90\": rpc error: code = NotFound desc = could not find container \"d8a6bac3430c6543812fd003e7d8973c790843d265471ec855dddeffbcfbdd90\": container with ID starting with d8a6bac3430c6543812fd003e7d8973c790843d265471ec855dddeffbcfbdd90 not found: ID does not exist"
Sep 16 19:30:00.249880 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:00.249848    2586 scope.go:117] "RemoveContainer" containerID="729b4bd7c6a4610e3927362cd8198045782617415d212e1c4e782f3184d74fa1"
Sep 16 19:30:00.250070 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:30:00.250050    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"729b4bd7c6a4610e3927362cd8198045782617415d212e1c4e782f3184d74fa1\": container with ID starting with 729b4bd7c6a4610e3927362cd8198045782617415d212e1c4e782f3184d74fa1 not found: ID does not exist" containerID="729b4bd7c6a4610e3927362cd8198045782617415d212e1c4e782f3184d74fa1"
Sep 16 19:30:00.250117 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:00.250077    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"729b4bd7c6a4610e3927362cd8198045782617415d212e1c4e782f3184d74fa1"} err="failed to get container status \"729b4bd7c6a4610e3927362cd8198045782617415d212e1c4e782f3184d74fa1\": rpc error: code = NotFound desc = could not find container \"729b4bd7c6a4610e3927362cd8198045782617415d212e1c4e782f3184d74fa1\": container with ID starting with 729b4bd7c6a4610e3927362cd8198045782617415d212e1c4e782f3184d74fa1 not found: ID does not exist"
Sep 16 19:30:00.255223 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:00.255199    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-m4465"]
Sep 16 19:30:00.264751 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:00.264724    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-m4465"]
Sep 16 19:30:01.092834 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:01.092797    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62b1e971-1993-48a8-a34c-e9af455fbd1b" path="/var/lib/kubelet/pods/62b1e971-1993-48a8-a34c-e9af455fbd1b/volumes"
Sep 16 19:30:01.228830 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:01.228785    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t" podUID="82f0b30c-a8bf-4e2d-b96a-d83462e31716" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Sep 16 19:30:01.889797 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:01.889759    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps"]
Sep 16 19:30:01.890163 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:01.890149    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62b1e971-1993-48a8-a34c-e9af455fbd1b" containerName="storage-initializer"
Sep 16 19:30:01.890257 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:01.890164    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="62b1e971-1993-48a8-a34c-e9af455fbd1b" containerName="storage-initializer"
Sep 16 19:30:01.890257 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:01.890176    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62b1e971-1993-48a8-a34c-e9af455fbd1b" containerName="kube-rbac-proxy"
Sep 16 19:30:01.890257 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:01.890182    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="62b1e971-1993-48a8-a34c-e9af455fbd1b" containerName="kube-rbac-proxy"
Sep 16 19:30:01.890257 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:01.890199    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62b1e971-1993-48a8-a34c-e9af455fbd1b" containerName="kserve-container"
Sep 16 19:30:01.890257 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:01.890206    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="62b1e971-1993-48a8-a34c-e9af455fbd1b" containerName="kserve-container"
Sep 16 19:30:01.890519 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:01.890272    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="62b1e971-1993-48a8-a34c-e9af455fbd1b" containerName="kserve-container"
Sep 16 19:30:01.890519 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:01.890281    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="62b1e971-1993-48a8-a34c-e9af455fbd1b" containerName="kube-rbac-proxy"
Sep 16 19:30:01.893407 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:01.893386    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps"
Sep 16 19:30:01.897284 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:01.897260    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 16 19:30:01.897403 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:01.897314    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
Sep 16 19:30:01.897403 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:01.897271    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:30:01.910422 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:01.910393    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps"]
Sep 16 19:30:02.057149 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:02.057106    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/adb2b534-4d53-4015-9bd1-80c366d24ec5-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps\" (UID: \"adb2b534-4d53-4015-9bd1-80c366d24ec5\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps"
Sep 16 19:30:02.057399 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:02.057161    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/adb2b534-4d53-4015-9bd1-80c366d24ec5-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps\" (UID: \"adb2b534-4d53-4015-9bd1-80c366d24ec5\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps"
Sep 16 19:30:02.057399 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:02.057185    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/adb2b534-4d53-4015-9bd1-80c366d24ec5-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps\" (UID: \"adb2b534-4d53-4015-9bd1-80c366d24ec5\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps"
Sep 16 19:30:02.057399 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:02.057309    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/adb2b534-4d53-4015-9bd1-80c366d24ec5-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps\" (UID: \"adb2b534-4d53-4015-9bd1-80c366d24ec5\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps"
Sep 16 19:30:02.057399 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:02.057360    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjz6r\" (UniqueName: \"kubernetes.io/projected/adb2b534-4d53-4015-9bd1-80c366d24ec5-kube-api-access-cjz6r\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps\" (UID: \"adb2b534-4d53-4015-9bd1-80c366d24ec5\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps"
Sep 16 19:30:02.157915 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:02.157816    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/adb2b534-4d53-4015-9bd1-80c366d24ec5-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps\" (UID: \"adb2b534-4d53-4015-9bd1-80c366d24ec5\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps"
Sep 16 19:30:02.157915 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:02.157854    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/adb2b534-4d53-4015-9bd1-80c366d24ec5-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps\" (UID: \"adb2b534-4d53-4015-9bd1-80c366d24ec5\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps"
Sep 16 19:30:02.157915 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:02.157887    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/adb2b534-4d53-4015-9bd1-80c366d24ec5-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps\" (UID: \"adb2b534-4d53-4015-9bd1-80c366d24ec5\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps"
Sep 16 19:30:02.158201 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:02.157919    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjz6r\" (UniqueName: \"kubernetes.io/projected/adb2b534-4d53-4015-9bd1-80c366d24ec5-kube-api-access-cjz6r\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps\" (UID: \"adb2b534-4d53-4015-9bd1-80c366d24ec5\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps"
Sep 16 19:30:02.158201 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:02.158008    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/adb2b534-4d53-4015-9bd1-80c366d24ec5-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps\" (UID: \"adb2b534-4d53-4015-9bd1-80c366d24ec5\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps"
Sep 16 19:30:02.158201 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:30:02.158122    2586 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-serving-cert: secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found
Sep 16 19:30:02.158393 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:30:02.158208    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/adb2b534-4d53-4015-9bd1-80c366d24ec5-proxy-tls podName:adb2b534-4d53-4015-9bd1-80c366d24ec5 nodeName:}" failed. No retries permitted until 2026-09-16 19:30:02.658185769 +0000 UTC m=+2978.102047341 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/adb2b534-4d53-4015-9bd1-80c366d24ec5-proxy-tls") pod "isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps" (UID: "adb2b534-4d53-4015-9bd1-80c366d24ec5") : secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found
Sep 16 19:30:02.158469 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:02.158448    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/adb2b534-4d53-4015-9bd1-80c366d24ec5-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps\" (UID: \"adb2b534-4d53-4015-9bd1-80c366d24ec5\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps"
Sep 16 19:30:02.158666 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:02.158646    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/adb2b534-4d53-4015-9bd1-80c366d24ec5-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps\" (UID: \"adb2b534-4d53-4015-9bd1-80c366d24ec5\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps"
Sep 16 19:30:02.158706 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:02.158647    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/adb2b534-4d53-4015-9bd1-80c366d24ec5-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps\" (UID: \"adb2b534-4d53-4015-9bd1-80c366d24ec5\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps"
Sep 16 19:30:02.167844 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:02.167813    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjz6r\" (UniqueName: \"kubernetes.io/projected/adb2b534-4d53-4015-9bd1-80c366d24ec5-kube-api-access-cjz6r\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps\" (UID: \"adb2b534-4d53-4015-9bd1-80c366d24ec5\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps"
Sep 16 19:30:02.662575 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:02.662531    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/adb2b534-4d53-4015-9bd1-80c366d24ec5-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps\" (UID: \"adb2b534-4d53-4015-9bd1-80c366d24ec5\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps"
Sep 16 19:30:02.664943 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:02.664921    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/adb2b534-4d53-4015-9bd1-80c366d24ec5-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps\" (UID: \"adb2b534-4d53-4015-9bd1-80c366d24ec5\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps"
Sep 16 19:30:02.804282 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:02.804220    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps"
Sep 16 19:30:03.089494 ip-10-0-129-190 kubenswrapper[2586]: W0916 19:30:03.089460    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadb2b534_4d53_4015_9bd1_80c366d24ec5.slice/crio-dee03199c2c1980cd018e573376b209c189ebcc479d4e3a4f2bf65fbd60bfe18 WatchSource:0}: Error finding container dee03199c2c1980cd018e573376b209c189ebcc479d4e3a4f2bf65fbd60bfe18: Status 404 returned error can't find the container with id dee03199c2c1980cd018e573376b209c189ebcc479d4e3a4f2bf65fbd60bfe18
Sep 16 19:30:03.093097 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:03.093079    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps"]
Sep 16 19:30:03.241204 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:03.241160    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps" event={"ID":"adb2b534-4d53-4015-9bd1-80c366d24ec5","Type":"ContainerStarted","Data":"9c82ce8c7299d010490e982cacd7efab75414d605338684d06bc2ace6bc1b53d"}
Sep 16 19:30:03.241366 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:03.241257    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps" event={"ID":"adb2b534-4d53-4015-9bd1-80c366d24ec5","Type":"ContainerStarted","Data":"dee03199c2c1980cd018e573376b209c189ebcc479d4e3a4f2bf65fbd60bfe18"}
Sep 16 19:30:06.233698 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:06.233667    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t"
Sep 16 19:30:06.234260 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:06.234211    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t" podUID="82f0b30c-a8bf-4e2d-b96a-d83462e31716" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Sep 16 19:30:09.272879 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:09.272843    2586 generic.go:356] "Generic (PLEG): container finished" podID="adb2b534-4d53-4015-9bd1-80c366d24ec5" containerID="9c82ce8c7299d010490e982cacd7efab75414d605338684d06bc2ace6bc1b53d" exitCode=0
Sep 16 19:30:09.272879 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:09.272884    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps" event={"ID":"adb2b534-4d53-4015-9bd1-80c366d24ec5","Type":"ContainerDied","Data":"9c82ce8c7299d010490e982cacd7efab75414d605338684d06bc2ace6bc1b53d"}
Sep 16 19:30:10.283591 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:10.283557    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps" event={"ID":"adb2b534-4d53-4015-9bd1-80c366d24ec5","Type":"ContainerStarted","Data":"235b588311dfb31acfa2e4702d75f72ecc6bfddc6fe8cdba553cd9fa30e68feb"}
Sep 16 19:30:10.283964 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:10.283626    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps" event={"ID":"adb2b534-4d53-4015-9bd1-80c366d24ec5","Type":"ContainerStarted","Data":"9bb3016c59b3269fd818ed56ef5e5076af91c49eae7ea45caeb3ef0e027b0ded"}
Sep 16 19:30:10.283964 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:10.283912    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps"
Sep 16 19:30:10.284067 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:10.283969    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps"
Sep 16 19:30:10.285397 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:10.285371    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps" podUID="adb2b534-4d53-4015-9bd1-80c366d24ec5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Sep 16 19:30:10.312408 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:10.312357    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps" podStartSLOduration=3.651462385 podStartE2EDuration="9.312339595s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.66087721s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:30:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:30:10.309782645 +0000 UTC m=+2985.753644221" watchObservedRunningTime="2026-09-16 19:30:10.312339595 +0000 UTC m=+2985.756201170"
Sep 16 19:30:11.289237 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:11.289199    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps" podUID="adb2b534-4d53-4015-9bd1-80c366d24ec5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Sep 16 19:30:16.234459 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:16.234410    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t" podUID="82f0b30c-a8bf-4e2d-b96a-d83462e31716" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Sep 16 19:30:16.294543 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:16.294514    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps"
Sep 16 19:30:16.295028 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:16.294998    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps" podUID="adb2b534-4d53-4015-9bd1-80c366d24ec5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Sep 16 19:30:26.235245 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:26.235178    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t" podUID="82f0b30c-a8bf-4e2d-b96a-d83462e31716" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Sep 16 19:30:26.295741 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:26.295703    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps" podUID="adb2b534-4d53-4015-9bd1-80c366d24ec5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Sep 16 19:30:36.234424 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:36.234382    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t" podUID="82f0b30c-a8bf-4e2d-b96a-d83462e31716" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Sep 16 19:30:36.295401 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:36.295358    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps" podUID="adb2b534-4d53-4015-9bd1-80c366d24ec5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Sep 16 19:30:46.234313 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:46.234272    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t" podUID="82f0b30c-a8bf-4e2d-b96a-d83462e31716" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Sep 16 19:30:46.295729 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:46.295688    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps" podUID="adb2b534-4d53-4015-9bd1-80c366d24ec5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Sep 16 19:30:56.235150 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:56.235111    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t" podUID="82f0b30c-a8bf-4e2d-b96a-d83462e31716" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Sep 16 19:30:56.295386 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:30:56.295349    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps"
Sep 16 19:31:03.308674 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:03.308618    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps"]
Sep 16 19:31:03.309224 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:03.308968    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps" podUID="adb2b534-4d53-4015-9bd1-80c366d24ec5" containerName="kserve-container" containerID="cri-o://9bb3016c59b3269fd818ed56ef5e5076af91c49eae7ea45caeb3ef0e027b0ded" gracePeriod=30
Sep 16 19:31:03.309224 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:03.309035    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps" podUID="adb2b534-4d53-4015-9bd1-80c366d24ec5" containerName="kube-rbac-proxy" containerID="cri-o://235b588311dfb31acfa2e4702d75f72ecc6bfddc6fe8cdba553cd9fa30e68feb" gracePeriod=30
Sep 16 19:31:03.551612 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:03.551578    2586 generic.go:356] "Generic (PLEG): container finished" podID="adb2b534-4d53-4015-9bd1-80c366d24ec5" containerID="235b588311dfb31acfa2e4702d75f72ecc6bfddc6fe8cdba553cd9fa30e68feb" exitCode=2
Sep 16 19:31:03.551790 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:03.551652    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps" event={"ID":"adb2b534-4d53-4015-9bd1-80c366d24ec5","Type":"ContainerDied","Data":"235b588311dfb31acfa2e4702d75f72ecc6bfddc6fe8cdba553cd9fa30e68feb"}
Sep 16 19:31:06.159008 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:06.158981    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps"
Sep 16 19:31:06.220491 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:06.220403    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjz6r\" (UniqueName: \"kubernetes.io/projected/adb2b534-4d53-4015-9bd1-80c366d24ec5-kube-api-access-cjz6r\") pod \"adb2b534-4d53-4015-9bd1-80c366d24ec5\" (UID: \"adb2b534-4d53-4015-9bd1-80c366d24ec5\") "
Sep 16 19:31:06.220491 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:06.220439    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/adb2b534-4d53-4015-9bd1-80c366d24ec5-cabundle-cert\") pod \"adb2b534-4d53-4015-9bd1-80c366d24ec5\" (UID: \"adb2b534-4d53-4015-9bd1-80c366d24ec5\") "
Sep 16 19:31:06.220491 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:06.220464    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/adb2b534-4d53-4015-9bd1-80c366d24ec5-proxy-tls\") pod \"adb2b534-4d53-4015-9bd1-80c366d24ec5\" (UID: \"adb2b534-4d53-4015-9bd1-80c366d24ec5\") "
Sep 16 19:31:06.220491 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:06.220481    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/adb2b534-4d53-4015-9bd1-80c366d24ec5-kserve-provision-location\") pod \"adb2b534-4d53-4015-9bd1-80c366d24ec5\" (UID: \"adb2b534-4d53-4015-9bd1-80c366d24ec5\") "
Sep 16 19:31:06.220832 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:06.220499    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/adb2b534-4d53-4015-9bd1-80c366d24ec5-kube-rbac-proxy-sar-config\") pod \"adb2b534-4d53-4015-9bd1-80c366d24ec5\" (UID: \"adb2b534-4d53-4015-9bd1-80c366d24ec5\") "
Sep 16 19:31:06.220894 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:06.220851    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adb2b534-4d53-4015-9bd1-80c366d24ec5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "adb2b534-4d53-4015-9bd1-80c366d24ec5" (UID: "adb2b534-4d53-4015-9bd1-80c366d24ec5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:31:06.220966 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:06.220898    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adb2b534-4d53-4015-9bd1-80c366d24ec5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "adb2b534-4d53-4015-9bd1-80c366d24ec5" (UID: "adb2b534-4d53-4015-9bd1-80c366d24ec5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:31:06.222823 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:06.222790    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adb2b534-4d53-4015-9bd1-80c366d24ec5-kube-api-access-cjz6r" (OuterVolumeSpecName: "kube-api-access-cjz6r") pod "adb2b534-4d53-4015-9bd1-80c366d24ec5" (UID: "adb2b534-4d53-4015-9bd1-80c366d24ec5"). InnerVolumeSpecName "kube-api-access-cjz6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:31:06.222933 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:06.222922    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adb2b534-4d53-4015-9bd1-80c366d24ec5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "adb2b534-4d53-4015-9bd1-80c366d24ec5" (UID: "adb2b534-4d53-4015-9bd1-80c366d24ec5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:31:06.231637 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:06.231607    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb2b534-4d53-4015-9bd1-80c366d24ec5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "adb2b534-4d53-4015-9bd1-80c366d24ec5" (UID: "adb2b534-4d53-4015-9bd1-80c366d24ec5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:31:06.234964 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:06.234935    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t" podUID="82f0b30c-a8bf-4e2d-b96a-d83462e31716" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Sep 16 19:31:06.321675 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:06.321644    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/adb2b534-4d53-4015-9bd1-80c366d24ec5-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:31:06.321675 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:06.321671    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/adb2b534-4d53-4015-9bd1-80c366d24ec5-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:31:06.321675 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:06.321680    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/adb2b534-4d53-4015-9bd1-80c366d24ec5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:31:06.321899 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:06.321690    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjz6r\" (UniqueName: \"kubernetes.io/projected/adb2b534-4d53-4015-9bd1-80c366d24ec5-kube-api-access-cjz6r\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:31:06.321899 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:06.321698    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/adb2b534-4d53-4015-9bd1-80c366d24ec5-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:31:06.568312 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:06.568216    2586 generic.go:356] "Generic (PLEG): container finished" podID="adb2b534-4d53-4015-9bd1-80c366d24ec5" containerID="9bb3016c59b3269fd818ed56ef5e5076af91c49eae7ea45caeb3ef0e027b0ded" exitCode=0
Sep 16 19:31:06.568312 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:06.568261    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps" event={"ID":"adb2b534-4d53-4015-9bd1-80c366d24ec5","Type":"ContainerDied","Data":"9bb3016c59b3269fd818ed56ef5e5076af91c49eae7ea45caeb3ef0e027b0ded"}
Sep 16 19:31:06.568312 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:06.568297    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps"
Sep 16 19:31:06.568529 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:06.568369    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps" event={"ID":"adb2b534-4d53-4015-9bd1-80c366d24ec5","Type":"ContainerDied","Data":"dee03199c2c1980cd018e573376b209c189ebcc479d4e3a4f2bf65fbd60bfe18"}
Sep 16 19:31:06.568529 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:06.568409    2586 scope.go:117] "RemoveContainer" containerID="235b588311dfb31acfa2e4702d75f72ecc6bfddc6fe8cdba553cd9fa30e68feb"
Sep 16 19:31:06.577132 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:06.577114    2586 scope.go:117] "RemoveContainer" containerID="9bb3016c59b3269fd818ed56ef5e5076af91c49eae7ea45caeb3ef0e027b0ded"
Sep 16 19:31:06.584984 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:06.584966    2586 scope.go:117] "RemoveContainer" containerID="9c82ce8c7299d010490e982cacd7efab75414d605338684d06bc2ace6bc1b53d"
Sep 16 19:31:06.592149 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:06.592132    2586 scope.go:117] "RemoveContainer" containerID="235b588311dfb31acfa2e4702d75f72ecc6bfddc6fe8cdba553cd9fa30e68feb"
Sep 16 19:31:06.592441 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:31:06.592422    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"235b588311dfb31acfa2e4702d75f72ecc6bfddc6fe8cdba553cd9fa30e68feb\": container with ID starting with 235b588311dfb31acfa2e4702d75f72ecc6bfddc6fe8cdba553cd9fa30e68feb not found: ID does not exist" containerID="235b588311dfb31acfa2e4702d75f72ecc6bfddc6fe8cdba553cd9fa30e68feb"
Sep 16 19:31:06.592519 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:06.592449    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"235b588311dfb31acfa2e4702d75f72ecc6bfddc6fe8cdba553cd9fa30e68feb"} err="failed to get container status \"235b588311dfb31acfa2e4702d75f72ecc6bfddc6fe8cdba553cd9fa30e68feb\": rpc error: code = NotFound desc = could not find container \"235b588311dfb31acfa2e4702d75f72ecc6bfddc6fe8cdba553cd9fa30e68feb\": container with ID starting with 235b588311dfb31acfa2e4702d75f72ecc6bfddc6fe8cdba553cd9fa30e68feb not found: ID does not exist"
Sep 16 19:31:06.592519 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:06.592466    2586 scope.go:117] "RemoveContainer" containerID="9bb3016c59b3269fd818ed56ef5e5076af91c49eae7ea45caeb3ef0e027b0ded"
Sep 16 19:31:06.592700 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:31:06.592685    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bb3016c59b3269fd818ed56ef5e5076af91c49eae7ea45caeb3ef0e027b0ded\": container with ID starting with 9bb3016c59b3269fd818ed56ef5e5076af91c49eae7ea45caeb3ef0e027b0ded not found: ID does not exist" containerID="9bb3016c59b3269fd818ed56ef5e5076af91c49eae7ea45caeb3ef0e027b0ded"
Sep 16 19:31:06.592742 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:06.592702    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bb3016c59b3269fd818ed56ef5e5076af91c49eae7ea45caeb3ef0e027b0ded"} err="failed to get container status \"9bb3016c59b3269fd818ed56ef5e5076af91c49eae7ea45caeb3ef0e027b0ded\": rpc error: code = NotFound desc = could not find container \"9bb3016c59b3269fd818ed56ef5e5076af91c49eae7ea45caeb3ef0e027b0ded\": container with ID starting with 9bb3016c59b3269fd818ed56ef5e5076af91c49eae7ea45caeb3ef0e027b0ded not found: ID does not exist"
Sep 16 19:31:06.592742 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:06.592713    2586 scope.go:117] "RemoveContainer" containerID="9c82ce8c7299d010490e982cacd7efab75414d605338684d06bc2ace6bc1b53d"
Sep 16 19:31:06.592909 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:31:06.592895    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c82ce8c7299d010490e982cacd7efab75414d605338684d06bc2ace6bc1b53d\": container with ID starting with 9c82ce8c7299d010490e982cacd7efab75414d605338684d06bc2ace6bc1b53d not found: ID does not exist" containerID="9c82ce8c7299d010490e982cacd7efab75414d605338684d06bc2ace6bc1b53d"
Sep 16 19:31:06.592954 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:06.592910    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c82ce8c7299d010490e982cacd7efab75414d605338684d06bc2ace6bc1b53d"} err="failed to get container status \"9c82ce8c7299d010490e982cacd7efab75414d605338684d06bc2ace6bc1b53d\": rpc error: code = NotFound desc = could not find container \"9c82ce8c7299d010490e982cacd7efab75414d605338684d06bc2ace6bc1b53d\": container with ID starting with 9c82ce8c7299d010490e982cacd7efab75414d605338684d06bc2ace6bc1b53d not found: ID does not exist"
Sep 16 19:31:06.600604 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:06.600580    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps"]
Sep 16 19:31:06.619595 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:06.619570    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-ccfps"]
Sep 16 19:31:07.092311 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:07.092282    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adb2b534-4d53-4015-9bd1-80c366d24ec5" path="/var/lib/kubelet/pods/adb2b534-4d53-4015-9bd1-80c366d24ec5/volumes"
Sep 16 19:31:08.610092 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:08.610052    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-lmbkk"]
Sep 16 19:31:08.610704 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:08.610568    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adb2b534-4d53-4015-9bd1-80c366d24ec5" containerName="kube-rbac-proxy"
Sep 16 19:31:08.610704 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:08.610590    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="adb2b534-4d53-4015-9bd1-80c366d24ec5" containerName="kube-rbac-proxy"
Sep 16 19:31:08.610704 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:08.610604    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adb2b534-4d53-4015-9bd1-80c366d24ec5" containerName="storage-initializer"
Sep 16 19:31:08.610704 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:08.610613    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="adb2b534-4d53-4015-9bd1-80c366d24ec5" containerName="storage-initializer"
Sep 16 19:31:08.610704 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:08.610629    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adb2b534-4d53-4015-9bd1-80c366d24ec5" containerName="kserve-container"
Sep 16 19:31:08.610704 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:08.610638    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="adb2b534-4d53-4015-9bd1-80c366d24ec5" containerName="kserve-container"
Sep 16 19:31:08.611091 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:08.610724    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="adb2b534-4d53-4015-9bd1-80c366d24ec5" containerName="kube-rbac-proxy"
Sep 16 19:31:08.611091 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:08.610739    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="adb2b534-4d53-4015-9bd1-80c366d24ec5" containerName="kserve-container"
Sep 16 19:31:08.614238 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:08.614203    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-lmbkk"
Sep 16 19:31:08.617308 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:08.617287    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\""
Sep 16 19:31:08.617433 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:08.617288    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
Sep 16 19:31:08.619638 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:08.619620    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:31:08.627004 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:08.626979    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-lmbkk"]
Sep 16 19:31:08.637669 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:08.637633    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45982012-46bd-45cc-960f-7fb3371800a6-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-lmbkk\" (UID: \"45982012-46bd-45cc-960f-7fb3371800a6\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-lmbkk"
Sep 16 19:31:08.637813 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:08.637742    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45982012-46bd-45cc-960f-7fb3371800a6-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-lmbkk\" (UID: \"45982012-46bd-45cc-960f-7fb3371800a6\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-lmbkk"
Sep 16 19:31:08.637813 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:08.637774    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45982012-46bd-45cc-960f-7fb3371800a6-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-lmbkk\" (UID: \"45982012-46bd-45cc-960f-7fb3371800a6\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-lmbkk"
Sep 16 19:31:08.637901 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:08.637809    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgn4p\" (UniqueName: \"kubernetes.io/projected/45982012-46bd-45cc-960f-7fb3371800a6-kube-api-access-lgn4p\") pod \"isvc-pmml-predictor-8674795f59-lmbkk\" (UID: \"45982012-46bd-45cc-960f-7fb3371800a6\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-lmbkk"
Sep 16 19:31:08.637954 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:08.637889    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45982012-46bd-45cc-960f-7fb3371800a6-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-lmbkk\" (UID: \"45982012-46bd-45cc-960f-7fb3371800a6\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-lmbkk"
Sep 16 19:31:08.738638 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:08.738604    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45982012-46bd-45cc-960f-7fb3371800a6-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-lmbkk\" (UID: \"45982012-46bd-45cc-960f-7fb3371800a6\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-lmbkk"
Sep 16 19:31:08.738638 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:08.738644    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45982012-46bd-45cc-960f-7fb3371800a6-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-lmbkk\" (UID: \"45982012-46bd-45cc-960f-7fb3371800a6\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-lmbkk"
Sep 16 19:31:08.738900 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:08.738671    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgn4p\" (UniqueName: \"kubernetes.io/projected/45982012-46bd-45cc-960f-7fb3371800a6-kube-api-access-lgn4p\") pod \"isvc-pmml-predictor-8674795f59-lmbkk\" (UID: \"45982012-46bd-45cc-960f-7fb3371800a6\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-lmbkk"
Sep 16 19:31:08.738900 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:08.738700    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45982012-46bd-45cc-960f-7fb3371800a6-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-lmbkk\" (UID: \"45982012-46bd-45cc-960f-7fb3371800a6\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-lmbkk"
Sep 16 19:31:08.738900 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:08.738728    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45982012-46bd-45cc-960f-7fb3371800a6-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-lmbkk\" (UID: \"45982012-46bd-45cc-960f-7fb3371800a6\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-lmbkk"
Sep 16 19:31:08.738900 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:31:08.738845    2586 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-pmml-predictor-serving-cert: secret "isvc-pmml-predictor-serving-cert" not found
Sep 16 19:31:08.739124 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:31:08.738924    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45982012-46bd-45cc-960f-7fb3371800a6-proxy-tls podName:45982012-46bd-45cc-960f-7fb3371800a6 nodeName:}" failed. No retries permitted until 2026-09-16 19:31:09.238905737 +0000 UTC m=+3044.682767304 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/45982012-46bd-45cc-960f-7fb3371800a6-proxy-tls") pod "isvc-pmml-predictor-8674795f59-lmbkk" (UID: "45982012-46bd-45cc-960f-7fb3371800a6") : secret "isvc-pmml-predictor-serving-cert" not found
Sep 16 19:31:08.739279 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:08.739260    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45982012-46bd-45cc-960f-7fb3371800a6-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-lmbkk\" (UID: \"45982012-46bd-45cc-960f-7fb3371800a6\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-lmbkk"
Sep 16 19:31:08.739486 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:08.739464    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45982012-46bd-45cc-960f-7fb3371800a6-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-lmbkk\" (UID: \"45982012-46bd-45cc-960f-7fb3371800a6\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-lmbkk"
Sep 16 19:31:08.739574 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:08.739551    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45982012-46bd-45cc-960f-7fb3371800a6-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-lmbkk\" (UID: \"45982012-46bd-45cc-960f-7fb3371800a6\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-lmbkk"
Sep 16 19:31:08.751082 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:08.751051    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgn4p\" (UniqueName: \"kubernetes.io/projected/45982012-46bd-45cc-960f-7fb3371800a6-kube-api-access-lgn4p\") pod \"isvc-pmml-predictor-8674795f59-lmbkk\" (UID: \"45982012-46bd-45cc-960f-7fb3371800a6\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-lmbkk"
Sep 16 19:31:09.243090 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:09.243033    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45982012-46bd-45cc-960f-7fb3371800a6-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-lmbkk\" (UID: \"45982012-46bd-45cc-960f-7fb3371800a6\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-lmbkk"
Sep 16 19:31:09.245482 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:09.245451    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45982012-46bd-45cc-960f-7fb3371800a6-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-lmbkk\" (UID: \"45982012-46bd-45cc-960f-7fb3371800a6\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-lmbkk"
Sep 16 19:31:09.525290 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:09.525173    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-lmbkk"
Sep 16 19:31:09.815980 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:09.815885    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-lmbkk"]
Sep 16 19:31:09.820356 ip-10-0-129-190 kubenswrapper[2586]: W0916 19:31:09.820327    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45982012_46bd_45cc_960f_7fb3371800a6.slice/crio-6094052b53253db21a2d071b2524e03e287278261438cf051cf4e9e2b2e46850 WatchSource:0}: Error finding container 6094052b53253db21a2d071b2524e03e287278261438cf051cf4e9e2b2e46850: Status 404 returned error can't find the container with id 6094052b53253db21a2d071b2524e03e287278261438cf051cf4e9e2b2e46850
Sep 16 19:31:10.591843 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:10.591804    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-lmbkk" event={"ID":"45982012-46bd-45cc-960f-7fb3371800a6","Type":"ContainerStarted","Data":"587c4d99df28cfee11b87840f26addba5e0567ad0469b15d6e4c65b510b0fe74"}
Sep 16 19:31:10.592028 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:10.591868    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-lmbkk" event={"ID":"45982012-46bd-45cc-960f-7fb3371800a6","Type":"ContainerStarted","Data":"6094052b53253db21a2d071b2524e03e287278261438cf051cf4e9e2b2e46850"}
Sep 16 19:31:14.615089 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:14.615046    2586 generic.go:356] "Generic (PLEG): container finished" podID="45982012-46bd-45cc-960f-7fb3371800a6" containerID="587c4d99df28cfee11b87840f26addba5e0567ad0469b15d6e4c65b510b0fe74" exitCode=0
Sep 16 19:31:14.615470 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:14.615116    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-lmbkk" event={"ID":"45982012-46bd-45cc-960f-7fb3371800a6","Type":"ContainerDied","Data":"587c4d99df28cfee11b87840f26addba5e0567ad0469b15d6e4c65b510b0fe74"}
Sep 16 19:31:14.616426 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:14.616409    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:31:16.235094 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:16.235041    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t"
Sep 16 19:31:22.660699 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:22.660656    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-lmbkk" event={"ID":"45982012-46bd-45cc-960f-7fb3371800a6","Type":"ContainerStarted","Data":"1f601cfec09c6319ba36d43b7bf100f5c82c13ff663a20c880546af278a7f2b4"}
Sep 16 19:31:22.661173 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:22.660764    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-lmbkk" event={"ID":"45982012-46bd-45cc-960f-7fb3371800a6","Type":"ContainerStarted","Data":"552b286870e7ec204d831bff8adfa01c2847a540e5fd0832a66d7c4e250db463"}
Sep 16 19:31:22.661173 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:22.660998    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-lmbkk"
Sep 16 19:31:22.695215 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:22.695170    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-lmbkk" podStartSLOduration=3.620272292 podStartE2EDuration="14.695155674s" totalImagesPullingTime="7.21777411s" totalInitContainerRuntime="3.857109272s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:31:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:31:22.691685513 +0000 UTC m=+3058.135547067" watchObservedRunningTime="2026-09-16 19:31:22.695155674 +0000 UTC m=+3058.139017250"
Sep 16 19:31:23.665603 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:23.665571    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-lmbkk"
Sep 16 19:31:23.666988 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:23.666962    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-lmbkk" podUID="45982012-46bd-45cc-960f-7fb3371800a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.70:8080: connect: connection refused"
Sep 16 19:31:24.670053 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:24.670010    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-lmbkk" podUID="45982012-46bd-45cc-960f-7fb3371800a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.70:8080: connect: connection refused"
Sep 16 19:31:26.266732 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:26.266698    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t"]
Sep 16 19:31:26.267203 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:26.267018    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t" podUID="82f0b30c-a8bf-4e2d-b96a-d83462e31716" containerName="kserve-container" containerID="cri-o://b4461bb0851d2869d4eacac93657fba9f83fba3812c644d595f9efc096226562" gracePeriod=30
Sep 16 19:31:26.267203 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:26.267093    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t" podUID="82f0b30c-a8bf-4e2d-b96a-d83462e31716" containerName="kube-rbac-proxy" containerID="cri-o://6e18467128715ea4b3a2709271cb993cdeaeb77f7b078409f4d8bd62d14aed09" gracePeriod=30
Sep 16 19:31:26.681251 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:26.681195    2586 generic.go:356] "Generic (PLEG): container finished" podID="82f0b30c-a8bf-4e2d-b96a-d83462e31716" containerID="6e18467128715ea4b3a2709271cb993cdeaeb77f7b078409f4d8bd62d14aed09" exitCode=2
Sep 16 19:31:26.681418 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:26.681272    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t" event={"ID":"82f0b30c-a8bf-4e2d-b96a-d83462e31716","Type":"ContainerDied","Data":"6e18467128715ea4b3a2709271cb993cdeaeb77f7b078409f4d8bd62d14aed09"}
Sep 16 19:31:27.548754 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:27.548718    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22"]
Sep 16 19:31:27.551932 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:27.551912    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22"
Sep 16 19:31:27.563278 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:27.563258    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
Sep 16 19:31:27.563551 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:27.563535    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
Sep 16 19:31:27.600802 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:27.600757    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcb53242-1dd8-4603-8836-07a8c0f99c83-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22\" (UID: \"dcb53242-1dd8-4603-8836-07a8c0f99c83\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22"
Sep 16 19:31:27.600802 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:27.600802    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dcb53242-1dd8-4603-8836-07a8c0f99c83-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22\" (UID: \"dcb53242-1dd8-4603-8836-07a8c0f99c83\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22"
Sep 16 19:31:27.601033 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:27.600860    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dcb53242-1dd8-4603-8836-07a8c0f99c83-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22\" (UID: \"dcb53242-1dd8-4603-8836-07a8c0f99c83\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22"
Sep 16 19:31:27.601033 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:27.600924    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68mbv\" (UniqueName: \"kubernetes.io/projected/dcb53242-1dd8-4603-8836-07a8c0f99c83-kube-api-access-68mbv\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22\" (UID: \"dcb53242-1dd8-4603-8836-07a8c0f99c83\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22"
Sep 16 19:31:27.626389 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:27.626356    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22"]
Sep 16 19:31:27.701729 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:27.701690    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68mbv\" (UniqueName: \"kubernetes.io/projected/dcb53242-1dd8-4603-8836-07a8c0f99c83-kube-api-access-68mbv\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22\" (UID: \"dcb53242-1dd8-4603-8836-07a8c0f99c83\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22"
Sep 16 19:31:27.701903 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:27.701766    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcb53242-1dd8-4603-8836-07a8c0f99c83-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22\" (UID: \"dcb53242-1dd8-4603-8836-07a8c0f99c83\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22"
Sep 16 19:31:27.701903 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:27.701791    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dcb53242-1dd8-4603-8836-07a8c0f99c83-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22\" (UID: \"dcb53242-1dd8-4603-8836-07a8c0f99c83\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22"
Sep 16 19:31:27.701903 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:27.701819    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dcb53242-1dd8-4603-8836-07a8c0f99c83-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22\" (UID: \"dcb53242-1dd8-4603-8836-07a8c0f99c83\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22"
Sep 16 19:31:27.702215 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:27.702193    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dcb53242-1dd8-4603-8836-07a8c0f99c83-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22\" (UID: \"dcb53242-1dd8-4603-8836-07a8c0f99c83\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22"
Sep 16 19:31:27.702609 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:27.702586    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dcb53242-1dd8-4603-8836-07a8c0f99c83-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22\" (UID: \"dcb53242-1dd8-4603-8836-07a8c0f99c83\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22"
Sep 16 19:31:27.704412 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:27.704395    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcb53242-1dd8-4603-8836-07a8c0f99c83-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22\" (UID: \"dcb53242-1dd8-4603-8836-07a8c0f99c83\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22"
Sep 16 19:31:27.714849 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:27.714824    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68mbv\" (UniqueName: \"kubernetes.io/projected/dcb53242-1dd8-4603-8836-07a8c0f99c83-kube-api-access-68mbv\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22\" (UID: \"dcb53242-1dd8-4603-8836-07a8c0f99c83\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22"
Sep 16 19:31:27.862033 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:27.861999    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22"
Sep 16 19:31:28.142018 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:28.141990    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22"]
Sep 16 19:31:28.144814 ip-10-0-129-190 kubenswrapper[2586]: W0916 19:31:28.144787    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcb53242_1dd8_4603_8836_07a8c0f99c83.slice/crio-48e74f7a6b2223dc81bb8734c9c44160c2b973b5b5b9566f7688ef838703d7c2 WatchSource:0}: Error finding container 48e74f7a6b2223dc81bb8734c9c44160c2b973b5b5b9566f7688ef838703d7c2: Status 404 returned error can't find the container with id 48e74f7a6b2223dc81bb8734c9c44160c2b973b5b5b9566f7688ef838703d7c2
Sep 16 19:31:28.693505 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:28.693466    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22" event={"ID":"dcb53242-1dd8-4603-8836-07a8c0f99c83","Type":"ContainerStarted","Data":"03901805de2a8722347035f5a2d291016cd13916ff251a506defa13691cb9b29"}
Sep 16 19:31:28.693876 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:28.693537    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22" event={"ID":"dcb53242-1dd8-4603-8836-07a8c0f99c83","Type":"ContainerStarted","Data":"48e74f7a6b2223dc81bb8734c9c44160c2b973b5b5b9566f7688ef838703d7c2"}
Sep 16 19:31:29.674774 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:29.674742    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-lmbkk"
Sep 16 19:31:29.675225 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:29.675200    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-lmbkk" podUID="45982012-46bd-45cc-960f-7fb3371800a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.70:8080: connect: connection refused"
Sep 16 19:31:30.110758 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:30.110725    2586 scope.go:117] "RemoveContainer" containerID="c364653a79d5e61ae738082be23e5abe8dd8eb91d8db2be7de997ebe6119389e"
Sep 16 19:31:30.119137 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:30.119118    2586 scope.go:117] "RemoveContainer" containerID="93949c4b96c1b37c6b7d31d522dbe42b5727674bd559bac6bf0af1350c43b1af"
Sep 16 19:31:30.126597 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:30.126581    2586 scope.go:117] "RemoveContainer" containerID="ce8320311b668c53a8a677f5f473bf0c69de060a4896874400cc5cb9395ced6a"
Sep 16 19:31:31.229955 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:31.229904    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t" podUID="82f0b30c-a8bf-4e2d-b96a-d83462e31716" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.68:8643/healthz\": dial tcp 10.134.0.68:8643: connect: connection refused"
Sep 16 19:31:31.514956 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:31.514934    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t"
Sep 16 19:31:31.638920 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:31.638888    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82f0b30c-a8bf-4e2d-b96a-d83462e31716-kserve-provision-location\") pod \"82f0b30c-a8bf-4e2d-b96a-d83462e31716\" (UID: \"82f0b30c-a8bf-4e2d-b96a-d83462e31716\") "
Sep 16 19:31:31.639090 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:31.638932    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82f0b30c-a8bf-4e2d-b96a-d83462e31716-kube-rbac-proxy-sar-config\") pod \"82f0b30c-a8bf-4e2d-b96a-d83462e31716\" (UID: \"82f0b30c-a8bf-4e2d-b96a-d83462e31716\") "
Sep 16 19:31:31.639090 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:31.638951    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82f0b30c-a8bf-4e2d-b96a-d83462e31716-proxy-tls\") pod \"82f0b30c-a8bf-4e2d-b96a-d83462e31716\" (UID: \"82f0b30c-a8bf-4e2d-b96a-d83462e31716\") "
Sep 16 19:31:31.639090 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:31.638986    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hb5bq\" (UniqueName: \"kubernetes.io/projected/82f0b30c-a8bf-4e2d-b96a-d83462e31716-kube-api-access-hb5bq\") pod \"82f0b30c-a8bf-4e2d-b96a-d83462e31716\" (UID: \"82f0b30c-a8bf-4e2d-b96a-d83462e31716\") "
Sep 16 19:31:31.639090 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:31.639007    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/82f0b30c-a8bf-4e2d-b96a-d83462e31716-cabundle-cert\") pod \"82f0b30c-a8bf-4e2d-b96a-d83462e31716\" (UID: \"82f0b30c-a8bf-4e2d-b96a-d83462e31716\") "
Sep 16 19:31:31.639336 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:31.639270    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82f0b30c-a8bf-4e2d-b96a-d83462e31716-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "82f0b30c-a8bf-4e2d-b96a-d83462e31716" (UID: "82f0b30c-a8bf-4e2d-b96a-d83462e31716"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:31:31.639422 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:31.639397    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82f0b30c-a8bf-4e2d-b96a-d83462e31716-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "82f0b30c-a8bf-4e2d-b96a-d83462e31716" (UID: "82f0b30c-a8bf-4e2d-b96a-d83462e31716"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:31:31.639507 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:31.639484    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82f0b30c-a8bf-4e2d-b96a-d83462e31716-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "82f0b30c-a8bf-4e2d-b96a-d83462e31716" (UID: "82f0b30c-a8bf-4e2d-b96a-d83462e31716"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:31:31.640979 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:31.640960    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82f0b30c-a8bf-4e2d-b96a-d83462e31716-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "82f0b30c-a8bf-4e2d-b96a-d83462e31716" (UID: "82f0b30c-a8bf-4e2d-b96a-d83462e31716"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:31:31.641042 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:31.640994    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82f0b30c-a8bf-4e2d-b96a-d83462e31716-kube-api-access-hb5bq" (OuterVolumeSpecName: "kube-api-access-hb5bq") pod "82f0b30c-a8bf-4e2d-b96a-d83462e31716" (UID: "82f0b30c-a8bf-4e2d-b96a-d83462e31716"). InnerVolumeSpecName "kube-api-access-hb5bq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:31:31.707425 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:31.707398    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22_dcb53242-1dd8-4603-8836-07a8c0f99c83/storage-initializer/0.log"
Sep 16 19:31:31.707558 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:31.707437    2586 generic.go:356] "Generic (PLEG): container finished" podID="dcb53242-1dd8-4603-8836-07a8c0f99c83" containerID="03901805de2a8722347035f5a2d291016cd13916ff251a506defa13691cb9b29" exitCode=1
Sep 16 19:31:31.707558 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:31.707514    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22" event={"ID":"dcb53242-1dd8-4603-8836-07a8c0f99c83","Type":"ContainerDied","Data":"03901805de2a8722347035f5a2d291016cd13916ff251a506defa13691cb9b29"}
Sep 16 19:31:31.709661 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:31.709638    2586 generic.go:356] "Generic (PLEG): container finished" podID="82f0b30c-a8bf-4e2d-b96a-d83462e31716" containerID="b4461bb0851d2869d4eacac93657fba9f83fba3812c644d595f9efc096226562" exitCode=0
Sep 16 19:31:31.709765 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:31.709688    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t"
Sep 16 19:31:31.709765 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:31.709719    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t" event={"ID":"82f0b30c-a8bf-4e2d-b96a-d83462e31716","Type":"ContainerDied","Data":"b4461bb0851d2869d4eacac93657fba9f83fba3812c644d595f9efc096226562"}
Sep 16 19:31:31.716295 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:31.710194    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t" event={"ID":"82f0b30c-a8bf-4e2d-b96a-d83462e31716","Type":"ContainerDied","Data":"a978a76c8789ddb9ee9dacb77d79a3e5923a2909f197b84cb6983b7ceb9f1168"}
Sep 16 19:31:31.716295 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:31.710290    2586 scope.go:117] "RemoveContainer" containerID="6e18467128715ea4b3a2709271cb993cdeaeb77f7b078409f4d8bd62d14aed09"
Sep 16 19:31:31.726376 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:31.726356    2586 scope.go:117] "RemoveContainer" containerID="b4461bb0851d2869d4eacac93657fba9f83fba3812c644d595f9efc096226562"
Sep 16 19:31:31.734178 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:31.734158    2586 scope.go:117] "RemoveContainer" containerID="a800416b5a7c272a1f7cedbf4458df4eaa189303a5767462ece4a2bfb3640059"
Sep 16 19:31:31.740214 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:31.740189    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb5bq\" (UniqueName: \"kubernetes.io/projected/82f0b30c-a8bf-4e2d-b96a-d83462e31716-kube-api-access-hb5bq\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:31:31.740340 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:31.740218    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/82f0b30c-a8bf-4e2d-b96a-d83462e31716-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:31:31.740340 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:31.740247    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/82f0b30c-a8bf-4e2d-b96a-d83462e31716-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:31:31.740340 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:31.740261    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/82f0b30c-a8bf-4e2d-b96a-d83462e31716-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:31:31.740340 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:31.740274    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/82f0b30c-a8bf-4e2d-b96a-d83462e31716-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:31:31.744181 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:31.744137    2586 scope.go:117] "RemoveContainer" containerID="6e18467128715ea4b3a2709271cb993cdeaeb77f7b078409f4d8bd62d14aed09"
Sep 16 19:31:31.744567 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:31:31.744539    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e18467128715ea4b3a2709271cb993cdeaeb77f7b078409f4d8bd62d14aed09\": container with ID starting with 6e18467128715ea4b3a2709271cb993cdeaeb77f7b078409f4d8bd62d14aed09 not found: ID does not exist" containerID="6e18467128715ea4b3a2709271cb993cdeaeb77f7b078409f4d8bd62d14aed09"
Sep 16 19:31:31.744645 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:31.744580    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e18467128715ea4b3a2709271cb993cdeaeb77f7b078409f4d8bd62d14aed09"} err="failed to get container status \"6e18467128715ea4b3a2709271cb993cdeaeb77f7b078409f4d8bd62d14aed09\": rpc error: code = NotFound desc = could not find container \"6e18467128715ea4b3a2709271cb993cdeaeb77f7b078409f4d8bd62d14aed09\": container with ID starting with 6e18467128715ea4b3a2709271cb993cdeaeb77f7b078409f4d8bd62d14aed09 not found: ID does not exist"
Sep 16 19:31:31.744645 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:31.744610    2586 scope.go:117] "RemoveContainer" containerID="b4461bb0851d2869d4eacac93657fba9f83fba3812c644d595f9efc096226562"
Sep 16 19:31:31.744957 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:31:31.744922    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4461bb0851d2869d4eacac93657fba9f83fba3812c644d595f9efc096226562\": container with ID starting with b4461bb0851d2869d4eacac93657fba9f83fba3812c644d595f9efc096226562 not found: ID does not exist" containerID="b4461bb0851d2869d4eacac93657fba9f83fba3812c644d595f9efc096226562"
Sep 16 19:31:31.745042 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:31.744963    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4461bb0851d2869d4eacac93657fba9f83fba3812c644d595f9efc096226562"} err="failed to get container status \"b4461bb0851d2869d4eacac93657fba9f83fba3812c644d595f9efc096226562\": rpc error: code = NotFound desc = could not find container \"b4461bb0851d2869d4eacac93657fba9f83fba3812c644d595f9efc096226562\": container with ID starting with b4461bb0851d2869d4eacac93657fba9f83fba3812c644d595f9efc096226562 not found: ID does not exist"
Sep 16 19:31:31.745042 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:31.744984    2586 scope.go:117] "RemoveContainer" containerID="a800416b5a7c272a1f7cedbf4458df4eaa189303a5767462ece4a2bfb3640059"
Sep 16 19:31:31.745340 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:31:31.745313    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a800416b5a7c272a1f7cedbf4458df4eaa189303a5767462ece4a2bfb3640059\": container with ID starting with a800416b5a7c272a1f7cedbf4458df4eaa189303a5767462ece4a2bfb3640059 not found: ID does not exist" containerID="a800416b5a7c272a1f7cedbf4458df4eaa189303a5767462ece4a2bfb3640059"
Sep 16 19:31:31.745421 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:31.745343    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a800416b5a7c272a1f7cedbf4458df4eaa189303a5767462ece4a2bfb3640059"} err="failed to get container status \"a800416b5a7c272a1f7cedbf4458df4eaa189303a5767462ece4a2bfb3640059\": rpc error: code = NotFound desc = could not find container \"a800416b5a7c272a1f7cedbf4458df4eaa189303a5767462ece4a2bfb3640059\": container with ID starting with a800416b5a7c272a1f7cedbf4458df4eaa189303a5767462ece4a2bfb3640059 not found: ID does not exist"
Sep 16 19:31:31.769822 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:31.769798    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t"]
Sep 16 19:31:31.775874 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:31.775852    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-55954796bf-2hd4t"]
Sep 16 19:31:32.716351 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:32.716322    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22_dcb53242-1dd8-4603-8836-07a8c0f99c83/storage-initializer/0.log"
Sep 16 19:31:32.716735 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:32.716424    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22" event={"ID":"dcb53242-1dd8-4603-8836-07a8c0f99c83","Type":"ContainerStarted","Data":"f2b1d7b77a6363263fa5835bf0b8755bc555d58a0ff03d592419a85130d17112"}
Sep 16 19:31:33.092323 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:33.092248    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82f0b30c-a8bf-4e2d-b96a-d83462e31716" path="/var/lib/kubelet/pods/82f0b30c-a8bf-4e2d-b96a-d83462e31716/volumes"
Sep 16 19:31:35.734782 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:35.734754    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22_dcb53242-1dd8-4603-8836-07a8c0f99c83/storage-initializer/1.log"
Sep 16 19:31:35.735178 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:35.735147    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22_dcb53242-1dd8-4603-8836-07a8c0f99c83/storage-initializer/0.log"
Sep 16 19:31:35.735223 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:35.735176    2586 generic.go:356] "Generic (PLEG): container finished" podID="dcb53242-1dd8-4603-8836-07a8c0f99c83" containerID="f2b1d7b77a6363263fa5835bf0b8755bc555d58a0ff03d592419a85130d17112" exitCode=1
Sep 16 19:31:35.735288 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:35.735262    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22" event={"ID":"dcb53242-1dd8-4603-8836-07a8c0f99c83","Type":"ContainerDied","Data":"f2b1d7b77a6363263fa5835bf0b8755bc555d58a0ff03d592419a85130d17112"}
Sep 16 19:31:35.735365 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:35.735351    2586 scope.go:117] "RemoveContainer" containerID="03901805de2a8722347035f5a2d291016cd13916ff251a506defa13691cb9b29"
Sep 16 19:31:35.735767 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:35.735740    2586 scope.go:117] "RemoveContainer" containerID="03901805de2a8722347035f5a2d291016cd13916ff251a506defa13691cb9b29"
Sep 16 19:31:35.746023 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:31:35.745999    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22_kserve-ci-e2e-test(dcb53242-1dd8-4603-8836-07a8c0f99c83)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22" podUID="dcb53242-1dd8-4603-8836-07a8c0f99c83"
Sep 16 19:31:35.746159 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:31:35.746132    2586 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22_kserve-ci-e2e-test_dcb53242-1dd8-4603-8836-07a8c0f99c83_0 in pod sandbox 48e74f7a6b2223dc81bb8734c9c44160c2b973b5b5b9566f7688ef838703d7c2 from index: no such id: '03901805de2a8722347035f5a2d291016cd13916ff251a506defa13691cb9b29'" containerID="03901805de2a8722347035f5a2d291016cd13916ff251a506defa13691cb9b29"
Sep 16 19:31:35.746251 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:35.746165    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03901805de2a8722347035f5a2d291016cd13916ff251a506defa13691cb9b29"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22_kserve-ci-e2e-test_dcb53242-1dd8-4603-8836-07a8c0f99c83_0 in pod sandbox 48e74f7a6b2223dc81bb8734c9c44160c2b973b5b5b9566f7688ef838703d7c2 from index: no such id: '03901805de2a8722347035f5a2d291016cd13916ff251a506defa13691cb9b29'"
Sep 16 19:31:36.742048 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:36.742019    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22_dcb53242-1dd8-4603-8836-07a8c0f99c83/storage-initializer/1.log"
Sep 16 19:31:37.476269 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:37.476220    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22"]
Sep 16 19:31:37.610453 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:37.610429    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22_dcb53242-1dd8-4603-8836-07a8c0f99c83/storage-initializer/1.log"
Sep 16 19:31:37.612586 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:37.612569    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22"
Sep 16 19:31:37.692530 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:37.692490    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcb53242-1dd8-4603-8836-07a8c0f99c83-proxy-tls\") pod \"dcb53242-1dd8-4603-8836-07a8c0f99c83\" (UID: \"dcb53242-1dd8-4603-8836-07a8c0f99c83\") "
Sep 16 19:31:37.692692 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:37.692558    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dcb53242-1dd8-4603-8836-07a8c0f99c83-kube-rbac-proxy-sar-config\") pod \"dcb53242-1dd8-4603-8836-07a8c0f99c83\" (UID: \"dcb53242-1dd8-4603-8836-07a8c0f99c83\") "
Sep 16 19:31:37.692692 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:37.692616    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68mbv\" (UniqueName: \"kubernetes.io/projected/dcb53242-1dd8-4603-8836-07a8c0f99c83-kube-api-access-68mbv\") pod \"dcb53242-1dd8-4603-8836-07a8c0f99c83\" (UID: \"dcb53242-1dd8-4603-8836-07a8c0f99c83\") "
Sep 16 19:31:37.692839 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:37.692699    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dcb53242-1dd8-4603-8836-07a8c0f99c83-kserve-provision-location\") pod \"dcb53242-1dd8-4603-8836-07a8c0f99c83\" (UID: \"dcb53242-1dd8-4603-8836-07a8c0f99c83\") "
Sep 16 19:31:37.693030 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:37.693000    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcb53242-1dd8-4603-8836-07a8c0f99c83-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dcb53242-1dd8-4603-8836-07a8c0f99c83" (UID: "dcb53242-1dd8-4603-8836-07a8c0f99c83"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:31:37.693030 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:37.693016    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcb53242-1dd8-4603-8836-07a8c0f99c83-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dcb53242-1dd8-4603-8836-07a8c0f99c83" (UID: "dcb53242-1dd8-4603-8836-07a8c0f99c83"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:31:37.694707 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:37.694684    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcb53242-1dd8-4603-8836-07a8c0f99c83-kube-api-access-68mbv" (OuterVolumeSpecName: "kube-api-access-68mbv") pod "dcb53242-1dd8-4603-8836-07a8c0f99c83" (UID: "dcb53242-1dd8-4603-8836-07a8c0f99c83"). InnerVolumeSpecName "kube-api-access-68mbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:31:37.694960 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:37.694943    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcb53242-1dd8-4603-8836-07a8c0f99c83-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dcb53242-1dd8-4603-8836-07a8c0f99c83" (UID: "dcb53242-1dd8-4603-8836-07a8c0f99c83"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:31:37.747855 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:37.747784    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22_dcb53242-1dd8-4603-8836-07a8c0f99c83/storage-initializer/1.log"
Sep 16 19:31:37.747855 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:37.747848    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22" event={"ID":"dcb53242-1dd8-4603-8836-07a8c0f99c83","Type":"ContainerDied","Data":"48e74f7a6b2223dc81bb8734c9c44160c2b973b5b5b9566f7688ef838703d7c2"}
Sep 16 19:31:37.748336 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:37.747881    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22"
Sep 16 19:31:37.748336 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:37.747906    2586 scope.go:117] "RemoveContainer" containerID="f2b1d7b77a6363263fa5835bf0b8755bc555d58a0ff03d592419a85130d17112"
Sep 16 19:31:37.793324 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:37.793297    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dcb53242-1dd8-4603-8836-07a8c0f99c83-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:31:37.793324 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:37.793320    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcb53242-1dd8-4603-8836-07a8c0f99c83-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:31:37.793324 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:37.793330    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dcb53242-1dd8-4603-8836-07a8c0f99c83-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:31:37.793507 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:37.793338    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68mbv\" (UniqueName: \"kubernetes.io/projected/dcb53242-1dd8-4603-8836-07a8c0f99c83-kube-api-access-68mbv\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:31:37.803174 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:37.803150    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22"]
Sep 16 19:31:37.810672 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:37.810649    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-7d7c8797f9-28r22"]
Sep 16 19:31:38.721939 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:38.721904    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-scj7n"]
Sep 16 19:31:38.722259 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:38.722246    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82f0b30c-a8bf-4e2d-b96a-d83462e31716" containerName="kserve-container"
Sep 16 19:31:38.722318 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:38.722262    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="82f0b30c-a8bf-4e2d-b96a-d83462e31716" containerName="kserve-container"
Sep 16 19:31:38.722318 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:38.722271    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcb53242-1dd8-4603-8836-07a8c0f99c83" containerName="storage-initializer"
Sep 16 19:31:38.722318 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:38.722276    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcb53242-1dd8-4603-8836-07a8c0f99c83" containerName="storage-initializer"
Sep 16 19:31:38.722318 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:38.722294    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcb53242-1dd8-4603-8836-07a8c0f99c83" containerName="storage-initializer"
Sep 16 19:31:38.722318 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:38.722300    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcb53242-1dd8-4603-8836-07a8c0f99c83" containerName="storage-initializer"
Sep 16 19:31:38.722318 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:38.722310    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82f0b30c-a8bf-4e2d-b96a-d83462e31716" containerName="storage-initializer"
Sep 16 19:31:38.722318 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:38.722317    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="82f0b30c-a8bf-4e2d-b96a-d83462e31716" containerName="storage-initializer"
Sep 16 19:31:38.722533 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:38.722327    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82f0b30c-a8bf-4e2d-b96a-d83462e31716" containerName="kube-rbac-proxy"
Sep 16 19:31:38.722533 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:38.722333    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="82f0b30c-a8bf-4e2d-b96a-d83462e31716" containerName="kube-rbac-proxy"
Sep 16 19:31:38.722533 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:38.722386    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcb53242-1dd8-4603-8836-07a8c0f99c83" containerName="storage-initializer"
Sep 16 19:31:38.722533 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:38.722396    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="82f0b30c-a8bf-4e2d-b96a-d83462e31716" containerName="kube-rbac-proxy"
Sep 16 19:31:38.722533 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:38.722403    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="82f0b30c-a8bf-4e2d-b96a-d83462e31716" containerName="kserve-container"
Sep 16 19:31:38.722533 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:38.722508    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcb53242-1dd8-4603-8836-07a8c0f99c83" containerName="storage-initializer"
Sep 16 19:31:38.727004 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:38.726983    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-scj7n"
Sep 16 19:31:38.730114 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:38.730090    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\""
Sep 16 19:31:38.730477 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:38.730452    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
Sep 16 19:31:38.751714 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:38.751675    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-scj7n"]
Sep 16 19:31:38.803565 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:38.803537    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27kzv\" (UniqueName: \"kubernetes.io/projected/3d174115-ec34-4c0d-839e-6ec8276bb6a5-kube-api-access-27kzv\") pod \"isvc-pmml-runtime-predictor-885fc8777-scj7n\" (UID: \"3d174115-ec34-4c0d-839e-6ec8276bb6a5\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-scj7n"
Sep 16 19:31:38.803742 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:38.803584    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d174115-ec34-4c0d-839e-6ec8276bb6a5-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-scj7n\" (UID: \"3d174115-ec34-4c0d-839e-6ec8276bb6a5\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-scj7n"
Sep 16 19:31:38.803742 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:38.803674    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d174115-ec34-4c0d-839e-6ec8276bb6a5-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-scj7n\" (UID: \"3d174115-ec34-4c0d-839e-6ec8276bb6a5\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-scj7n"
Sep 16 19:31:38.803742 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:38.803724    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d174115-ec34-4c0d-839e-6ec8276bb6a5-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-scj7n\" (UID: \"3d174115-ec34-4c0d-839e-6ec8276bb6a5\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-scj7n"
Sep 16 19:31:38.803869 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:38.803813    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d174115-ec34-4c0d-839e-6ec8276bb6a5-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-scj7n\" (UID: \"3d174115-ec34-4c0d-839e-6ec8276bb6a5\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-scj7n"
Sep 16 19:31:38.904997 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:38.904955    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d174115-ec34-4c0d-839e-6ec8276bb6a5-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-scj7n\" (UID: \"3d174115-ec34-4c0d-839e-6ec8276bb6a5\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-scj7n"
Sep 16 19:31:38.904997 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:38.905004    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27kzv\" (UniqueName: \"kubernetes.io/projected/3d174115-ec34-4c0d-839e-6ec8276bb6a5-kube-api-access-27kzv\") pod \"isvc-pmml-runtime-predictor-885fc8777-scj7n\" (UID: \"3d174115-ec34-4c0d-839e-6ec8276bb6a5\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-scj7n"
Sep 16 19:31:38.905292 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:38.905138    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d174115-ec34-4c0d-839e-6ec8276bb6a5-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-scj7n\" (UID: \"3d174115-ec34-4c0d-839e-6ec8276bb6a5\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-scj7n"
Sep 16 19:31:38.905292 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:38.905240    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d174115-ec34-4c0d-839e-6ec8276bb6a5-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-scj7n\" (UID: \"3d174115-ec34-4c0d-839e-6ec8276bb6a5\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-scj7n"
Sep 16 19:31:38.905292 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:38.905274    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d174115-ec34-4c0d-839e-6ec8276bb6a5-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-scj7n\" (UID: \"3d174115-ec34-4c0d-839e-6ec8276bb6a5\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-scj7n"
Sep 16 19:31:38.905443 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:31:38.905315    2586 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-pmml-runtime-predictor-serving-cert: secret "isvc-pmml-runtime-predictor-serving-cert" not found
Sep 16 19:31:38.905443 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:38.905371    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d174115-ec34-4c0d-839e-6ec8276bb6a5-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-scj7n\" (UID: \"3d174115-ec34-4c0d-839e-6ec8276bb6a5\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-scj7n"
Sep 16 19:31:38.905443 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:31:38.905401    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d174115-ec34-4c0d-839e-6ec8276bb6a5-proxy-tls podName:3d174115-ec34-4c0d-839e-6ec8276bb6a5 nodeName:}" failed. No retries permitted until 2026-09-16 19:31:39.405380536 +0000 UTC m=+3074.849242104 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3d174115-ec34-4c0d-839e-6ec8276bb6a5-proxy-tls") pod "isvc-pmml-runtime-predictor-885fc8777-scj7n" (UID: "3d174115-ec34-4c0d-839e-6ec8276bb6a5") : secret "isvc-pmml-runtime-predictor-serving-cert" not found
Sep 16 19:31:38.905826 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:38.905803    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d174115-ec34-4c0d-839e-6ec8276bb6a5-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-scj7n\" (UID: \"3d174115-ec34-4c0d-839e-6ec8276bb6a5\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-scj7n"
Sep 16 19:31:38.905930 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:38.905908    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d174115-ec34-4c0d-839e-6ec8276bb6a5-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-scj7n\" (UID: \"3d174115-ec34-4c0d-839e-6ec8276bb6a5\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-scj7n"
Sep 16 19:31:38.918210 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:38.918177    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27kzv\" (UniqueName: \"kubernetes.io/projected/3d174115-ec34-4c0d-839e-6ec8276bb6a5-kube-api-access-27kzv\") pod \"isvc-pmml-runtime-predictor-885fc8777-scj7n\" (UID: \"3d174115-ec34-4c0d-839e-6ec8276bb6a5\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-scj7n"
Sep 16 19:31:39.094928 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:39.094849    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcb53242-1dd8-4603-8836-07a8c0f99c83" path="/var/lib/kubelet/pods/dcb53242-1dd8-4603-8836-07a8c0f99c83/volumes"
Sep 16 19:31:39.409926 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:39.409888    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d174115-ec34-4c0d-839e-6ec8276bb6a5-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-scj7n\" (UID: \"3d174115-ec34-4c0d-839e-6ec8276bb6a5\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-scj7n"
Sep 16 19:31:39.412471 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:39.412448    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d174115-ec34-4c0d-839e-6ec8276bb6a5-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-scj7n\" (UID: \"3d174115-ec34-4c0d-839e-6ec8276bb6a5\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-scj7n"
Sep 16 19:31:39.637477 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:39.637424    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-scj7n"
Sep 16 19:31:39.676076 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:39.675987    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-lmbkk" podUID="45982012-46bd-45cc-960f-7fb3371800a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.70:8080: connect: connection refused"
Sep 16 19:31:39.923344 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:39.923317    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-scj7n"]
Sep 16 19:31:39.926177 ip-10-0-129-190 kubenswrapper[2586]: W0916 19:31:39.926110    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d174115_ec34_4c0d_839e_6ec8276bb6a5.slice/crio-94bec421b24502ad32de86bd06ec2b8fbc91222e90f10a9ebbf03d906444fcdd WatchSource:0}: Error finding container 94bec421b24502ad32de86bd06ec2b8fbc91222e90f10a9ebbf03d906444fcdd: Status 404 returned error can't find the container with id 94bec421b24502ad32de86bd06ec2b8fbc91222e90f10a9ebbf03d906444fcdd
Sep 16 19:31:40.767270 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:40.767215    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-scj7n" event={"ID":"3d174115-ec34-4c0d-839e-6ec8276bb6a5","Type":"ContainerStarted","Data":"37d4f9e374bdc6081a1084de4d8466bdc45f58e9b7f807aeb73dd778a318549c"}
Sep 16 19:31:40.767476 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:40.767289    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-scj7n" event={"ID":"3d174115-ec34-4c0d-839e-6ec8276bb6a5","Type":"ContainerStarted","Data":"94bec421b24502ad32de86bd06ec2b8fbc91222e90f10a9ebbf03d906444fcdd"}
Sep 16 19:31:44.789319 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:44.789282    2586 generic.go:356] "Generic (PLEG): container finished" podID="3d174115-ec34-4c0d-839e-6ec8276bb6a5" containerID="37d4f9e374bdc6081a1084de4d8466bdc45f58e9b7f807aeb73dd778a318549c" exitCode=0
Sep 16 19:31:44.789695 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:44.789357    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-scj7n" event={"ID":"3d174115-ec34-4c0d-839e-6ec8276bb6a5","Type":"ContainerDied","Data":"37d4f9e374bdc6081a1084de4d8466bdc45f58e9b7f807aeb73dd778a318549c"}
Sep 16 19:31:45.797296 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:45.797262    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-scj7n" event={"ID":"3d174115-ec34-4c0d-839e-6ec8276bb6a5","Type":"ContainerStarted","Data":"e55a9850fae859e9004109255808127ee06bf73b27828261420594fa0d3bf1b5"}
Sep 16 19:31:45.797713 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:45.797323    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-scj7n" event={"ID":"3d174115-ec34-4c0d-839e-6ec8276bb6a5","Type":"ContainerStarted","Data":"726da3149fc9163f65e670eb1ef9c5af5cbe8cb7440ae513e2f2420e3afd848f"}
Sep 16 19:31:45.797713 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:45.797620    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-scj7n"
Sep 16 19:31:45.797818 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:45.797781    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-scj7n"
Sep 16 19:31:45.799007 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:45.798980    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-scj7n" podUID="3d174115-ec34-4c0d-839e-6ec8276bb6a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.72:8080: connect: connection refused"
Sep 16 19:31:45.833029 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:45.832983    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-scj7n" podStartSLOduration=3.95092497 podStartE2EDuration="7.832968939s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.882043969s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:31:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:31:45.830543505 +0000 UTC m=+3081.274405082" watchObservedRunningTime="2026-09-16 19:31:45.832968939 +0000 UTC m=+3081.276830514"
Sep 16 19:31:46.801394 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:46.801351    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-scj7n" podUID="3d174115-ec34-4c0d-839e-6ec8276bb6a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.72:8080: connect: connection refused"
Sep 16 19:31:49.675983 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:49.675944    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-lmbkk" podUID="45982012-46bd-45cc-960f-7fb3371800a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.70:8080: connect: connection refused"
Sep 16 19:31:51.805744 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:51.805712    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-scj7n"
Sep 16 19:31:51.806225 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:51.806194    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-scj7n" podUID="3d174115-ec34-4c0d-839e-6ec8276bb6a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.72:8080: connect: connection refused"
Sep 16 19:31:59.675177 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:31:59.675136    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-lmbkk" podUID="45982012-46bd-45cc-960f-7fb3371800a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.70:8080: connect: connection refused"
Sep 16 19:32:01.806764 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:01.806721    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-scj7n" podUID="3d174115-ec34-4c0d-839e-6ec8276bb6a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.72:8080: connect: connection refused"
Sep 16 19:32:09.675586 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:09.675546    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-lmbkk" podUID="45982012-46bd-45cc-960f-7fb3371800a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.70:8080: connect: connection refused"
Sep 16 19:32:11.806381 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:11.806336    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-scj7n" podUID="3d174115-ec34-4c0d-839e-6ec8276bb6a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.72:8080: connect: connection refused"
Sep 16 19:32:19.675406 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:19.675367    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-lmbkk" podUID="45982012-46bd-45cc-960f-7fb3371800a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.70:8080: connect: connection refused"
Sep 16 19:32:21.806457 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:21.806416    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-scj7n" podUID="3d174115-ec34-4c0d-839e-6ec8276bb6a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.72:8080: connect: connection refused"
Sep 16 19:32:29.675276 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:29.675210    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-lmbkk" podUID="45982012-46bd-45cc-960f-7fb3371800a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.70:8080: connect: connection refused"
Sep 16 19:32:31.806277 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:31.806220    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-scj7n" podUID="3d174115-ec34-4c0d-839e-6ec8276bb6a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.72:8080: connect: connection refused"
Sep 16 19:32:39.675806 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:39.675761    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-lmbkk" podUID="45982012-46bd-45cc-960f-7fb3371800a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.70:8080: connect: connection refused"
Sep 16 19:32:41.807050 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:41.807012    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-scj7n" podUID="3d174115-ec34-4c0d-839e-6ec8276bb6a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.72:8080: connect: connection refused"
Sep 16 19:32:44.087896 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:44.087865    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-lmbkk"
Sep 16 19:32:49.602587 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:49.602492    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-lmbkk"]
Sep 16 19:32:49.603024 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:49.602716    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-lmbkk" podUID="45982012-46bd-45cc-960f-7fb3371800a6" containerName="kserve-container" containerID="cri-o://552b286870e7ec204d831bff8adfa01c2847a540e5fd0832a66d7c4e250db463" gracePeriod=30
Sep 16 19:32:49.603024 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:49.602767    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-lmbkk" podUID="45982012-46bd-45cc-960f-7fb3371800a6" containerName="kube-rbac-proxy" containerID="cri-o://1f601cfec09c6319ba36d43b7bf100f5c82c13ff663a20c880546af278a7f2b4" gracePeriod=30
Sep 16 19:32:49.671010 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:49.670965    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-lmbkk" podUID="45982012-46bd-45cc-960f-7fb3371800a6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.70:8643/healthz\": dial tcp 10.134.0.70:8643: connect: connection refused"
Sep 16 19:32:50.107295 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:50.107261    2586 generic.go:356] "Generic (PLEG): container finished" podID="45982012-46bd-45cc-960f-7fb3371800a6" containerID="1f601cfec09c6319ba36d43b7bf100f5c82c13ff663a20c880546af278a7f2b4" exitCode=2
Sep 16 19:32:50.107295 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:50.107299    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-lmbkk" event={"ID":"45982012-46bd-45cc-960f-7fb3371800a6","Type":"ContainerDied","Data":"1f601cfec09c6319ba36d43b7bf100f5c82c13ff663a20c880546af278a7f2b4"}
Sep 16 19:32:51.806687 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:51.806648    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-scj7n" podUID="3d174115-ec34-4c0d-839e-6ec8276bb6a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.72:8080: connect: connection refused"
Sep 16 19:32:53.355048 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:53.355024    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-lmbkk"
Sep 16 19:32:53.549862 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:53.549777    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45982012-46bd-45cc-960f-7fb3371800a6-kube-rbac-proxy-sar-config\") pod \"45982012-46bd-45cc-960f-7fb3371800a6\" (UID: \"45982012-46bd-45cc-960f-7fb3371800a6\") "
Sep 16 19:32:53.549862 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:53.549815    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45982012-46bd-45cc-960f-7fb3371800a6-cabundle-cert\") pod \"45982012-46bd-45cc-960f-7fb3371800a6\" (UID: \"45982012-46bd-45cc-960f-7fb3371800a6\") "
Sep 16 19:32:53.549862 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:53.549833    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgn4p\" (UniqueName: \"kubernetes.io/projected/45982012-46bd-45cc-960f-7fb3371800a6-kube-api-access-lgn4p\") pod \"45982012-46bd-45cc-960f-7fb3371800a6\" (UID: \"45982012-46bd-45cc-960f-7fb3371800a6\") "
Sep 16 19:32:53.550101 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:53.549970    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45982012-46bd-45cc-960f-7fb3371800a6-proxy-tls\") pod \"45982012-46bd-45cc-960f-7fb3371800a6\" (UID: \"45982012-46bd-45cc-960f-7fb3371800a6\") "
Sep 16 19:32:53.550101 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:53.550025    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45982012-46bd-45cc-960f-7fb3371800a6-kserve-provision-location\") pod \"45982012-46bd-45cc-960f-7fb3371800a6\" (UID: \"45982012-46bd-45cc-960f-7fb3371800a6\") "
Sep 16 19:32:53.550204 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:53.550176    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45982012-46bd-45cc-960f-7fb3371800a6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "45982012-46bd-45cc-960f-7fb3371800a6" (UID: "45982012-46bd-45cc-960f-7fb3371800a6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:32:53.550418 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:53.550381    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45982012-46bd-45cc-960f-7fb3371800a6-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:32:53.550418 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:53.550392    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45982012-46bd-45cc-960f-7fb3371800a6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "45982012-46bd-45cc-960f-7fb3371800a6" (UID: "45982012-46bd-45cc-960f-7fb3371800a6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:32:53.550570 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:53.550441    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45982012-46bd-45cc-960f-7fb3371800a6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "45982012-46bd-45cc-960f-7fb3371800a6" (UID: "45982012-46bd-45cc-960f-7fb3371800a6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:32:53.552035 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:53.552011    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45982012-46bd-45cc-960f-7fb3371800a6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "45982012-46bd-45cc-960f-7fb3371800a6" (UID: "45982012-46bd-45cc-960f-7fb3371800a6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:32:53.552113 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:53.552075    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45982012-46bd-45cc-960f-7fb3371800a6-kube-api-access-lgn4p" (OuterVolumeSpecName: "kube-api-access-lgn4p") pod "45982012-46bd-45cc-960f-7fb3371800a6" (UID: "45982012-46bd-45cc-960f-7fb3371800a6"). InnerVolumeSpecName "kube-api-access-lgn4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:32:53.651566 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:53.651516    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45982012-46bd-45cc-960f-7fb3371800a6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:32:53.651566 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:53.651563    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgn4p\" (UniqueName: \"kubernetes.io/projected/45982012-46bd-45cc-960f-7fb3371800a6-kube-api-access-lgn4p\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:32:53.651566 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:53.651576    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45982012-46bd-45cc-960f-7fb3371800a6-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:32:53.651852 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:53.651591    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45982012-46bd-45cc-960f-7fb3371800a6-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:32:54.087572 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:54.087527    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-scj7n" podUID="3d174115-ec34-4c0d-839e-6ec8276bb6a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.72:8080: connect: connection refused"
Sep 16 19:32:54.126504 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:54.126473    2586 generic.go:356] "Generic (PLEG): container finished" podID="45982012-46bd-45cc-960f-7fb3371800a6" containerID="552b286870e7ec204d831bff8adfa01c2847a540e5fd0832a66d7c4e250db463" exitCode=0
Sep 16 19:32:54.126660 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:54.126528    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-lmbkk"
Sep 16 19:32:54.126660 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:54.126566    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-lmbkk" event={"ID":"45982012-46bd-45cc-960f-7fb3371800a6","Type":"ContainerDied","Data":"552b286870e7ec204d831bff8adfa01c2847a540e5fd0832a66d7c4e250db463"}
Sep 16 19:32:54.126660 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:54.126633    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-lmbkk" event={"ID":"45982012-46bd-45cc-960f-7fb3371800a6","Type":"ContainerDied","Data":"6094052b53253db21a2d071b2524e03e287278261438cf051cf4e9e2b2e46850"}
Sep 16 19:32:54.126775 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:54.126672    2586 scope.go:117] "RemoveContainer" containerID="1f601cfec09c6319ba36d43b7bf100f5c82c13ff663a20c880546af278a7f2b4"
Sep 16 19:32:54.136146 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:54.136126    2586 scope.go:117] "RemoveContainer" containerID="552b286870e7ec204d831bff8adfa01c2847a540e5fd0832a66d7c4e250db463"
Sep 16 19:32:54.144323 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:54.144306    2586 scope.go:117] "RemoveContainer" containerID="587c4d99df28cfee11b87840f26addba5e0567ad0469b15d6e4c65b510b0fe74"
Sep 16 19:32:54.151936 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:54.151900    2586 scope.go:117] "RemoveContainer" containerID="1f601cfec09c6319ba36d43b7bf100f5c82c13ff663a20c880546af278a7f2b4"
Sep 16 19:32:54.152347 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:32:54.152292    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f601cfec09c6319ba36d43b7bf100f5c82c13ff663a20c880546af278a7f2b4\": container with ID starting with 1f601cfec09c6319ba36d43b7bf100f5c82c13ff663a20c880546af278a7f2b4 not found: ID does not exist" containerID="1f601cfec09c6319ba36d43b7bf100f5c82c13ff663a20c880546af278a7f2b4"
Sep 16 19:32:54.152347 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:54.152329    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f601cfec09c6319ba36d43b7bf100f5c82c13ff663a20c880546af278a7f2b4"} err="failed to get container status \"1f601cfec09c6319ba36d43b7bf100f5c82c13ff663a20c880546af278a7f2b4\": rpc error: code = NotFound desc = could not find container \"1f601cfec09c6319ba36d43b7bf100f5c82c13ff663a20c880546af278a7f2b4\": container with ID starting with 1f601cfec09c6319ba36d43b7bf100f5c82c13ff663a20c880546af278a7f2b4 not found: ID does not exist"
Sep 16 19:32:54.152347 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:54.152351    2586 scope.go:117] "RemoveContainer" containerID="552b286870e7ec204d831bff8adfa01c2847a540e5fd0832a66d7c4e250db463"
Sep 16 19:32:54.152704 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:32:54.152680    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"552b286870e7ec204d831bff8adfa01c2847a540e5fd0832a66d7c4e250db463\": container with ID starting with 552b286870e7ec204d831bff8adfa01c2847a540e5fd0832a66d7c4e250db463 not found: ID does not exist" containerID="552b286870e7ec204d831bff8adfa01c2847a540e5fd0832a66d7c4e250db463"
Sep 16 19:32:54.152757 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:54.152714    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"552b286870e7ec204d831bff8adfa01c2847a540e5fd0832a66d7c4e250db463"} err="failed to get container status \"552b286870e7ec204d831bff8adfa01c2847a540e5fd0832a66d7c4e250db463\": rpc error: code = NotFound desc = could not find container \"552b286870e7ec204d831bff8adfa01c2847a540e5fd0832a66d7c4e250db463\": container with ID starting with 552b286870e7ec204d831bff8adfa01c2847a540e5fd0832a66d7c4e250db463 not found: ID does not exist"
Sep 16 19:32:54.152757 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:54.152736    2586 scope.go:117] "RemoveContainer" containerID="587c4d99df28cfee11b87840f26addba5e0567ad0469b15d6e4c65b510b0fe74"
Sep 16 19:32:54.152995 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:32:54.152965    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"587c4d99df28cfee11b87840f26addba5e0567ad0469b15d6e4c65b510b0fe74\": container with ID starting with 587c4d99df28cfee11b87840f26addba5e0567ad0469b15d6e4c65b510b0fe74 not found: ID does not exist" containerID="587c4d99df28cfee11b87840f26addba5e0567ad0469b15d6e4c65b510b0fe74"
Sep 16 19:32:54.153048 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:54.152994    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"587c4d99df28cfee11b87840f26addba5e0567ad0469b15d6e4c65b510b0fe74"} err="failed to get container status \"587c4d99df28cfee11b87840f26addba5e0567ad0469b15d6e4c65b510b0fe74\": rpc error: code = NotFound desc = could not find container \"587c4d99df28cfee11b87840f26addba5e0567ad0469b15d6e4c65b510b0fe74\": container with ID starting with 587c4d99df28cfee11b87840f26addba5e0567ad0469b15d6e4c65b510b0fe74 not found: ID does not exist"
Sep 16 19:32:54.154005 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:54.153986    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-lmbkk"]
Sep 16 19:32:54.160752 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:54.160731    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-lmbkk"]
Sep 16 19:32:54.845118 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:54.845078    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-63d815-predictor-54888c955d-rw7wt"]
Sep 16 19:32:54.845589 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:54.845572    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45982012-46bd-45cc-960f-7fb3371800a6" containerName="kube-rbac-proxy"
Sep 16 19:32:54.845647 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:54.845591    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="45982012-46bd-45cc-960f-7fb3371800a6" containerName="kube-rbac-proxy"
Sep 16 19:32:54.845647 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:54.845625    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45982012-46bd-45cc-960f-7fb3371800a6" containerName="storage-initializer"
Sep 16 19:32:54.845647 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:54.845634    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="45982012-46bd-45cc-960f-7fb3371800a6" containerName="storage-initializer"
Sep 16 19:32:54.845749 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:54.845652    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45982012-46bd-45cc-960f-7fb3371800a6" containerName="kserve-container"
Sep 16 19:32:54.845749 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:54.845662    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="45982012-46bd-45cc-960f-7fb3371800a6" containerName="kserve-container"
Sep 16 19:32:54.845749 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:54.845734    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="45982012-46bd-45cc-960f-7fb3371800a6" containerName="kserve-container"
Sep 16 19:32:54.845749 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:54.845743    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="45982012-46bd-45cc-960f-7fb3371800a6" containerName="kube-rbac-proxy"
Sep 16 19:32:54.850702 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:54.850680    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-63d815-predictor-54888c955d-rw7wt"
Sep 16 19:32:54.853511 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:54.853489    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-63d815-predictor-serving-cert\""
Sep 16 19:32:54.853829 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:54.853806    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-63d815-kube-rbac-proxy-sar-config\""
Sep 16 19:32:54.861953 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:54.861928    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-63d815-predictor-54888c955d-rw7wt"]
Sep 16 19:32:54.862218 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:54.862198    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f20bd9fa-33f6-456e-aeb1-49f242479ed5-cabundle-cert\") pod \"isvc-primary-63d815-predictor-54888c955d-rw7wt\" (UID: \"f20bd9fa-33f6-456e-aeb1-49f242479ed5\") " pod="kserve-ci-e2e-test/isvc-primary-63d815-predictor-54888c955d-rw7wt"
Sep 16 19:32:54.862320 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:54.862248    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5chb2\" (UniqueName: \"kubernetes.io/projected/f20bd9fa-33f6-456e-aeb1-49f242479ed5-kube-api-access-5chb2\") pod \"isvc-primary-63d815-predictor-54888c955d-rw7wt\" (UID: \"f20bd9fa-33f6-456e-aeb1-49f242479ed5\") " pod="kserve-ci-e2e-test/isvc-primary-63d815-predictor-54888c955d-rw7wt"
Sep 16 19:32:54.862320 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:54.862277    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f20bd9fa-33f6-456e-aeb1-49f242479ed5-kserve-provision-location\") pod \"isvc-primary-63d815-predictor-54888c955d-rw7wt\" (UID: \"f20bd9fa-33f6-456e-aeb1-49f242479ed5\") " pod="kserve-ci-e2e-test/isvc-primary-63d815-predictor-54888c955d-rw7wt"
Sep 16 19:32:54.862320 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:54.862306    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f20bd9fa-33f6-456e-aeb1-49f242479ed5-kube-rbac-proxy-sar-config\") pod \"isvc-primary-63d815-predictor-54888c955d-rw7wt\" (UID: \"f20bd9fa-33f6-456e-aeb1-49f242479ed5\") " pod="kserve-ci-e2e-test/isvc-primary-63d815-predictor-54888c955d-rw7wt"
Sep 16 19:32:54.862462 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:54.862336    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f20bd9fa-33f6-456e-aeb1-49f242479ed5-proxy-tls\") pod \"isvc-primary-63d815-predictor-54888c955d-rw7wt\" (UID: \"f20bd9fa-33f6-456e-aeb1-49f242479ed5\") " pod="kserve-ci-e2e-test/isvc-primary-63d815-predictor-54888c955d-rw7wt"
Sep 16 19:32:54.963733 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:54.963701    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f20bd9fa-33f6-456e-aeb1-49f242479ed5-cabundle-cert\") pod \"isvc-primary-63d815-predictor-54888c955d-rw7wt\" (UID: \"f20bd9fa-33f6-456e-aeb1-49f242479ed5\") " pod="kserve-ci-e2e-test/isvc-primary-63d815-predictor-54888c955d-rw7wt"
Sep 16 19:32:54.963733 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:54.963736    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5chb2\" (UniqueName: \"kubernetes.io/projected/f20bd9fa-33f6-456e-aeb1-49f242479ed5-kube-api-access-5chb2\") pod \"isvc-primary-63d815-predictor-54888c955d-rw7wt\" (UID: \"f20bd9fa-33f6-456e-aeb1-49f242479ed5\") " pod="kserve-ci-e2e-test/isvc-primary-63d815-predictor-54888c955d-rw7wt"
Sep 16 19:32:54.963997 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:54.963753    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f20bd9fa-33f6-456e-aeb1-49f242479ed5-kserve-provision-location\") pod \"isvc-primary-63d815-predictor-54888c955d-rw7wt\" (UID: \"f20bd9fa-33f6-456e-aeb1-49f242479ed5\") " pod="kserve-ci-e2e-test/isvc-primary-63d815-predictor-54888c955d-rw7wt"
Sep 16 19:32:54.963997 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:54.963776    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f20bd9fa-33f6-456e-aeb1-49f242479ed5-kube-rbac-proxy-sar-config\") pod \"isvc-primary-63d815-predictor-54888c955d-rw7wt\" (UID: \"f20bd9fa-33f6-456e-aeb1-49f242479ed5\") " pod="kserve-ci-e2e-test/isvc-primary-63d815-predictor-54888c955d-rw7wt"
Sep 16 19:32:54.963997 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:54.963804    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f20bd9fa-33f6-456e-aeb1-49f242479ed5-proxy-tls\") pod \"isvc-primary-63d815-predictor-54888c955d-rw7wt\" (UID: \"f20bd9fa-33f6-456e-aeb1-49f242479ed5\") " pod="kserve-ci-e2e-test/isvc-primary-63d815-predictor-54888c955d-rw7wt"
Sep 16 19:32:54.964253 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:32:54.964015    2586 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-primary-63d815-predictor-serving-cert: secret "isvc-primary-63d815-predictor-serving-cert" not found
Sep 16 19:32:54.964253 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:32:54.964084    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f20bd9fa-33f6-456e-aeb1-49f242479ed5-proxy-tls podName:f20bd9fa-33f6-456e-aeb1-49f242479ed5 nodeName:}" failed. No retries permitted until 2026-09-16 19:32:55.464064717 +0000 UTC m=+3150.907926285 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f20bd9fa-33f6-456e-aeb1-49f242479ed5-proxy-tls") pod "isvc-primary-63d815-predictor-54888c955d-rw7wt" (UID: "f20bd9fa-33f6-456e-aeb1-49f242479ed5") : secret "isvc-primary-63d815-predictor-serving-cert" not found
Sep 16 19:32:54.964253 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:54.964205    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f20bd9fa-33f6-456e-aeb1-49f242479ed5-kserve-provision-location\") pod \"isvc-primary-63d815-predictor-54888c955d-rw7wt\" (UID: \"f20bd9fa-33f6-456e-aeb1-49f242479ed5\") " pod="kserve-ci-e2e-test/isvc-primary-63d815-predictor-54888c955d-rw7wt"
Sep 16 19:32:54.964414 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:54.964397    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f20bd9fa-33f6-456e-aeb1-49f242479ed5-cabundle-cert\") pod \"isvc-primary-63d815-predictor-54888c955d-rw7wt\" (UID: \"f20bd9fa-33f6-456e-aeb1-49f242479ed5\") " pod="kserve-ci-e2e-test/isvc-primary-63d815-predictor-54888c955d-rw7wt"
Sep 16 19:32:54.964579 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:54.964559    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f20bd9fa-33f6-456e-aeb1-49f242479ed5-kube-rbac-proxy-sar-config\") pod \"isvc-primary-63d815-predictor-54888c955d-rw7wt\" (UID: \"f20bd9fa-33f6-456e-aeb1-49f242479ed5\") " pod="kserve-ci-e2e-test/isvc-primary-63d815-predictor-54888c955d-rw7wt"
Sep 16 19:32:54.975929 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:54.975905    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5chb2\" (UniqueName: \"kubernetes.io/projected/f20bd9fa-33f6-456e-aeb1-49f242479ed5-kube-api-access-5chb2\") pod \"isvc-primary-63d815-predictor-54888c955d-rw7wt\" (UID: \"f20bd9fa-33f6-456e-aeb1-49f242479ed5\") " pod="kserve-ci-e2e-test/isvc-primary-63d815-predictor-54888c955d-rw7wt"
Sep 16 19:32:55.093330 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:55.093289    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45982012-46bd-45cc-960f-7fb3371800a6" path="/var/lib/kubelet/pods/45982012-46bd-45cc-960f-7fb3371800a6/volumes"
Sep 16 19:32:55.467880 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:55.467841    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f20bd9fa-33f6-456e-aeb1-49f242479ed5-proxy-tls\") pod \"isvc-primary-63d815-predictor-54888c955d-rw7wt\" (UID: \"f20bd9fa-33f6-456e-aeb1-49f242479ed5\") " pod="kserve-ci-e2e-test/isvc-primary-63d815-predictor-54888c955d-rw7wt"
Sep 16 19:32:55.470283 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:55.470259    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f20bd9fa-33f6-456e-aeb1-49f242479ed5-proxy-tls\") pod \"isvc-primary-63d815-predictor-54888c955d-rw7wt\" (UID: \"f20bd9fa-33f6-456e-aeb1-49f242479ed5\") " pod="kserve-ci-e2e-test/isvc-primary-63d815-predictor-54888c955d-rw7wt"
Sep 16 19:32:55.762138 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:55.762048    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-63d815-predictor-54888c955d-rw7wt"
Sep 16 19:32:56.047465 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:56.047412    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-63d815-predictor-54888c955d-rw7wt"]
Sep 16 19:32:56.050147 ip-10-0-129-190 kubenswrapper[2586]: W0916 19:32:56.050120    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf20bd9fa_33f6_456e_aeb1_49f242479ed5.slice/crio-4027487b5e7f5075c9feab59a14e61ab5fd5edc27ddd2f750ed677355558a5c9 WatchSource:0}: Error finding container 4027487b5e7f5075c9feab59a14e61ab5fd5edc27ddd2f750ed677355558a5c9: Status 404 returned error can't find the container with id 4027487b5e7f5075c9feab59a14e61ab5fd5edc27ddd2f750ed677355558a5c9
Sep 16 19:32:56.138789 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:56.138754    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-63d815-predictor-54888c955d-rw7wt" event={"ID":"f20bd9fa-33f6-456e-aeb1-49f242479ed5","Type":"ContainerStarted","Data":"119d9f84351e8bd7a38ec73e349e799aaf18fc6178ca7ad9205c971bf7fc2c66"}
Sep 16 19:32:56.138977 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:32:56.138808    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-63d815-predictor-54888c955d-rw7wt" event={"ID":"f20bd9fa-33f6-456e-aeb1-49f242479ed5","Type":"ContainerStarted","Data":"4027487b5e7f5075c9feab59a14e61ab5fd5edc27ddd2f750ed677355558a5c9"}
Sep 16 19:33:00.158528 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:00.158488    2586 generic.go:356] "Generic (PLEG): container finished" podID="f20bd9fa-33f6-456e-aeb1-49f242479ed5" containerID="119d9f84351e8bd7a38ec73e349e799aaf18fc6178ca7ad9205c971bf7fc2c66" exitCode=0
Sep 16 19:33:00.158910 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:00.158564    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-63d815-predictor-54888c955d-rw7wt" event={"ID":"f20bd9fa-33f6-456e-aeb1-49f242479ed5","Type":"ContainerDied","Data":"119d9f84351e8bd7a38ec73e349e799aaf18fc6178ca7ad9205c971bf7fc2c66"}
Sep 16 19:33:01.165666 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:01.165628    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-63d815-predictor-54888c955d-rw7wt" event={"ID":"f20bd9fa-33f6-456e-aeb1-49f242479ed5","Type":"ContainerStarted","Data":"5bc916ba328055e96f40b82af5372edbac4a4214bf29d7d1b0de7c5c16da5525"}
Sep 16 19:33:01.166056 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:01.165688    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-63d815-predictor-54888c955d-rw7wt" event={"ID":"f20bd9fa-33f6-456e-aeb1-49f242479ed5","Type":"ContainerStarted","Data":"ed32e43470645a2079a11e8007b6d9d3ad9d69a2a7a4073213f93de639224242"}
Sep 16 19:33:01.166056 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:01.165954    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-63d815-predictor-54888c955d-rw7wt"
Sep 16 19:33:01.166056 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:01.166011    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-63d815-predictor-54888c955d-rw7wt"
Sep 16 19:33:01.167295 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:01.167269    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-63d815-predictor-54888c955d-rw7wt" podUID="f20bd9fa-33f6-456e-aeb1-49f242479ed5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.73:8080: connect: connection refused"
Sep 16 19:33:01.198004 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:01.197953    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-63d815-predictor-54888c955d-rw7wt" podStartSLOduration=3.4830924420000002 podStartE2EDuration="7.197940126s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.714847684s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:32:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:33:01.196386246 +0000 UTC m=+3156.640247823" watchObservedRunningTime="2026-09-16 19:33:01.197940126 +0000 UTC m=+3156.641801699"
Sep 16 19:33:02.170193 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:02.170153    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-63d815-predictor-54888c955d-rw7wt" podUID="f20bd9fa-33f6-456e-aeb1-49f242479ed5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.73:8080: connect: connection refused"
Sep 16 19:33:04.087873 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:04.087830    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-scj7n" podUID="3d174115-ec34-4c0d-839e-6ec8276bb6a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.72:8080: connect: connection refused"
Sep 16 19:33:07.174859 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:07.174825    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-63d815-predictor-54888c955d-rw7wt"
Sep 16 19:33:07.175406 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:07.175378    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-63d815-predictor-54888c955d-rw7wt" podUID="f20bd9fa-33f6-456e-aeb1-49f242479ed5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.73:8080: connect: connection refused"
Sep 16 19:33:14.088375 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:14.088342    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-scj7n"
Sep 16 19:33:17.176177 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:17.176136    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-63d815-predictor-54888c955d-rw7wt" podUID="f20bd9fa-33f6-456e-aeb1-49f242479ed5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.73:8080: connect: connection refused"
Sep 16 19:33:19.833197 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:19.833157    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-scj7n"]
Sep 16 19:33:19.833794 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:19.833410    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-scj7n" podUID="3d174115-ec34-4c0d-839e-6ec8276bb6a5" containerName="kserve-container" containerID="cri-o://726da3149fc9163f65e670eb1ef9c5af5cbe8cb7440ae513e2f2420e3afd848f" gracePeriod=30
Sep 16 19:33:19.833794 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:19.833445    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-scj7n" podUID="3d174115-ec34-4c0d-839e-6ec8276bb6a5" containerName="kube-rbac-proxy" containerID="cri-o://e55a9850fae859e9004109255808127ee06bf73b27828261420594fa0d3bf1b5" gracePeriod=30
Sep 16 19:33:20.252478 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:20.252444    2586 generic.go:356] "Generic (PLEG): container finished" podID="3d174115-ec34-4c0d-839e-6ec8276bb6a5" containerID="e55a9850fae859e9004109255808127ee06bf73b27828261420594fa0d3bf1b5" exitCode=2
Sep 16 19:33:20.252668 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:20.252523    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-scj7n" event={"ID":"3d174115-ec34-4c0d-839e-6ec8276bb6a5","Type":"ContainerDied","Data":"e55a9850fae859e9004109255808127ee06bf73b27828261420594fa0d3bf1b5"}
Sep 16 19:33:21.802310 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:21.802268    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-scj7n" podUID="3d174115-ec34-4c0d-839e-6ec8276bb6a5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.72:8643/healthz\": dial tcp 10.134.0.72:8643: connect: connection refused"
Sep 16 19:33:23.590777 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:23.590752    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-scj7n"
Sep 16 19:33:23.709068 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:23.708982    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-27kzv\" (UniqueName: \"kubernetes.io/projected/3d174115-ec34-4c0d-839e-6ec8276bb6a5-kube-api-access-27kzv\") pod \"3d174115-ec34-4c0d-839e-6ec8276bb6a5\" (UID: \"3d174115-ec34-4c0d-839e-6ec8276bb6a5\") "
Sep 16 19:33:23.709068 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:23.709032    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d174115-ec34-4c0d-839e-6ec8276bb6a5-proxy-tls\") pod \"3d174115-ec34-4c0d-839e-6ec8276bb6a5\" (UID: \"3d174115-ec34-4c0d-839e-6ec8276bb6a5\") "
Sep 16 19:33:23.709068 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:23.709070    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d174115-ec34-4c0d-839e-6ec8276bb6a5-kserve-provision-location\") pod \"3d174115-ec34-4c0d-839e-6ec8276bb6a5\" (UID: \"3d174115-ec34-4c0d-839e-6ec8276bb6a5\") "
Sep 16 19:33:23.709382 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:23.709092    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d174115-ec34-4c0d-839e-6ec8276bb6a5-kube-rbac-proxy-sar-config\") pod \"3d174115-ec34-4c0d-839e-6ec8276bb6a5\" (UID: \"3d174115-ec34-4c0d-839e-6ec8276bb6a5\") "
Sep 16 19:33:23.709382 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:23.709125    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d174115-ec34-4c0d-839e-6ec8276bb6a5-cabundle-cert\") pod \"3d174115-ec34-4c0d-839e-6ec8276bb6a5\" (UID: \"3d174115-ec34-4c0d-839e-6ec8276bb6a5\") "
Sep 16 19:33:23.709507 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:23.709478    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d174115-ec34-4c0d-839e-6ec8276bb6a5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3d174115-ec34-4c0d-839e-6ec8276bb6a5" (UID: "3d174115-ec34-4c0d-839e-6ec8276bb6a5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:33:23.709564 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:23.709540    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d174115-ec34-4c0d-839e-6ec8276bb6a5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3d174115-ec34-4c0d-839e-6ec8276bb6a5" (UID: "3d174115-ec34-4c0d-839e-6ec8276bb6a5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:33:23.709606 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:23.709544    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d174115-ec34-4c0d-839e-6ec8276bb6a5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3d174115-ec34-4c0d-839e-6ec8276bb6a5" (UID: "3d174115-ec34-4c0d-839e-6ec8276bb6a5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:33:23.711171 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:23.711146    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d174115-ec34-4c0d-839e-6ec8276bb6a5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3d174115-ec34-4c0d-839e-6ec8276bb6a5" (UID: "3d174115-ec34-4c0d-839e-6ec8276bb6a5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:33:23.711279 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:23.711190    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d174115-ec34-4c0d-839e-6ec8276bb6a5-kube-api-access-27kzv" (OuterVolumeSpecName: "kube-api-access-27kzv") pod "3d174115-ec34-4c0d-839e-6ec8276bb6a5" (UID: "3d174115-ec34-4c0d-839e-6ec8276bb6a5"). InnerVolumeSpecName "kube-api-access-27kzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:33:23.809839 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:23.809805    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27kzv\" (UniqueName: \"kubernetes.io/projected/3d174115-ec34-4c0d-839e-6ec8276bb6a5-kube-api-access-27kzv\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:33:23.809839 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:23.809837    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d174115-ec34-4c0d-839e-6ec8276bb6a5-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:33:23.810022 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:23.809849    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d174115-ec34-4c0d-839e-6ec8276bb6a5-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:33:23.810022 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:23.809859    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d174115-ec34-4c0d-839e-6ec8276bb6a5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:33:23.810022 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:23.809868    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3d174115-ec34-4c0d-839e-6ec8276bb6a5-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:33:24.273652 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:24.273620    2586 generic.go:356] "Generic (PLEG): container finished" podID="3d174115-ec34-4c0d-839e-6ec8276bb6a5" containerID="726da3149fc9163f65e670eb1ef9c5af5cbe8cb7440ae513e2f2420e3afd848f" exitCode=0
Sep 16 19:33:24.273841 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:24.273672    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-scj7n"
Sep 16 19:33:24.273841 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:24.273702    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-scj7n" event={"ID":"3d174115-ec34-4c0d-839e-6ec8276bb6a5","Type":"ContainerDied","Data":"726da3149fc9163f65e670eb1ef9c5af5cbe8cb7440ae513e2f2420e3afd848f"}
Sep 16 19:33:24.273841 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:24.273778    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-scj7n" event={"ID":"3d174115-ec34-4c0d-839e-6ec8276bb6a5","Type":"ContainerDied","Data":"94bec421b24502ad32de86bd06ec2b8fbc91222e90f10a9ebbf03d906444fcdd"}
Sep 16 19:33:24.273841 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:24.273818    2586 scope.go:117] "RemoveContainer" containerID="e55a9850fae859e9004109255808127ee06bf73b27828261420594fa0d3bf1b5"
Sep 16 19:33:24.282740 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:24.282724    2586 scope.go:117] "RemoveContainer" containerID="726da3149fc9163f65e670eb1ef9c5af5cbe8cb7440ae513e2f2420e3afd848f"
Sep 16 19:33:24.290331 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:24.290311    2586 scope.go:117] "RemoveContainer" containerID="37d4f9e374bdc6081a1084de4d8466bdc45f58e9b7f807aeb73dd778a318549c"
Sep 16 19:33:24.298993 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:24.298939    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-scj7n"]
Sep 16 19:33:24.299069 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:24.299034    2586 scope.go:117] "RemoveContainer" containerID="e55a9850fae859e9004109255808127ee06bf73b27828261420594fa0d3bf1b5"
Sep 16 19:33:24.299324 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:33:24.299306    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e55a9850fae859e9004109255808127ee06bf73b27828261420594fa0d3bf1b5\": container with ID starting with e55a9850fae859e9004109255808127ee06bf73b27828261420594fa0d3bf1b5 not found: ID does not exist" containerID="e55a9850fae859e9004109255808127ee06bf73b27828261420594fa0d3bf1b5"
Sep 16 19:33:24.299386 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:24.299331    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e55a9850fae859e9004109255808127ee06bf73b27828261420594fa0d3bf1b5"} err="failed to get container status \"e55a9850fae859e9004109255808127ee06bf73b27828261420594fa0d3bf1b5\": rpc error: code = NotFound desc = could not find container \"e55a9850fae859e9004109255808127ee06bf73b27828261420594fa0d3bf1b5\": container with ID starting with e55a9850fae859e9004109255808127ee06bf73b27828261420594fa0d3bf1b5 not found: ID does not exist"
Sep 16 19:33:24.299386 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:24.299347    2586 scope.go:117] "RemoveContainer" containerID="726da3149fc9163f65e670eb1ef9c5af5cbe8cb7440ae513e2f2420e3afd848f"
Sep 16 19:33:24.299595 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:33:24.299577    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"726da3149fc9163f65e670eb1ef9c5af5cbe8cb7440ae513e2f2420e3afd848f\": container with ID starting with 726da3149fc9163f65e670eb1ef9c5af5cbe8cb7440ae513e2f2420e3afd848f not found: ID does not exist" containerID="726da3149fc9163f65e670eb1ef9c5af5cbe8cb7440ae513e2f2420e3afd848f"
Sep 16 19:33:24.299642 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:24.299601    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"726da3149fc9163f65e670eb1ef9c5af5cbe8cb7440ae513e2f2420e3afd848f"} err="failed to get container status \"726da3149fc9163f65e670eb1ef9c5af5cbe8cb7440ae513e2f2420e3afd848f\": rpc error: code = NotFound desc = could not find container \"726da3149fc9163f65e670eb1ef9c5af5cbe8cb7440ae513e2f2420e3afd848f\": container with ID starting with 726da3149fc9163f65e670eb1ef9c5af5cbe8cb7440ae513e2f2420e3afd848f not found: ID does not exist"
Sep 16 19:33:24.299642 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:24.299616    2586 scope.go:117] "RemoveContainer" containerID="37d4f9e374bdc6081a1084de4d8466bdc45f58e9b7f807aeb73dd778a318549c"
Sep 16 19:33:24.299859 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:33:24.299841    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37d4f9e374bdc6081a1084de4d8466bdc45f58e9b7f807aeb73dd778a318549c\": container with ID starting with 37d4f9e374bdc6081a1084de4d8466bdc45f58e9b7f807aeb73dd778a318549c not found: ID does not exist" containerID="37d4f9e374bdc6081a1084de4d8466bdc45f58e9b7f807aeb73dd778a318549c"
Sep 16 19:33:24.299897 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:24.299862    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37d4f9e374bdc6081a1084de4d8466bdc45f58e9b7f807aeb73dd778a318549c"} err="failed to get container status \"37d4f9e374bdc6081a1084de4d8466bdc45f58e9b7f807aeb73dd778a318549c\": rpc error: code = NotFound desc = could not find container \"37d4f9e374bdc6081a1084de4d8466bdc45f58e9b7f807aeb73dd778a318549c\": container with ID starting with 37d4f9e374bdc6081a1084de4d8466bdc45f58e9b7f807aeb73dd778a318549c not found: ID does not exist"
Sep 16 19:33:24.302657 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:24.302636    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-scj7n"]
Sep 16 19:33:25.052549 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:25.052512    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt"]
Sep 16 19:33:25.052937 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:25.052835    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d174115-ec34-4c0d-839e-6ec8276bb6a5" containerName="storage-initializer"
Sep 16 19:33:25.052937 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:25.052846    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d174115-ec34-4c0d-839e-6ec8276bb6a5" containerName="storage-initializer"
Sep 16 19:33:25.052937 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:25.052864    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d174115-ec34-4c0d-839e-6ec8276bb6a5" containerName="kube-rbac-proxy"
Sep 16 19:33:25.052937 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:25.052869    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d174115-ec34-4c0d-839e-6ec8276bb6a5" containerName="kube-rbac-proxy"
Sep 16 19:33:25.052937 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:25.052887    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d174115-ec34-4c0d-839e-6ec8276bb6a5" containerName="kserve-container"
Sep 16 19:33:25.052937 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:25.052893    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d174115-ec34-4c0d-839e-6ec8276bb6a5" containerName="kserve-container"
Sep 16 19:33:25.053202 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:25.052945    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d174115-ec34-4c0d-839e-6ec8276bb6a5" containerName="kube-rbac-proxy"
Sep 16 19:33:25.053202 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:25.052956    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d174115-ec34-4c0d-839e-6ec8276bb6a5" containerName="kserve-container"
Sep 16 19:33:25.057736 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:25.057714    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt"
Sep 16 19:33:25.060556 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:25.060535    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
Sep 16 19:33:25.061191 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:25.061163    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 16 19:33:25.067963 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:25.067942    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt"]
Sep 16 19:33:25.093826 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:25.093800    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d174115-ec34-4c0d-839e-6ec8276bb6a5" path="/var/lib/kubelet/pods/3d174115-ec34-4c0d-839e-6ec8276bb6a5/volumes"
Sep 16 19:33:25.120927 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:25.120896    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4ddb4a1-8d29-408b-abea-9bfb76725409-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt\" (UID: \"a4ddb4a1-8d29-408b-abea-9bfb76725409\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt"
Sep 16 19:33:25.121064 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:25.120944    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4ddb4a1-8d29-408b-abea-9bfb76725409-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt\" (UID: \"a4ddb4a1-8d29-408b-abea-9bfb76725409\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt"
Sep 16 19:33:25.121064 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:25.120972    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4ddb4a1-8d29-408b-abea-9bfb76725409-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt\" (UID: \"a4ddb4a1-8d29-408b-abea-9bfb76725409\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt"
Sep 16 19:33:25.121064 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:25.121036    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4ddb4a1-8d29-408b-abea-9bfb76725409-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt\" (UID: \"a4ddb4a1-8d29-408b-abea-9bfb76725409\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt"
Sep 16 19:33:25.121220 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:25.121094    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9htn\" (UniqueName: \"kubernetes.io/projected/a4ddb4a1-8d29-408b-abea-9bfb76725409-kube-api-access-s9htn\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt\" (UID: \"a4ddb4a1-8d29-408b-abea-9bfb76725409\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt"
Sep 16 19:33:25.222106 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:25.222069    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9htn\" (UniqueName: \"kubernetes.io/projected/a4ddb4a1-8d29-408b-abea-9bfb76725409-kube-api-access-s9htn\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt\" (UID: \"a4ddb4a1-8d29-408b-abea-9bfb76725409\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt"
Sep 16 19:33:25.222513 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:25.222467    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4ddb4a1-8d29-408b-abea-9bfb76725409-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt\" (UID: \"a4ddb4a1-8d29-408b-abea-9bfb76725409\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt"
Sep 16 19:33:25.222652 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:25.222549    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4ddb4a1-8d29-408b-abea-9bfb76725409-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt\" (UID: \"a4ddb4a1-8d29-408b-abea-9bfb76725409\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt"
Sep 16 19:33:25.222652 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:25.222583    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4ddb4a1-8d29-408b-abea-9bfb76725409-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt\" (UID: \"a4ddb4a1-8d29-408b-abea-9bfb76725409\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt"
Sep 16 19:33:25.222652 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:25.222625    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4ddb4a1-8d29-408b-abea-9bfb76725409-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt\" (UID: \"a4ddb4a1-8d29-408b-abea-9bfb76725409\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt"
Sep 16 19:33:25.222843 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:25.222772    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4ddb4a1-8d29-408b-abea-9bfb76725409-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt\" (UID: \"a4ddb4a1-8d29-408b-abea-9bfb76725409\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt"
Sep 16 19:33:25.223273 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:25.223224    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4ddb4a1-8d29-408b-abea-9bfb76725409-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt\" (UID: \"a4ddb4a1-8d29-408b-abea-9bfb76725409\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt"
Sep 16 19:33:25.223364 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:25.223253    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4ddb4a1-8d29-408b-abea-9bfb76725409-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt\" (UID: \"a4ddb4a1-8d29-408b-abea-9bfb76725409\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt"
Sep 16 19:33:25.224977 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:25.224957    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4ddb4a1-8d29-408b-abea-9bfb76725409-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt\" (UID: \"a4ddb4a1-8d29-408b-abea-9bfb76725409\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt"
Sep 16 19:33:25.232753 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:25.232730    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9htn\" (UniqueName: \"kubernetes.io/projected/a4ddb4a1-8d29-408b-abea-9bfb76725409-kube-api-access-s9htn\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt\" (UID: \"a4ddb4a1-8d29-408b-abea-9bfb76725409\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt"
Sep 16 19:33:25.369351 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:25.369319    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt"
Sep 16 19:33:25.649382 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:25.649191    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt"]
Sep 16 19:33:25.651652 ip-10-0-129-190 kubenswrapper[2586]: W0916 19:33:25.651625    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4ddb4a1_8d29_408b_abea_9bfb76725409.slice/crio-1667ada0681bdd2cccd83d3666377a52623effe5893b2d018df2ee911ab36450 WatchSource:0}: Error finding container 1667ada0681bdd2cccd83d3666377a52623effe5893b2d018df2ee911ab36450: Status 404 returned error can't find the container with id 1667ada0681bdd2cccd83d3666377a52623effe5893b2d018df2ee911ab36450
Sep 16 19:33:26.286741 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:26.286703    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt" event={"ID":"a4ddb4a1-8d29-408b-abea-9bfb76725409","Type":"ContainerStarted","Data":"c392c69ed24ecafc72e10ae6d4e95c50e86ff5b54449f8cbc0bf969a0562b6af"}
Sep 16 19:33:26.287129 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:26.286763    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt" event={"ID":"a4ddb4a1-8d29-408b-abea-9bfb76725409","Type":"ContainerStarted","Data":"1667ada0681bdd2cccd83d3666377a52623effe5893b2d018df2ee911ab36450"}
Sep 16 19:33:27.175336 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:27.175297    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-63d815-predictor-54888c955d-rw7wt" podUID="f20bd9fa-33f6-456e-aeb1-49f242479ed5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.73:8080: connect: connection refused"
Sep 16 19:33:30.306887 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:30.306853    2586 generic.go:356] "Generic (PLEG): container finished" podID="a4ddb4a1-8d29-408b-abea-9bfb76725409" containerID="c392c69ed24ecafc72e10ae6d4e95c50e86ff5b54449f8cbc0bf969a0562b6af" exitCode=0
Sep 16 19:33:30.307278 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:30.306928    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt" event={"ID":"a4ddb4a1-8d29-408b-abea-9bfb76725409","Type":"ContainerDied","Data":"c392c69ed24ecafc72e10ae6d4e95c50e86ff5b54449f8cbc0bf969a0562b6af"}
Sep 16 19:33:31.313838 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:31.313803    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt" event={"ID":"a4ddb4a1-8d29-408b-abea-9bfb76725409","Type":"ContainerStarted","Data":"39575122bdcdc892538b1b8b8721dc137d8de3d932a8588855778da524f1cd3a"}
Sep 16 19:33:31.314329 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:31.313879    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt" event={"ID":"a4ddb4a1-8d29-408b-abea-9bfb76725409","Type":"ContainerStarted","Data":"4f922868788d5776a6843757057e1ff7047d1ac5e45e45c99fcd3157eabc7e95"}
Sep 16 19:33:31.314329 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:31.314115    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt"
Sep 16 19:33:31.340692 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:31.340638    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt" podStartSLOduration=2.5625936190000003 podStartE2EDuration="6.340624388s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.778030769s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:33:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:33:31.339311499 +0000 UTC m=+3186.783173075" watchObservedRunningTime="2026-09-16 19:33:31.340624388 +0000 UTC m=+3186.784485965"
Sep 16 19:33:32.319287 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:32.319256    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt"
Sep 16 19:33:32.320532 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:32.320506    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt" podUID="a4ddb4a1-8d29-408b-abea-9bfb76725409" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.74:8080: connect: connection refused"
Sep 16 19:33:33.323971 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:33.323928    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt" podUID="a4ddb4a1-8d29-408b-abea-9bfb76725409" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.74:8080: connect: connection refused"
Sep 16 19:33:37.176244 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:37.176189    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-63d815-predictor-54888c955d-rw7wt" podUID="f20bd9fa-33f6-456e-aeb1-49f242479ed5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.73:8080: connect: connection refused"
Sep 16 19:33:38.328694 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:38.328659    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt"
Sep 16 19:33:38.329313 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:38.329278    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt" podUID="a4ddb4a1-8d29-408b-abea-9bfb76725409" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.74:8080: connect: connection refused"
Sep 16 19:33:47.175468 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:47.175430    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-63d815-predictor-54888c955d-rw7wt" podUID="f20bd9fa-33f6-456e-aeb1-49f242479ed5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.73:8080: connect: connection refused"
Sep 16 19:33:48.329505 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:48.329459    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt" podUID="a4ddb4a1-8d29-408b-abea-9bfb76725409" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.74:8080: connect: connection refused"
Sep 16 19:33:57.176203 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:57.176155    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-63d815-predictor-54888c955d-rw7wt" podUID="f20bd9fa-33f6-456e-aeb1-49f242479ed5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.73:8080: connect: connection refused"
Sep 16 19:33:58.329872 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:33:58.329816    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt" podUID="a4ddb4a1-8d29-408b-abea-9bfb76725409" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.74:8080: connect: connection refused"
Sep 16 19:34:07.175638 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:34:07.175595    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-63d815-predictor-54888c955d-rw7wt" podUID="f20bd9fa-33f6-456e-aeb1-49f242479ed5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.73:8080: connect: connection refused"
Sep 16 19:34:08.329623 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:34:08.329568    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt" podUID="a4ddb4a1-8d29-408b-abea-9bfb76725409" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.74:8080: connect: connection refused"
Sep 16 19:34:17.176404 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:34:17.176373    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-63d815-predictor-54888c955d-rw7wt"
Sep 16 19:34:18.330086 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:34:18.330046    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt" podUID="a4ddb4a1-8d29-408b-abea-9bfb76725409" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.74:8080: connect: connection refused"
Sep 16 19:34:28.329369 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:34:28.329327    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt" podUID="a4ddb4a1-8d29-408b-abea-9bfb76725409" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.74:8080: connect: connection refused"
Sep 16 19:34:38.329735 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:34:38.329686    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt" podUID="a4ddb4a1-8d29-408b-abea-9bfb76725409" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.74:8080: connect: connection refused"
Sep 16 19:34:48.330013 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:34:48.329961    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt" podUID="a4ddb4a1-8d29-408b-abea-9bfb76725409" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.74:8080: connect: connection refused"
Sep 16 19:34:52.087388 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:34:52.087358    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt"
Sep 16 19:34:56.092355 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:34:56.092322    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt"]
Sep 16 19:34:56.092751 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:34:56.092568    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt" podUID="a4ddb4a1-8d29-408b-abea-9bfb76725409" containerName="kserve-container" containerID="cri-o://4f922868788d5776a6843757057e1ff7047d1ac5e45e45c99fcd3157eabc7e95" gracePeriod=30
Sep 16 19:34:56.092751 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:34:56.092627    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt" podUID="a4ddb4a1-8d29-408b-abea-9bfb76725409" containerName="kube-rbac-proxy" containerID="cri-o://39575122bdcdc892538b1b8b8721dc137d8de3d932a8588855778da524f1cd3a" gracePeriod=30
Sep 16 19:34:56.702892 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:34:56.702860    2586 generic.go:356] "Generic (PLEG): container finished" podID="a4ddb4a1-8d29-408b-abea-9bfb76725409" containerID="39575122bdcdc892538b1b8b8721dc137d8de3d932a8588855778da524f1cd3a" exitCode=2
Sep 16 19:34:56.703073 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:34:56.702935    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt" event={"ID":"a4ddb4a1-8d29-408b-abea-9bfb76725409","Type":"ContainerDied","Data":"39575122bdcdc892538b1b8b8721dc137d8de3d932a8588855778da524f1cd3a"}
Sep 16 19:34:58.253298 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:34:58.253257    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-63d815-predictor-54888c955d-rw7wt"]
Sep 16 19:34:58.253714 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:34:58.253503    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-63d815-predictor-54888c955d-rw7wt" podUID="f20bd9fa-33f6-456e-aeb1-49f242479ed5" containerName="kserve-container" containerID="cri-o://ed32e43470645a2079a11e8007b6d9d3ad9d69a2a7a4073213f93de639224242" gracePeriod=30
Sep 16 19:34:58.253714 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:34:58.253539    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-63d815-predictor-54888c955d-rw7wt" podUID="f20bd9fa-33f6-456e-aeb1-49f242479ed5" containerName="kube-rbac-proxy" containerID="cri-o://5bc916ba328055e96f40b82af5372edbac4a4214bf29d7d1b0de7c5c16da5525" gracePeriod=30
Sep 16 19:34:58.325087 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:34:58.325047    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt" podUID="a4ddb4a1-8d29-408b-abea-9bfb76725409" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.74:8643/healthz\": dial tcp 10.134.0.74:8643: connect: connection refused"
Sep 16 19:34:58.714498 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:34:58.714468    2586 generic.go:356] "Generic (PLEG): container finished" podID="f20bd9fa-33f6-456e-aeb1-49f242479ed5" containerID="5bc916ba328055e96f40b82af5372edbac4a4214bf29d7d1b0de7c5c16da5525" exitCode=2
Sep 16 19:34:58.714498 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:34:58.714507    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-63d815-predictor-54888c955d-rw7wt" event={"ID":"f20bd9fa-33f6-456e-aeb1-49f242479ed5","Type":"ContainerDied","Data":"5bc916ba328055e96f40b82af5372edbac4a4214bf29d7d1b0de7c5c16da5525"}
Sep 16 19:34:59.945521 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:34:59.945489    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt"
Sep 16 19:35:00.073442 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:00.073349    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4ddb4a1-8d29-408b-abea-9bfb76725409-kube-rbac-proxy-sar-config\") pod \"a4ddb4a1-8d29-408b-abea-9bfb76725409\" (UID: \"a4ddb4a1-8d29-408b-abea-9bfb76725409\") "
Sep 16 19:35:00.073442 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:00.073431    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4ddb4a1-8d29-408b-abea-9bfb76725409-cabundle-cert\") pod \"a4ddb4a1-8d29-408b-abea-9bfb76725409\" (UID: \"a4ddb4a1-8d29-408b-abea-9bfb76725409\") "
Sep 16 19:35:00.073712 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:00.073517    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4ddb4a1-8d29-408b-abea-9bfb76725409-kserve-provision-location\") pod \"a4ddb4a1-8d29-408b-abea-9bfb76725409\" (UID: \"a4ddb4a1-8d29-408b-abea-9bfb76725409\") "
Sep 16 19:35:00.073712 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:00.073548    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9htn\" (UniqueName: \"kubernetes.io/projected/a4ddb4a1-8d29-408b-abea-9bfb76725409-kube-api-access-s9htn\") pod \"a4ddb4a1-8d29-408b-abea-9bfb76725409\" (UID: \"a4ddb4a1-8d29-408b-abea-9bfb76725409\") "
Sep 16 19:35:00.073712 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:00.073581    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4ddb4a1-8d29-408b-abea-9bfb76725409-proxy-tls\") pod \"a4ddb4a1-8d29-408b-abea-9bfb76725409\" (UID: \"a4ddb4a1-8d29-408b-abea-9bfb76725409\") "
Sep 16 19:35:00.073884 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:00.073843    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4ddb4a1-8d29-408b-abea-9bfb76725409-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a4ddb4a1-8d29-408b-abea-9bfb76725409" (UID: "a4ddb4a1-8d29-408b-abea-9bfb76725409"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:35:00.073945 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:00.073900    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4ddb4a1-8d29-408b-abea-9bfb76725409-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a4ddb4a1-8d29-408b-abea-9bfb76725409" (UID: "a4ddb4a1-8d29-408b-abea-9bfb76725409"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:35:00.073945 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:00.073922    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4ddb4a1-8d29-408b-abea-9bfb76725409-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a4ddb4a1-8d29-408b-abea-9bfb76725409" (UID: "a4ddb4a1-8d29-408b-abea-9bfb76725409"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:35:00.075802 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:00.075779    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4ddb4a1-8d29-408b-abea-9bfb76725409-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a4ddb4a1-8d29-408b-abea-9bfb76725409" (UID: "a4ddb4a1-8d29-408b-abea-9bfb76725409"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:35:00.075904 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:00.075797    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4ddb4a1-8d29-408b-abea-9bfb76725409-kube-api-access-s9htn" (OuterVolumeSpecName: "kube-api-access-s9htn") pod "a4ddb4a1-8d29-408b-abea-9bfb76725409" (UID: "a4ddb4a1-8d29-408b-abea-9bfb76725409"). InnerVolumeSpecName "kube-api-access-s9htn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:35:00.174953 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:00.174909    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4ddb4a1-8d29-408b-abea-9bfb76725409-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:35:00.174953 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:00.174947    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4ddb4a1-8d29-408b-abea-9bfb76725409-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:35:00.174953 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:00.174958    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9htn\" (UniqueName: \"kubernetes.io/projected/a4ddb4a1-8d29-408b-abea-9bfb76725409-kube-api-access-s9htn\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:35:00.175193 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:00.174967    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4ddb4a1-8d29-408b-abea-9bfb76725409-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:35:00.175193 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:00.174976    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4ddb4a1-8d29-408b-abea-9bfb76725409-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:35:00.725837 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:00.725803    2586 generic.go:356] "Generic (PLEG): container finished" podID="a4ddb4a1-8d29-408b-abea-9bfb76725409" containerID="4f922868788d5776a6843757057e1ff7047d1ac5e45e45c99fcd3157eabc7e95" exitCode=0
Sep 16 19:35:00.726062 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:00.725850    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt" event={"ID":"a4ddb4a1-8d29-408b-abea-9bfb76725409","Type":"ContainerDied","Data":"4f922868788d5776a6843757057e1ff7047d1ac5e45e45c99fcd3157eabc7e95"}
Sep 16 19:35:00.726062 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:00.725880    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt"
Sep 16 19:35:00.726062 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:00.725897    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt" event={"ID":"a4ddb4a1-8d29-408b-abea-9bfb76725409","Type":"ContainerDied","Data":"1667ada0681bdd2cccd83d3666377a52623effe5893b2d018df2ee911ab36450"}
Sep 16 19:35:00.726062 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:00.725923    2586 scope.go:117] "RemoveContainer" containerID="39575122bdcdc892538b1b8b8721dc137d8de3d932a8588855778da524f1cd3a"
Sep 16 19:35:00.735453 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:00.735432    2586 scope.go:117] "RemoveContainer" containerID="4f922868788d5776a6843757057e1ff7047d1ac5e45e45c99fcd3157eabc7e95"
Sep 16 19:35:00.744027 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:00.744008    2586 scope.go:117] "RemoveContainer" containerID="c392c69ed24ecafc72e10ae6d4e95c50e86ff5b54449f8cbc0bf969a0562b6af"
Sep 16 19:35:00.751661 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:00.751643    2586 scope.go:117] "RemoveContainer" containerID="39575122bdcdc892538b1b8b8721dc137d8de3d932a8588855778da524f1cd3a"
Sep 16 19:35:00.751978 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:35:00.751959    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39575122bdcdc892538b1b8b8721dc137d8de3d932a8588855778da524f1cd3a\": container with ID starting with 39575122bdcdc892538b1b8b8721dc137d8de3d932a8588855778da524f1cd3a not found: ID does not exist" containerID="39575122bdcdc892538b1b8b8721dc137d8de3d932a8588855778da524f1cd3a"
Sep 16 19:35:00.752033 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:00.751987    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39575122bdcdc892538b1b8b8721dc137d8de3d932a8588855778da524f1cd3a"} err="failed to get container status \"39575122bdcdc892538b1b8b8721dc137d8de3d932a8588855778da524f1cd3a\": rpc error: code = NotFound desc = could not find container \"39575122bdcdc892538b1b8b8721dc137d8de3d932a8588855778da524f1cd3a\": container with ID starting with 39575122bdcdc892538b1b8b8721dc137d8de3d932a8588855778da524f1cd3a not found: ID does not exist"
Sep 16 19:35:00.752033 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:00.752005    2586 scope.go:117] "RemoveContainer" containerID="4f922868788d5776a6843757057e1ff7047d1ac5e45e45c99fcd3157eabc7e95"
Sep 16 19:35:00.752327 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:35:00.752304    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f922868788d5776a6843757057e1ff7047d1ac5e45e45c99fcd3157eabc7e95\": container with ID starting with 4f922868788d5776a6843757057e1ff7047d1ac5e45e45c99fcd3157eabc7e95 not found: ID does not exist" containerID="4f922868788d5776a6843757057e1ff7047d1ac5e45e45c99fcd3157eabc7e95"
Sep 16 19:35:00.752377 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:00.752335    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f922868788d5776a6843757057e1ff7047d1ac5e45e45c99fcd3157eabc7e95"} err="failed to get container status \"4f922868788d5776a6843757057e1ff7047d1ac5e45e45c99fcd3157eabc7e95\": rpc error: code = NotFound desc = could not find container \"4f922868788d5776a6843757057e1ff7047d1ac5e45e45c99fcd3157eabc7e95\": container with ID starting with 4f922868788d5776a6843757057e1ff7047d1ac5e45e45c99fcd3157eabc7e95 not found: ID does not exist"
Sep 16 19:35:00.752377 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:00.752355    2586 scope.go:117] "RemoveContainer" containerID="c392c69ed24ecafc72e10ae6d4e95c50e86ff5b54449f8cbc0bf969a0562b6af"
Sep 16 19:35:00.752617 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:35:00.752599    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c392c69ed24ecafc72e10ae6d4e95c50e86ff5b54449f8cbc0bf969a0562b6af\": container with ID starting with c392c69ed24ecafc72e10ae6d4e95c50e86ff5b54449f8cbc0bf969a0562b6af not found: ID does not exist" containerID="c392c69ed24ecafc72e10ae6d4e95c50e86ff5b54449f8cbc0bf969a0562b6af"
Sep 16 19:35:00.752669 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:00.752622    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c392c69ed24ecafc72e10ae6d4e95c50e86ff5b54449f8cbc0bf969a0562b6af"} err="failed to get container status \"c392c69ed24ecafc72e10ae6d4e95c50e86ff5b54449f8cbc0bf969a0562b6af\": rpc error: code = NotFound desc = could not find container \"c392c69ed24ecafc72e10ae6d4e95c50e86ff5b54449f8cbc0bf969a0562b6af\": container with ID starting with c392c69ed24ecafc72e10ae6d4e95c50e86ff5b54449f8cbc0bf969a0562b6af not found: ID does not exist"
Sep 16 19:35:00.753138 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:00.753121    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt"]
Sep 16 19:35:00.758723 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:00.758701    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5pgt"]
Sep 16 19:35:01.092520 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:01.092442    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4ddb4a1-8d29-408b-abea-9bfb76725409" path="/var/lib/kubelet/pods/a4ddb4a1-8d29-408b-abea-9bfb76725409/volumes"
Sep 16 19:35:02.170756 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:02.170710    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-63d815-predictor-54888c955d-rw7wt" podUID="f20bd9fa-33f6-456e-aeb1-49f242479ed5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.73:8643/healthz\": dial tcp 10.134.0.73:8643: connect: connection refused"
Sep 16 19:35:03.402468 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:03.402444    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-63d815-predictor-54888c955d-rw7wt"
Sep 16 19:35:03.504711 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:03.504621    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5chb2\" (UniqueName: \"kubernetes.io/projected/f20bd9fa-33f6-456e-aeb1-49f242479ed5-kube-api-access-5chb2\") pod \"f20bd9fa-33f6-456e-aeb1-49f242479ed5\" (UID: \"f20bd9fa-33f6-456e-aeb1-49f242479ed5\") "
Sep 16 19:35:03.504711 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:03.504654    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f20bd9fa-33f6-456e-aeb1-49f242479ed5-kube-rbac-proxy-sar-config\") pod \"f20bd9fa-33f6-456e-aeb1-49f242479ed5\" (UID: \"f20bd9fa-33f6-456e-aeb1-49f242479ed5\") "
Sep 16 19:35:03.504711 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:03.504692    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f20bd9fa-33f6-456e-aeb1-49f242479ed5-proxy-tls\") pod \"f20bd9fa-33f6-456e-aeb1-49f242479ed5\" (UID: \"f20bd9fa-33f6-456e-aeb1-49f242479ed5\") "
Sep 16 19:35:03.504970 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:03.504858    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f20bd9fa-33f6-456e-aeb1-49f242479ed5-kserve-provision-location\") pod \"f20bd9fa-33f6-456e-aeb1-49f242479ed5\" (UID: \"f20bd9fa-33f6-456e-aeb1-49f242479ed5\") "
Sep 16 19:35:03.504970 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:03.504958    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f20bd9fa-33f6-456e-aeb1-49f242479ed5-cabundle-cert\") pod \"f20bd9fa-33f6-456e-aeb1-49f242479ed5\" (UID: \"f20bd9fa-33f6-456e-aeb1-49f242479ed5\") "
Sep 16 19:35:03.505091 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:03.505060    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f20bd9fa-33f6-456e-aeb1-49f242479ed5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f20bd9fa-33f6-456e-aeb1-49f242479ed5" (UID: "f20bd9fa-33f6-456e-aeb1-49f242479ed5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:35:03.505388 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:03.505365    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f20bd9fa-33f6-456e-aeb1-49f242479ed5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f20bd9fa-33f6-456e-aeb1-49f242479ed5" (UID: "f20bd9fa-33f6-456e-aeb1-49f242479ed5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:35:03.505388 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:03.505383    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f20bd9fa-33f6-456e-aeb1-49f242479ed5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:35:03.505553 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:03.505498    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f20bd9fa-33f6-456e-aeb1-49f242479ed5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f20bd9fa-33f6-456e-aeb1-49f242479ed5" (UID: "f20bd9fa-33f6-456e-aeb1-49f242479ed5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:35:03.506833 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:03.506816    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f20bd9fa-33f6-456e-aeb1-49f242479ed5-kube-api-access-5chb2" (OuterVolumeSpecName: "kube-api-access-5chb2") pod "f20bd9fa-33f6-456e-aeb1-49f242479ed5" (UID: "f20bd9fa-33f6-456e-aeb1-49f242479ed5"). InnerVolumeSpecName "kube-api-access-5chb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:35:03.506909 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:03.506895    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f20bd9fa-33f6-456e-aeb1-49f242479ed5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f20bd9fa-33f6-456e-aeb1-49f242479ed5" (UID: "f20bd9fa-33f6-456e-aeb1-49f242479ed5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:35:03.606014 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:03.605982    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f20bd9fa-33f6-456e-aeb1-49f242479ed5-cabundle-cert\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:35:03.606014 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:03.606010    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5chb2\" (UniqueName: \"kubernetes.io/projected/f20bd9fa-33f6-456e-aeb1-49f242479ed5-kube-api-access-5chb2\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:35:03.606014 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:03.606022    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f20bd9fa-33f6-456e-aeb1-49f242479ed5-proxy-tls\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:35:03.606317 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:03.606032    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f20bd9fa-33f6-456e-aeb1-49f242479ed5-kserve-provision-location\") on node \"ip-10-0-129-190.ec2.internal\" DevicePath \"\""
Sep 16 19:35:03.744697 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:03.744660    2586 generic.go:356] "Generic (PLEG): container finished" podID="f20bd9fa-33f6-456e-aeb1-49f242479ed5" containerID="ed32e43470645a2079a11e8007b6d9d3ad9d69a2a7a4073213f93de639224242" exitCode=0
Sep 16 19:35:03.744697 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:03.744701    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-63d815-predictor-54888c955d-rw7wt" event={"ID":"f20bd9fa-33f6-456e-aeb1-49f242479ed5","Type":"ContainerDied","Data":"ed32e43470645a2079a11e8007b6d9d3ad9d69a2a7a4073213f93de639224242"}
Sep 16 19:35:03.744900 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:03.744721    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-63d815-predictor-54888c955d-rw7wt"
Sep 16 19:35:03.744900 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:03.744748    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-63d815-predictor-54888c955d-rw7wt" event={"ID":"f20bd9fa-33f6-456e-aeb1-49f242479ed5","Type":"ContainerDied","Data":"4027487b5e7f5075c9feab59a14e61ab5fd5edc27ddd2f750ed677355558a5c9"}
Sep 16 19:35:03.744900 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:03.744776    2586 scope.go:117] "RemoveContainer" containerID="5bc916ba328055e96f40b82af5372edbac4a4214bf29d7d1b0de7c5c16da5525"
Sep 16 19:35:03.753276 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:03.753255    2586 scope.go:117] "RemoveContainer" containerID="ed32e43470645a2079a11e8007b6d9d3ad9d69a2a7a4073213f93de639224242"
Sep 16 19:35:03.760469 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:03.760449    2586 scope.go:117] "RemoveContainer" containerID="119d9f84351e8bd7a38ec73e349e799aaf18fc6178ca7ad9205c971bf7fc2c66"
Sep 16 19:35:03.767819 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:03.767801    2586 scope.go:117] "RemoveContainer" containerID="5bc916ba328055e96f40b82af5372edbac4a4214bf29d7d1b0de7c5c16da5525"
Sep 16 19:35:03.768064 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:35:03.768042    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bc916ba328055e96f40b82af5372edbac4a4214bf29d7d1b0de7c5c16da5525\": container with ID starting with 5bc916ba328055e96f40b82af5372edbac4a4214bf29d7d1b0de7c5c16da5525 not found: ID does not exist" containerID="5bc916ba328055e96f40b82af5372edbac4a4214bf29d7d1b0de7c5c16da5525"
Sep 16 19:35:03.768143 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:03.768076    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bc916ba328055e96f40b82af5372edbac4a4214bf29d7d1b0de7c5c16da5525"} err="failed to get container status \"5bc916ba328055e96f40b82af5372edbac4a4214bf29d7d1b0de7c5c16da5525\": rpc error: code = NotFound desc = could not find container \"5bc916ba328055e96f40b82af5372edbac4a4214bf29d7d1b0de7c5c16da5525\": container with ID starting with 5bc916ba328055e96f40b82af5372edbac4a4214bf29d7d1b0de7c5c16da5525 not found: ID does not exist"
Sep 16 19:35:03.768143 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:03.768098    2586 scope.go:117] "RemoveContainer" containerID="ed32e43470645a2079a11e8007b6d9d3ad9d69a2a7a4073213f93de639224242"
Sep 16 19:35:03.768400 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:35:03.768379    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed32e43470645a2079a11e8007b6d9d3ad9d69a2a7a4073213f93de639224242\": container with ID starting with ed32e43470645a2079a11e8007b6d9d3ad9d69a2a7a4073213f93de639224242 not found: ID does not exist" containerID="ed32e43470645a2079a11e8007b6d9d3ad9d69a2a7a4073213f93de639224242"
Sep 16 19:35:03.768449 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:03.768409    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed32e43470645a2079a11e8007b6d9d3ad9d69a2a7a4073213f93de639224242"} err="failed to get container status \"ed32e43470645a2079a11e8007b6d9d3ad9d69a2a7a4073213f93de639224242\": rpc error: code = NotFound desc = could not find container \"ed32e43470645a2079a11e8007b6d9d3ad9d69a2a7a4073213f93de639224242\": container with ID starting with ed32e43470645a2079a11e8007b6d9d3ad9d69a2a7a4073213f93de639224242 not found: ID does not exist"
Sep 16 19:35:03.768449 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:03.768425    2586 scope.go:117] "RemoveContainer" containerID="119d9f84351e8bd7a38ec73e349e799aaf18fc6178ca7ad9205c971bf7fc2c66"
Sep 16 19:35:03.768701 ip-10-0-129-190 kubenswrapper[2586]: E0916 19:35:03.768680    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"119d9f84351e8bd7a38ec73e349e799aaf18fc6178ca7ad9205c971bf7fc2c66\": container with ID starting with 119d9f84351e8bd7a38ec73e349e799aaf18fc6178ca7ad9205c971bf7fc2c66 not found: ID does not exist" containerID="119d9f84351e8bd7a38ec73e349e799aaf18fc6178ca7ad9205c971bf7fc2c66"
Sep 16 19:35:03.768789 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:03.768704    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"119d9f84351e8bd7a38ec73e349e799aaf18fc6178ca7ad9205c971bf7fc2c66"} err="failed to get container status \"119d9f84351e8bd7a38ec73e349e799aaf18fc6178ca7ad9205c971bf7fc2c66\": rpc error: code = NotFound desc = could not find container \"119d9f84351e8bd7a38ec73e349e799aaf18fc6178ca7ad9205c971bf7fc2c66\": container with ID starting with 119d9f84351e8bd7a38ec73e349e799aaf18fc6178ca7ad9205c971bf7fc2c66 not found: ID does not exist"
Sep 16 19:35:03.771634 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:03.771612    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-63d815-predictor-54888c955d-rw7wt"]
Sep 16 19:35:03.774285 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:03.774266    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-63d815-predictor-54888c955d-rw7wt"]
Sep 16 19:35:05.096481 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:05.096447    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f20bd9fa-33f6-456e-aeb1-49f242479ed5" path="/var/lib/kubelet/pods/f20bd9fa-33f6-456e-aeb1-49f242479ed5/volumes"
Sep 16 19:35:45.321937 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:45.321903    2586 ???:1] "http: TLS handshake error from 10.0.129.50:47252: EOF"
Sep 16 19:35:45.327820 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:45.327798    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8ktbc_8c42d544-c24c-4b4b-b88d-13cc74d9fa87/global-pull-secret-syncer/0.log"
Sep 16 19:35:45.564924 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:45.564891    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-7n8dt_6f68d3ed-0908-4d8a-a879-5b002a923ca9/konnectivity-agent/0.log"
Sep 16 19:35:45.640568 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:45.640540    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-190.ec2.internal_aacec50d8c91383d21cd8b8f8d4aaf7a/haproxy/0.log"
Sep 16 19:35:49.736275 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:49.736242    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-mggrx_f01001cd-d72f-4267-8e14-44359c95f66c/cluster-monitoring-operator/0.log"
Sep 16 19:35:50.058847 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:50.058818    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rdxf6_0ac2e187-f76e-4497-82dd-c8b902153ace/node-exporter/0.log"
Sep 16 19:35:50.091800 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:50.091779    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rdxf6_0ac2e187-f76e-4497-82dd-c8b902153ace/kube-rbac-proxy/0.log"
Sep 16 19:35:50.128950 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:50.128915    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rdxf6_0ac2e187-f76e-4497-82dd-c8b902153ace/init-textfile/0.log"
Sep 16 19:35:50.413136 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:50.413104    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-5ppvl_73f60141-a105-4a43-a0a3-a4e7b8263963/kube-rbac-proxy-main/0.log"
Sep 16 19:35:50.440685 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:50.440660    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-5ppvl_73f60141-a105-4a43-a0a3-a4e7b8263963/kube-rbac-proxy-self/0.log"
Sep 16 19:35:50.471260 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:50.471222    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-5ppvl_73f60141-a105-4a43-a0a3-a4e7b8263963/openshift-state-metrics/0.log"
Sep 16 19:35:53.220759 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:53.220728    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kdltt/perf-node-gather-daemonset-rt4bs"]
Sep 16 19:35:53.221151 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:53.221038    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f20bd9fa-33f6-456e-aeb1-49f242479ed5" containerName="storage-initializer"
Sep 16 19:35:53.221151 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:53.221049    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="f20bd9fa-33f6-456e-aeb1-49f242479ed5" containerName="storage-initializer"
Sep 16 19:35:53.221151 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:53.221056    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f20bd9fa-33f6-456e-aeb1-49f242479ed5" containerName="kube-rbac-proxy"
Sep 16 19:35:53.221151 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:53.221061    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="f20bd9fa-33f6-456e-aeb1-49f242479ed5" containerName="kube-rbac-proxy"
Sep 16 19:35:53.221151 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:53.221068    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4ddb4a1-8d29-408b-abea-9bfb76725409" containerName="storage-initializer"
Sep 16 19:35:53.221151 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:53.221073    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4ddb4a1-8d29-408b-abea-9bfb76725409" containerName="storage-initializer"
Sep 16 19:35:53.221151 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:53.221081    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f20bd9fa-33f6-456e-aeb1-49f242479ed5" containerName="kserve-container"
Sep 16 19:35:53.221151 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:53.221086    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="f20bd9fa-33f6-456e-aeb1-49f242479ed5" containerName="kserve-container"
Sep 16 19:35:53.221151 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:53.221102    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4ddb4a1-8d29-408b-abea-9bfb76725409" containerName="kserve-container"
Sep 16 19:35:53.221151 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:53.221106    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4ddb4a1-8d29-408b-abea-9bfb76725409" containerName="kserve-container"
Sep 16 19:35:53.221151 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:53.221112    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4ddb4a1-8d29-408b-abea-9bfb76725409" containerName="kube-rbac-proxy"
Sep 16 19:35:53.221151 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:53.221117    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4ddb4a1-8d29-408b-abea-9bfb76725409" containerName="kube-rbac-proxy"
Sep 16 19:35:53.221540 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:53.221170    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="f20bd9fa-33f6-456e-aeb1-49f242479ed5" containerName="kube-rbac-proxy"
Sep 16 19:35:53.221540 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:53.221178    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="f20bd9fa-33f6-456e-aeb1-49f242479ed5" containerName="kserve-container"
Sep 16 19:35:53.221540 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:53.221184    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4ddb4a1-8d29-408b-abea-9bfb76725409" containerName="kserve-container"
Sep 16 19:35:53.221540 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:53.221190    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4ddb4a1-8d29-408b-abea-9bfb76725409" containerName="kube-rbac-proxy"
Sep 16 19:35:53.224354 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:53.224336    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kdltt/perf-node-gather-daemonset-rt4bs"
Sep 16 19:35:53.226857 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:53.226834    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kdltt\"/\"openshift-service-ca.crt\""
Sep 16 19:35:53.226998 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:53.226889    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kdltt\"/\"kube-root-ca.crt\""
Sep 16 19:35:53.227706 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:53.227690    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kdltt\"/\"default-dockercfg-tdd8t\""
Sep 16 19:35:53.240186 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:53.240163    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kdltt/perf-node-gather-daemonset-rt4bs"]
Sep 16 19:35:53.316486 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:53.316446    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/36ea9f91-5616-4ca5-907b-f145f699db7a-proc\") pod \"perf-node-gather-daemonset-rt4bs\" (UID: \"36ea9f91-5616-4ca5-907b-f145f699db7a\") " pod="openshift-must-gather-kdltt/perf-node-gather-daemonset-rt4bs"
Sep 16 19:35:53.316664 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:53.316515    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/36ea9f91-5616-4ca5-907b-f145f699db7a-podres\") pod \"perf-node-gather-daemonset-rt4bs\" (UID: \"36ea9f91-5616-4ca5-907b-f145f699db7a\") " pod="openshift-must-gather-kdltt/perf-node-gather-daemonset-rt4bs"
Sep 16 19:35:53.316664 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:53.316554    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/36ea9f91-5616-4ca5-907b-f145f699db7a-sys\") pod \"perf-node-gather-daemonset-rt4bs\" (UID: \"36ea9f91-5616-4ca5-907b-f145f699db7a\") " pod="openshift-must-gather-kdltt/perf-node-gather-daemonset-rt4bs"
Sep 16 19:35:53.316664 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:53.316601    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/36ea9f91-5616-4ca5-907b-f145f699db7a-lib-modules\") pod \"perf-node-gather-daemonset-rt4bs\" (UID: \"36ea9f91-5616-4ca5-907b-f145f699db7a\") " pod="openshift-must-gather-kdltt/perf-node-gather-daemonset-rt4bs"
Sep 16 19:35:53.316664 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:53.316653    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwcnw\" (UniqueName: \"kubernetes.io/projected/36ea9f91-5616-4ca5-907b-f145f699db7a-kube-api-access-lwcnw\") pod \"perf-node-gather-daemonset-rt4bs\" (UID: \"36ea9f91-5616-4ca5-907b-f145f699db7a\") " pod="openshift-must-gather-kdltt/perf-node-gather-daemonset-rt4bs"
Sep 16 19:35:53.417806 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:53.417748    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/36ea9f91-5616-4ca5-907b-f145f699db7a-lib-modules\") pod \"perf-node-gather-daemonset-rt4bs\" (UID: \"36ea9f91-5616-4ca5-907b-f145f699db7a\") " pod="openshift-must-gather-kdltt/perf-node-gather-daemonset-rt4bs"
Sep 16 19:35:53.417806 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:53.417813    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwcnw\" (UniqueName: \"kubernetes.io/projected/36ea9f91-5616-4ca5-907b-f145f699db7a-kube-api-access-lwcnw\") pod \"perf-node-gather-daemonset-rt4bs\" (UID: \"36ea9f91-5616-4ca5-907b-f145f699db7a\") " pod="openshift-must-gather-kdltt/perf-node-gather-daemonset-rt4bs"
Sep 16 19:35:53.418033 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:53.417845    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/36ea9f91-5616-4ca5-907b-f145f699db7a-proc\") pod \"perf-node-gather-daemonset-rt4bs\" (UID: \"36ea9f91-5616-4ca5-907b-f145f699db7a\") " pod="openshift-must-gather-kdltt/perf-node-gather-daemonset-rt4bs"
Sep 16 19:35:53.418033 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:53.417883    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/36ea9f91-5616-4ca5-907b-f145f699db7a-podres\") pod \"perf-node-gather-daemonset-rt4bs\" (UID: \"36ea9f91-5616-4ca5-907b-f145f699db7a\") " pod="openshift-must-gather-kdltt/perf-node-gather-daemonset-rt4bs"
Sep 16 19:35:53.418033 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:53.417900    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/36ea9f91-5616-4ca5-907b-f145f699db7a-sys\") pod \"perf-node-gather-daemonset-rt4bs\" (UID: \"36ea9f91-5616-4ca5-907b-f145f699db7a\") " pod="openshift-must-gather-kdltt/perf-node-gather-daemonset-rt4bs"
Sep 16 19:35:53.418033 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:53.417932    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/36ea9f91-5616-4ca5-907b-f145f699db7a-lib-modules\") pod \"perf-node-gather-daemonset-rt4bs\" (UID: \"36ea9f91-5616-4ca5-907b-f145f699db7a\") " pod="openshift-must-gather-kdltt/perf-node-gather-daemonset-rt4bs"
Sep 16 19:35:53.418033 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:53.417964    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/36ea9f91-5616-4ca5-907b-f145f699db7a-proc\") pod \"perf-node-gather-daemonset-rt4bs\" (UID: \"36ea9f91-5616-4ca5-907b-f145f699db7a\") " pod="openshift-must-gather-kdltt/perf-node-gather-daemonset-rt4bs"
Sep 16 19:35:53.418033 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:53.417987    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/36ea9f91-5616-4ca5-907b-f145f699db7a-sys\") pod \"perf-node-gather-daemonset-rt4bs\" (UID: \"36ea9f91-5616-4ca5-907b-f145f699db7a\") " pod="openshift-must-gather-kdltt/perf-node-gather-daemonset-rt4bs"
Sep 16 19:35:53.418033 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:53.417988    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/36ea9f91-5616-4ca5-907b-f145f699db7a-podres\") pod \"perf-node-gather-daemonset-rt4bs\" (UID: \"36ea9f91-5616-4ca5-907b-f145f699db7a\") " pod="openshift-must-gather-kdltt/perf-node-gather-daemonset-rt4bs"
Sep 16 19:35:53.427148 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:53.427120    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwcnw\" (UniqueName: \"kubernetes.io/projected/36ea9f91-5616-4ca5-907b-f145f699db7a-kube-api-access-lwcnw\") pod \"perf-node-gather-daemonset-rt4bs\" (UID: \"36ea9f91-5616-4ca5-907b-f145f699db7a\") " pod="openshift-must-gather-kdltt/perf-node-gather-daemonset-rt4bs"
Sep 16 19:35:53.534088 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:53.534003    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kdltt/perf-node-gather-daemonset-rt4bs"
Sep 16 19:35:53.555053 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:53.555026    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-656b4bf9b7-f8rl7_c973db92-be99-42ac-8a85-96d211f806a4/console/0.log"
Sep 16 19:35:53.603546 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:53.603516    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-wd4p9_0fb1d7e8-fe62-4665-b5a8-8bcbd0eff71e/download-server/0.log"
Sep 16 19:35:53.808850 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:53.808771    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kdltt/perf-node-gather-daemonset-rt4bs"]
Sep 16 19:35:53.812048 ip-10-0-129-190 kubenswrapper[2586]: W0916 19:35:53.812003    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod36ea9f91_5616_4ca5_907b_f145f699db7a.slice/crio-0918a927fcc001e19f23daff449f69d5f40c712d46e47b4a18546d309fb4e93c WatchSource:0}: Error finding container 0918a927fcc001e19f23daff449f69d5f40c712d46e47b4a18546d309fb4e93c: Status 404 returned error can't find the container with id 0918a927fcc001e19f23daff449f69d5f40c712d46e47b4a18546d309fb4e93c
Sep 16 19:35:53.965335 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:53.965300    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kdltt/perf-node-gather-daemonset-rt4bs" event={"ID":"36ea9f91-5616-4ca5-907b-f145f699db7a","Type":"ContainerStarted","Data":"590e82c916d7d5f10f1d1d604118b7ae10e69ce83784c13631c40baeaf87d31f"}
Sep 16 19:35:53.965493 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:53.965351    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kdltt/perf-node-gather-daemonset-rt4bs" event={"ID":"36ea9f91-5616-4ca5-907b-f145f699db7a","Type":"ContainerStarted","Data":"0918a927fcc001e19f23daff449f69d5f40c712d46e47b4a18546d309fb4e93c"}
Sep 16 19:35:53.965493 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:53.965419    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kdltt/perf-node-gather-daemonset-rt4bs"
Sep 16 19:35:53.986907 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:53.986851    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kdltt/perf-node-gather-daemonset-rt4bs" podStartSLOduration=0.986835432 podStartE2EDuration="986.835432ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:35:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:35:53.98490571 +0000 UTC m=+3329.428767288" watchObservedRunningTime="2026-09-16 19:35:53.986835432 +0000 UTC m=+3329.430697008"
Sep 16 19:35:55.159898 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:55.159852    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6xjlp_141828f8-d629-4cbd-9ed2-875d52d4db56/dns/0.log"
Sep 16 19:35:55.191115 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:55.191088    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6xjlp_141828f8-d629-4cbd-9ed2-875d52d4db56/kube-rbac-proxy/0.log"
Sep 16 19:35:55.344244 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:55.344203    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cc7hj_b58ed079-b715-420e-a11f-2023c52e1321/dns-node-resolver/0.log"
Sep 16 19:35:55.907770 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:55.907734    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-64ccc449b-b924q_4c90494c-a26f-4e94-9ec4-61e7453a659b/registry/0.log"
Sep 16 19:35:55.951024 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:55.950990    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6298x_7fe5b556-37c9-40c7-a09b-5c6116b43e6f/node-ca/0.log"
Sep 16 19:35:56.905500 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:56.905472    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-986b9b8c8-9p8c8_04e761f8-c6e9-4238-bd6b-0ee2db961991/router/0.log"
Sep 16 19:35:57.304621 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:57.304548    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cgs7w_a437fd80-c809-4981-9f3e-9cb3ce96a470/serve-healthcheck-canary/0.log"
Sep 16 19:35:57.797543 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:57.797508    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-kb6s9_f801294c-a332-41af-bede-44bcd7a22e2c/insights-operator/0.log"
Sep 16 19:35:57.797830 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:57.797716    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-kb6s9_f801294c-a332-41af-bede-44bcd7a22e2c/insights-operator/1.log"
Sep 16 19:35:57.830783 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:57.830758    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ccfm2_cfb2ecb2-e9bb-4c47-bd68-3a835f63365e/kube-rbac-proxy/0.log"
Sep 16 19:35:57.860165 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:57.860120    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ccfm2_cfb2ecb2-e9bb-4c47-bd68-3a835f63365e/exporter/0.log"
Sep 16 19:35:57.890610 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:57.890565    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ccfm2_cfb2ecb2-e9bb-4c47-bd68-3a835f63365e/extractor/0.log"
Sep 16 19:35:59.980152 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:35:59.980123    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kdltt/perf-node-gather-daemonset-rt4bs"
Sep 16 19:36:00.642201 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:36:00.642171    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-599b79d756-tcz7f_23b125e0-f1ef-49c6-8a71-beec63cdd445/manager/0.log"
Sep 16 19:36:00.992949 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:36:00.992868    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-psx8f_38f003c8-ea78-4260-8fc0-4dc2d1415936/manager/0.log"
Sep 16 19:36:01.249985 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:36:01.249884    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-xssm2_12682bbb-5ad5-44a3-b405-f8840e7b2524/seaweedfs/0.log"
Sep 16 19:36:01.280864 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:36:01.280840    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-custom-8b44d767d-4szz7_4ff37a2e-7be7-4578-9b29-949089c4f8a2/seaweedfs-tls-custom/0.log"
Sep 16 19:36:01.314587 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:36:01.314525    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-serving-67b7477c6d-lbdvm_40aa28e7-7214-41ef-8483-14a7429bc0b9/seaweedfs-tls-serving/0.log"
Sep 16 19:36:06.848653 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:36:06.848604    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-ps6sd_3ab787a9-4d8d-42f1-b7cb-432be6e37285/kube-storage-version-migrator-operator/1.log"
Sep 16 19:36:06.850117 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:36:06.850097    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-ps6sd_3ab787a9-4d8d-42f1-b7cb-432be6e37285/kube-storage-version-migrator-operator/0.log"
Sep 16 19:36:07.890980 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:36:07.890948    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9s7jf_9716763a-ff84-4735-b478-deb614be27c4/kube-multus-additional-cni-plugins/0.log"
Sep 16 19:36:07.920495 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:36:07.920467    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9s7jf_9716763a-ff84-4735-b478-deb614be27c4/egress-router-binary-copy/0.log"
Sep 16 19:36:07.949874 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:36:07.949850    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9s7jf_9716763a-ff84-4735-b478-deb614be27c4/cni-plugins/0.log"
Sep 16 19:36:07.989371 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:36:07.989332    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9s7jf_9716763a-ff84-4735-b478-deb614be27c4/bond-cni-plugin/0.log"
Sep 16 19:36:08.035280 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:36:08.035223    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9s7jf_9716763a-ff84-4735-b478-deb614be27c4/routeoverride-cni/0.log"
Sep 16 19:36:08.083690 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:36:08.083656    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9s7jf_9716763a-ff84-4735-b478-deb614be27c4/whereabouts-cni-bincopy/0.log"
Sep 16 19:36:08.146621 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:36:08.146543    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9s7jf_9716763a-ff84-4735-b478-deb614be27c4/whereabouts-cni/0.log"
Sep 16 19:36:08.642887 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:36:08.642859    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-wbl55_66e76bc5-f7b3-4952-b67c-0fb014f21ecb/kube-multus/0.log"
Sep 16 19:36:08.831121 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:36:08.831088    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9vpbt_1e8169e6-6afe-417b-90f0-5a55bf294956/network-metrics-daemon/0.log"
Sep 16 19:36:08.856111 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:36:08.856088    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9vpbt_1e8169e6-6afe-417b-90f0-5a55bf294956/kube-rbac-proxy/0.log"
Sep 16 19:36:09.958722 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:36:09.958693    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wxhg2_f76a1a1e-3387-4aae-8964-63c913eb270a/ovn-controller/0.log"
Sep 16 19:36:09.999471 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:36:09.999445    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wxhg2_f76a1a1e-3387-4aae-8964-63c913eb270a/ovn-acl-logging/0.log"
Sep 16 19:36:10.045099 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:36:10.045070    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wxhg2_f76a1a1e-3387-4aae-8964-63c913eb270a/kube-rbac-proxy-node/0.log"
Sep 16 19:36:10.098073 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:36:10.098047    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wxhg2_f76a1a1e-3387-4aae-8964-63c913eb270a/kube-rbac-proxy-ovn-metrics/0.log"
Sep 16 19:36:10.135142 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:36:10.135099    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wxhg2_f76a1a1e-3387-4aae-8964-63c913eb270a/northd/0.log"
Sep 16 19:36:10.179556 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:36:10.179525    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wxhg2_f76a1a1e-3387-4aae-8964-63c913eb270a/nbdb/0.log"
Sep 16 19:36:10.220160 ip-10-0-129-190 kubenswrapper[2586]: I0916 19:36:10.220080    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wxhg2_f76a1a1e-3387-4aae-8964-63c913eb270a/sbdb/0.log"