Jul 28 12:53:23.086581 ip-10-0-129-177 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 12:53:23.520775 ip-10-0-129-177 kubenswrapper[2554]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 12:53:23.520775 ip-10-0-129-177 kubenswrapper[2554]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 12:53:23.520775 ip-10-0-129-177 kubenswrapper[2554]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 12:53:23.520775 ip-10-0-129-177 kubenswrapper[2554]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 12:53:23.520775 ip-10-0-129-177 kubenswrapper[2554]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 12:53:23.522846 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.522739    2554 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 12:53:23.531975 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.531955    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:53:23.531975 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.531971    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:53:23.531975 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.531975    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:53:23.531975 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.531978    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:53:23.531975 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.531981    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:53:23.532156 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.531984    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:53:23.532156 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.531987    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:53:23.532156 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.531990    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:53:23.532156 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.531994    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:53:23.532156 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.531999    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:53:23.532156 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532003    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:53:23.532156 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532005    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:53:23.532156 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532008    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:53:23.532156 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532011    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:53:23.532156 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532014    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:53:23.532156 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532017    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:53:23.532156 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532020    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:53:23.532156 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532023    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:53:23.532156 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532026    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:53:23.532156 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532029    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:53:23.532156 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532031    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:53:23.532156 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532034    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:53:23.532156 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532036    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:53:23.532156 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532039    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:53:23.532628 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532042    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:53:23.532628 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532044    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:53:23.532628 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532047    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:53:23.532628 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532049    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:53:23.532628 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532052    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:53:23.532628 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532054    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:53:23.532628 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532057    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:53:23.532628 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532059    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:53:23.532628 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532062    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:53:23.532628 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532064    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:53:23.532628 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532066    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:53:23.532628 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532069    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:53:23.532628 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532071    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:53:23.532628 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532074    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:53:23.532628 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532077    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:53:23.532628 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532080    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:53:23.532628 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532083    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:53:23.532628 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532085    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:53:23.532628 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532088    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:53:23.532628 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532092    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:53:23.533134 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532094    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:53:23.533134 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532097    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:53:23.533134 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532099    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:53:23.533134 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532102    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:53:23.533134 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532104    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:53:23.533134 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532107    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:53:23.533134 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532110    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:53:23.533134 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532113    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:53:23.533134 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532115    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:53:23.533134 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532118    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:53:23.533134 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532120    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:53:23.533134 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532122    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:53:23.533134 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532125    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:53:23.533134 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532127    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:53:23.533134 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532130    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:53:23.533134 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532132    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:53:23.533134 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532135    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:53:23.533134 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532138    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:53:23.533134 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532140    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:53:23.533616 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532143    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:53:23.533616 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532146    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:53:23.533616 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532150    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:53:23.533616 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532153    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:53:23.533616 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532156    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:53:23.533616 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532160    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:53:23.533616 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532163    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:53:23.533616 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532166    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:53:23.533616 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532169    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:53:23.533616 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532172    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:53:23.533616 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532174    2554 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:53:23.533616 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532178    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:53:23.533616 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532181    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:53:23.533616 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532184    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:53:23.533616 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532186    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:53:23.533616 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532189    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:53:23.533616 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532192    2554 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:53:23.533616 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532194    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:53:23.533616 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532197    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:53:23.533616 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532200    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:53:23.534086 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532203    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:53:23.534086 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532205    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:53:23.534086 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532207    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:53:23.534086 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532210    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:53:23.534086 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532212    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:53:23.534086 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532214    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:53:23.534086 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532217    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:53:23.534086 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532219    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:53:23.534086 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532222    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:53:23.534086 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532224    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:53:23.534086 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532227    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:53:23.534086 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532229    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:53:23.534086 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532233    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:53:23.534086 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532236    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:53:23.534086 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532240    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:53:23.534086 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.532242    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:53:23.534086 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534067    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:53:23.534086 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534075    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:53:23.534086 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534078    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:53:23.534556 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534081    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:53:23.534556 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534083    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:53:23.534556 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534086    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:53:23.534556 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534089    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:53:23.534556 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534092    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:53:23.534556 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534095    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:53:23.534556 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534097    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:53:23.534556 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534100    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:53:23.534556 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534104    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:53:23.534556 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534108    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:53:23.534556 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534111    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:53:23.534556 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534114    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:53:23.534556 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534117    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:53:23.534556 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534119    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:53:23.534556 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534123    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:53:23.534556 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534125    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:53:23.534556 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534128    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:53:23.534556 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534130    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:53:23.534556 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534133    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:53:23.535030 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534136    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:53:23.535030 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534138    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:53:23.535030 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534141    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:53:23.535030 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534143    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:53:23.535030 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534145    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:53:23.535030 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534148    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:53:23.535030 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534150    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:53:23.535030 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534153    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:53:23.535030 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534155    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:53:23.535030 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534158    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:53:23.535030 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534160    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:53:23.535030 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534163    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:53:23.535030 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534165    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:53:23.535030 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534168    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:53:23.535030 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534171    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:53:23.535030 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534174    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:53:23.535030 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534176    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:53:23.535030 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534179    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:53:23.535030 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534181    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:53:23.535030 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534184    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:53:23.535518 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534186    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:53:23.535518 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534189    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:53:23.535518 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534191    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:53:23.535518 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534194    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:53:23.535518 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534196    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:53:23.535518 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534199    2554 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:53:23.535518 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534201    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:53:23.535518 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534204    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:53:23.535518 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534206    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:53:23.535518 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534209    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:53:23.535518 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534211    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:53:23.535518 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534214    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:53:23.535518 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534216    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:53:23.535518 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534218    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:53:23.535518 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534222    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:53:23.535518 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534224    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:53:23.535518 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534227    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:53:23.535518 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534229    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:53:23.535518 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534232    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:53:23.535518 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534234    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:53:23.535995 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534236    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:53:23.535995 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534239    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:53:23.535995 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534241    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:53:23.535995 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534245    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:53:23.535995 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534247    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:53:23.535995 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534250    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:53:23.535995 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534254    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:53:23.535995 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534257    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:53:23.535995 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534260    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:53:23.535995 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534262    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:53:23.535995 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534265    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:53:23.535995 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534267    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:53:23.535995 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534285    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:53:23.535995 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534288    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:53:23.535995 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534290    2554 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:53:23.535995 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534293    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:53:23.535995 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534296    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:53:23.535995 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534298    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:53:23.535995 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534301    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:53:23.536472 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534303    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:53:23.536472 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534307    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:53:23.536472 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534310    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:53:23.536472 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534313    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:53:23.536472 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534315    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:53:23.536472 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534317    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:53:23.536472 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534320    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:53:23.536472 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534322    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:53:23.536472 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534325    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:53:23.536472 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534327    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:53:23.536472 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534330    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:53:23.536472 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534332    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:53:23.536472 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534335    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:53:23.536472 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534338    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:53:23.536472 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534340    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:53:23.536472 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534343    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:53:23.536472 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534345    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:53:23.536472 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534349    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:53:23.536472 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534712    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:53:23.536472 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534718    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:53:23.536970 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534721    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:53:23.536970 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534724    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:53:23.536970 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534727    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:53:23.536970 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534730    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:53:23.536970 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534733    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:53:23.536970 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534736    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:53:23.536970 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534738    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:53:23.536970 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534741    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:53:23.536970 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534743    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:53:23.536970 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534746    2554 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:53:23.536970 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534748    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:53:23.536970 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534751    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:53:23.536970 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534754    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:53:23.536970 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534757    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:53:23.536970 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534760    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:53:23.536970 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534763    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:53:23.536970 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534765    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:53:23.536970 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534767    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:53:23.536970 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534770    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:53:23.536970 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534772    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:53:23.537474 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534775    2554 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:53:23.537474 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534777    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:53:23.537474 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534780    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:53:23.537474 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534782    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:53:23.537474 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534784    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:53:23.537474 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534787    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:53:23.537474 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534789    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:53:23.537474 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534792    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:53:23.537474 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534794    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:53:23.537474 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534798    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:53:23.537474 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534800    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:53:23.537474 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534806    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:53:23.537474 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534808    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:53:23.537474 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534810    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:53:23.537474 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534813    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:53:23.537474 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534815    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:53:23.537474 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534818    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:53:23.537474 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534820    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:53:23.537474 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534823    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:53:23.537474 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534825    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:53:23.537948 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534827    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:53:23.537948 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534830    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:53:23.537948 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534832    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:53:23.537948 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534834    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:53:23.537948 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534837    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:53:23.537948 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534839    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:53:23.537948 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534841    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:53:23.537948 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534844    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:53:23.537948 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534846    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:53:23.537948 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534849    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:53:23.537948 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534852    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:53:23.537948 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534854    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:53:23.537948 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534857    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:53:23.537948 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534861    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:53:23.537948 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534865    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:53:23.537948 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534867    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:53:23.537948 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534870    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:53:23.537948 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534873    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:53:23.537948 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534876    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:53:23.537948 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534878    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:53:23.538480 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534881    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:53:23.538480 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534884    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:53:23.538480 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534887    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:53:23.538480 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534891    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:53:23.538480 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534894    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:53:23.538480 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534896    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:53:23.538480 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534899    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:53:23.538480 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534902    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:53:23.538480 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534904    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:53:23.538480 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534907    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:53:23.538480 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534909    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:53:23.538480 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534912    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:53:23.538480 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534914    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:53:23.538480 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534917    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:53:23.538480 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534919    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:53:23.538480 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534922    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:53:23.538480 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534925    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:53:23.538480 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534928    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:53:23.538480 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534932    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:53:23.538926 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534935    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:53:23.538926 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534938    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:53:23.538926 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534941    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:53:23.538926 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534944    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:53:23.538926 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534948    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:53:23.538926 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534950    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:53:23.538926 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534953    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:53:23.538926 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534956    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:53:23.538926 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534959    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:53:23.538926 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534961    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:53:23.538926 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534964    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:53:23.538926 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534966    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:53:23.538926 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534969    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:53:23.538926 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534971    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:53:23.538926 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534974    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:53:23.538926 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534976    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:53:23.538926 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534980    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:53:23.538926 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.534982    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:53:23.538926 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535049    2554 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 12:53:23.538926 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535055    2554 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 12:53:23.538926 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535061    2554 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 12:53:23.539434 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535067    2554 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 12:53:23.539434 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535073    2554 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 12:53:23.539434 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535078    2554 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 12:53:23.539434 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535084    2554 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 12:53:23.539434 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535091    2554 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 12:53:23.539434 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535095    2554 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 12:53:23.539434 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535098    2554 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 12:53:23.539434 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535102    2554 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 12:53:23.539434 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535105    2554 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 12:53:23.539434 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535108    2554 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 12:53:23.539434 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535110    2554 flags.go:64] FLAG: --cgroup-root=""
Jul 28 12:53:23.539434 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535113    2554 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 12:53:23.539434 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535116    2554 flags.go:64] FLAG: --client-ca-file=""
Jul 28 12:53:23.539434 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535119    2554 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 12:53:23.539434 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535123    2554 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 12:53:23.539434 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535128    2554 flags.go:64] FLAG: --cluster-domain=""
Jul 28 12:53:23.539434 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535131    2554 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 12:53:23.539434 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535134    2554 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 12:53:23.539434 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535137    2554 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 12:53:23.539434 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535140    2554 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 12:53:23.539434 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535143    2554 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 12:53:23.539434 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535146    2554 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 12:53:23.539434 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535149    2554 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 12:53:23.540022 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535152    2554 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 12:53:23.540022 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535155    2554 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 12:53:23.540022 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535158    2554 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 12:53:23.540022 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535161    2554 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 12:53:23.540022 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535164    2554 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 12:53:23.540022 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535167    2554 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 12:53:23.540022 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535171    2554 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 12:53:23.540022 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535174    2554 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 12:53:23.540022 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535176    2554 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 12:53:23.540022 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535179    2554 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 12:53:23.540022 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535181    2554 flags.go:64] FLAG: --enable-server="true"
Jul 28 12:53:23.540022 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535184    2554 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 12:53:23.540022 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535188    2554 flags.go:64] FLAG: --event-burst="100"
Jul 28 12:53:23.540022 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535191    2554 flags.go:64] FLAG: --event-qps="50"
Jul 28 12:53:23.540022 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535193    2554 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 12:53:23.540022 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535196    2554 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 12:53:23.540022 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535199    2554 flags.go:64] FLAG: --eviction-hard=""
Jul 28 12:53:23.540022 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535202    2554 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 12:53:23.540022 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535205    2554 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 12:53:23.540022 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535207    2554 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 12:53:23.540022 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535210    2554 flags.go:64] FLAG: --eviction-soft=""
Jul 28 12:53:23.540022 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535213    2554 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 12:53:23.540022 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535215    2554 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 12:53:23.540022 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535218    2554 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 12:53:23.540022 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535222    2554 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 12:53:23.540742 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535225    2554 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 12:53:23.540742 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535227    2554 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 12:53:23.540742 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535230    2554 flags.go:64] FLAG: --feature-gates=""
Jul 28 12:53:23.540742 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535233    2554 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 12:53:23.540742 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535236    2554 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 12:53:23.540742 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535239    2554 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 12:53:23.540742 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535242    2554 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 12:53:23.540742 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535244    2554 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 12:53:23.540742 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535247    2554 flags.go:64] FLAG: --help="false"
Jul 28 12:53:23.540742 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535250    2554 flags.go:64] FLAG: --hostname-override="ip-10-0-129-177.ec2.internal"
Jul 28 12:53:23.540742 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535253    2554 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 12:53:23.540742 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535256    2554 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 12:53:23.540742 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535259    2554 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 12:53:23.540742 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535262    2554 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 12:53:23.540742 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535265    2554 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 12:53:23.540742 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535268    2554 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 12:53:23.540742 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535285    2554 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 12:53:23.540742 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535287    2554 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 12:53:23.540742 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535290    2554 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 12:53:23.540742 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535293    2554 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 12:53:23.540742 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535296    2554 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 12:53:23.540742 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535299    2554 flags.go:64] FLAG: --kube-reserved=""
Jul 28 12:53:23.540742 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535302    2554 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 12:53:23.540742 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535305    2554 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 12:53:23.541359 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535307    2554 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 12:53:23.541359 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535310    2554 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 12:53:23.541359 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535313    2554 flags.go:64] FLAG: --lock-file=""
Jul 28 12:53:23.541359 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535315    2554 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 12:53:23.541359 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535318    2554 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 12:53:23.541359 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535321    2554 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 12:53:23.541359 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535326    2554 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 12:53:23.541359 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535330    2554 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 12:53:23.541359 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535333    2554 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 12:53:23.541359 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535336    2554 flags.go:64] FLAG: --logging-format="text"
Jul 28 12:53:23.541359 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535339    2554 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 12:53:23.541359 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535342    2554 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 12:53:23.541359 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535345    2554 flags.go:64] FLAG: --manifest-url=""
Jul 28 12:53:23.541359 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535348    2554 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 12:53:23.541359 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535352    2554 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 12:53:23.541359 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535355    2554 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 12:53:23.541359 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535359    2554 flags.go:64] FLAG: --max-pods="110"
Jul 28 12:53:23.541359 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535361    2554 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 12:53:23.541359 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535364    2554 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 12:53:23.541359 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535367    2554 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 12:53:23.541359 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535370    2554 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 12:53:23.541359 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535373    2554 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 12:53:23.541359 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535375    2554 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 12:53:23.541359 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535378    2554 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 12:53:23.541359 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535385    2554 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 12:53:23.541965 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535388    2554 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 12:53:23.541965 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535391    2554 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 12:53:23.541965 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535393    2554 flags.go:64] FLAG: --pod-cidr=""
Jul 28 12:53:23.541965 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535396    2554 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 12:53:23.541965 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535401    2554 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 12:53:23.541965 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535403    2554 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 12:53:23.541965 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535406    2554 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 12:53:23.541965 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535409    2554 flags.go:64] FLAG: --port="10250"
Jul 28 12:53:23.541965 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535411    2554 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 12:53:23.541965 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535414    2554 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08177515421f6ab40"
Jul 28 12:53:23.541965 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535417    2554 flags.go:64] FLAG: --qos-reserved=""
Jul 28 12:53:23.541965 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535420    2554 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 12:53:23.541965 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535422    2554 flags.go:64] FLAG: --register-node="true"
Jul 28 12:53:23.541965 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535425    2554 flags.go:64] FLAG: --register-with-taints=""
Jul 28 12:53:23.541965 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535429    2554 flags.go:64] FLAG: --registry-burst="10"
Jul 28 12:53:23.541965 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535433    2554 flags.go:64] FLAG: --registry-qps="5"
Jul 28 12:53:23.541965 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535435    2554 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 12:53:23.541965 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535438    2554 flags.go:64] FLAG: --reserved-memory=""
Jul 28 12:53:23.541965 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535441    2554 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 12:53:23.541965 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535444    2554 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 12:53:23.541965 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535447    2554 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 12:53:23.541965 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535449    2554 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 12:53:23.541965 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535457    2554 flags.go:64] FLAG: --runonce="false"
Jul 28 12:53:23.541965 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535459    2554 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 12:53:23.541965 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535462    2554 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 12:53:23.542675 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535465    2554 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 12:53:23.542675 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535468    2554 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 12:53:23.542675 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535471    2554 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 12:53:23.542675 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535474    2554 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 12:53:23.542675 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535477    2554 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 12:53:23.542675 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535480    2554 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 12:53:23.542675 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535483    2554 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 12:53:23.542675 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535485    2554 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 12:53:23.542675 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535488    2554 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 12:53:23.542675 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535490    2554 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 12:53:23.542675 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535493    2554 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 12:53:23.542675 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535496    2554 flags.go:64] FLAG: --system-cgroups=""
Jul 28 12:53:23.542675 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535499    2554 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 12:53:23.542675 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535504    2554 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 12:53:23.542675 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535506    2554 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 12:53:23.542675 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535509    2554 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 12:53:23.542675 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535512    2554 flags.go:64] FLAG: --tls-min-version=""
Jul 28 12:53:23.542675 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535515    2554 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 12:53:23.542675 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535518    2554 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 12:53:23.542675 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535520    2554 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 12:53:23.542675 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535523    2554 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 12:53:23.542675 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535527    2554 flags.go:64] FLAG: --v="2"
Jul 28 12:53:23.542675 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535531    2554 flags.go:64] FLAG: --version="false"
Jul 28 12:53:23.542675 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535536    2554 flags.go:64] FLAG: --vmodule=""
Jul 28 12:53:23.542675 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535543    2554 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 12:53:23.543746 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.535546    2554 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 12:53:23.543746 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535650    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:53:23.543746 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535654    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:53:23.543746 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535657    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:53:23.543746 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535659    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:53:23.543746 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535662    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:53:23.543746 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535664    2554 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:53:23.543746 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535668    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:53:23.543746 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535671    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:53:23.543746 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535673    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:53:23.543746 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535676    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:53:23.543746 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535678    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:53:23.543746 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535681    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:53:23.543746 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535683    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:53:23.543746 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535686    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:53:23.543746 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535688    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:53:23.543746 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535691    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:53:23.543746 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535693    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:53:23.543746 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535696    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:53:23.543746 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535698    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:53:23.544607 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535701    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:53:23.544607 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535703    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:53:23.544607 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535705    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:53:23.544607 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535708    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:53:23.544607 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535710    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:53:23.544607 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535713    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:53:23.544607 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535715    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:53:23.544607 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535728    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:53:23.544607 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535733    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:53:23.544607 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535735    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:53:23.544607 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535738    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:53:23.544607 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535741    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:53:23.544607 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535744    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:53:23.544607 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535746    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:53:23.544607 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535748    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:53:23.544607 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535751    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:53:23.544607 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535755    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:53:23.544607 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535758    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:53:23.544607 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535761    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:53:23.545342 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535765    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:53:23.545342 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535767    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:53:23.545342 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535770    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:53:23.545342 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535773    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:53:23.545342 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535776    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:53:23.545342 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535779    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:53:23.545342 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535782    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:53:23.545342 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535784    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:53:23.545342 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535787    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:53:23.545342 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535789    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:53:23.545342 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535792    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:53:23.545342 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535795    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:53:23.545342 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535797    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:53:23.545342 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535800    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:53:23.545342 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535802    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:53:23.545342 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535806    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:53:23.545342 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535809    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:53:23.545342 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535812    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:53:23.545342 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535814    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:53:23.545342 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535817    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:53:23.545827 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535819    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:53:23.545827 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535823    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:53:23.545827 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535826    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:53:23.545827 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535828    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:53:23.545827 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535830    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:53:23.545827 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535833    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:53:23.545827 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535836    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:53:23.545827 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535838    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:53:23.545827 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535841    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:53:23.545827 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535843    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:53:23.545827 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535846    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:53:23.545827 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535849    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:53:23.545827 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535854    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:53:23.545827 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535857    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:53:23.545827 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535860    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:53:23.545827 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535863    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:53:23.545827 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535865    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:53:23.545827 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535868    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:53:23.545827 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535870    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:53:23.545827 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535873    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:53:23.546708 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535876    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:53:23.546708 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535878    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:53:23.546708 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535881    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:53:23.546708 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535884    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:53:23.546708 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535886    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:53:23.546708 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535889    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:53:23.546708 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535892    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:53:23.546708 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535895    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:53:23.546708 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535897    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:53:23.546708 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535900    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:53:23.546708 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535903    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:53:23.546708 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535905    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:53:23.546708 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535908    2554 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:53:23.546708 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535913    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:53:23.546708 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535915    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:53:23.546708 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535918    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:53:23.546708 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535921    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:53:23.546708 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535924    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:53:23.546708 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535927    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:53:23.546708 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535930    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:53:23.547562 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.535932    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:53:23.547562 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.536667    2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 12:53:23.547562 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.545086    2554 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 12:53:23.547562 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.545107    2554 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 12:53:23.547562 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.545990    2554 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 12:53:23.547562 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.546009    2554 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 12:53:23.547562 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546099    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:53:23.547562 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546106    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:53:23.547562 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546111    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:53:23.547562 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546116    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:53:23.547562 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546121    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:53:23.547562 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546126    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:53:23.547562 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546130    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:53:23.547562 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546134    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:53:23.547945 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546139    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:53:23.547945 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546143    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:53:23.547945 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546147    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:53:23.547945 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546151    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:53:23.547945 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546155    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:53:23.547945 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546160    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:53:23.547945 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546164    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:53:23.547945 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546169    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:53:23.547945 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546173    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:53:23.547945 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546177    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:53:23.547945 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546181    2554 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:53:23.547945 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546186    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:53:23.547945 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546190    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:53:23.547945 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546194    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:53:23.547945 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546199    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:53:23.547945 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546204    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:53:23.547945 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546209    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:53:23.547945 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546213    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:53:23.547945 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546217    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:53:23.547945 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546221    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:53:23.548654 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546226    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:53:23.548654 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546231    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:53:23.548654 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546236    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:53:23.548654 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546243    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:53:23.548654 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546248    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:53:23.548654 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546253    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:53:23.548654 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546257    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:53:23.548654 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546261    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:53:23.548654 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546267    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:53:23.548654 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546290    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:53:23.548654 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546294    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:53:23.548654 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546298    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:53:23.548654 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546302    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:53:23.548654 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546306    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:53:23.548654 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546310    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:53:23.548654 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546313    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:53:23.548654 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546317    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:53:23.548654 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546321    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:53:23.548654 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546325    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:53:23.549457 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546328    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:53:23.549457 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546332    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:53:23.549457 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546336    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:53:23.549457 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546340    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:53:23.549457 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546344    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:53:23.549457 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546348    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:53:23.549457 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546351    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:53:23.549457 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546355    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:53:23.549457 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546359    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:53:23.549457 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546363    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:53:23.549457 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546368    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:53:23.549457 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546375    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:53:23.549457 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546380    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:53:23.549457 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546384    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:53:23.549457 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546389    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:53:23.549457 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546398    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:53:23.549457 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546403    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:53:23.549457 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546407    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:53:23.549457 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546411    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:53:23.549457 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546415    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:53:23.550025 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546420    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:53:23.550025 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546423    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:53:23.550025 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546428    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:53:23.550025 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546432    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:53:23.550025 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546436    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:53:23.550025 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546440    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:53:23.550025 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546445    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:53:23.550025 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546449    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:53:23.550025 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546453    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:53:23.550025 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546457    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:53:23.550025 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546461    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:53:23.550025 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546466    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:53:23.550025 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546470    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:53:23.550025 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546474    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:53:23.550025 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546478    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:53:23.550025 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546482    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:53:23.550025 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546487    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:53:23.550025 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546492    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:53:23.550025 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546496    2554 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:53:23.550025 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546500    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:53:23.550636 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546504    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:53:23.550636 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546508    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:53:23.550636 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546513    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:53:23.550636 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546517    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:53:23.550636 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546524    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:53:23.550636 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546529    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:53:23.550636 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546535    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:53:23.550636 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546539    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:53:23.550636 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546545    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:53:23.550636 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546550    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:53:23.550636 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546554    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:53:23.550636 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546559    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:53:23.550636 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.546566    2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 12:53:23.550636 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546770    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 12:53:23.550985 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546780    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 12:53:23.550985 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546787    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 12:53:23.550985 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546792    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 12:53:23.550985 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546798    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 12:53:23.550985 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546802    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 12:53:23.550985 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546807    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 12:53:23.550985 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546811    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 12:53:23.550985 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546816    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 12:53:23.550985 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546820    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 12:53:23.550985 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546824    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 12:53:23.550985 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546829    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 12:53:23.550985 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546833    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 12:53:23.550985 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546839    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 12:53:23.550985 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546844    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 12:53:23.550985 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546849    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 12:53:23.550985 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546854    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 12:53:23.550985 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546858    2554 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 12:53:23.550985 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546863    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 12:53:23.550985 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546867    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 12:53:23.551483 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546871    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 12:53:23.551483 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546875    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 12:53:23.551483 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546879    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 12:53:23.551483 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546884    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 12:53:23.551483 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546888    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 12:53:23.551483 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546892    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 12:53:23.551483 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546896    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 12:53:23.551483 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546900    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 12:53:23.551483 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546907    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 12:53:23.551483 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546932    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 12:53:23.551483 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546938    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 12:53:23.551483 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546943    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 12:53:23.551483 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546948    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 12:53:23.551483 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546952    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 12:53:23.551483 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546956    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 12:53:23.551483 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546960    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 12:53:23.551483 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546964    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 12:53:23.551483 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546968    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 12:53:23.551483 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546972    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 12:53:23.551943 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546976    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 12:53:23.551943 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546980    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 12:53:23.551943 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546984    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 12:53:23.551943 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546988    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 12:53:23.551943 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546992    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 12:53:23.551943 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.546996    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 12:53:23.551943 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.547000    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 12:53:23.551943 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.547005    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 12:53:23.551943 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.547009    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 12:53:23.551943 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.547013    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 12:53:23.551943 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.547017    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 12:53:23.551943 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.547021    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 12:53:23.551943 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.547025    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 12:53:23.551943 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.547029    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 12:53:23.551943 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.547033    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 12:53:23.551943 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.547037    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 12:53:23.551943 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.547041    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 12:53:23.551943 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.547045    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 12:53:23.551943 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.547050    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 12:53:23.551943 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.547054    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 12:53:23.552496 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.547058    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 12:53:23.552496 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.547062    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 12:53:23.552496 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.547067    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 12:53:23.552496 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.547071    2554 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 12:53:23.552496 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.547075    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 12:53:23.552496 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.547079    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 12:53:23.552496 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.547084    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 12:53:23.552496 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.547088    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 12:53:23.552496 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.547091    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 12:53:23.552496 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.547096    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 12:53:23.552496 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.547100    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 12:53:23.552496 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.547103    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 12:53:23.552496 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.547107    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 12:53:23.552496 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.547111    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 12:53:23.552496 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.547116    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 12:53:23.552496 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.547120    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 12:53:23.552496 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.547124    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 12:53:23.552496 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.547128    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 12:53:23.552496 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.547132    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 12:53:23.552496 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.547136    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 12:53:23.552974 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.547140    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 12:53:23.552974 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.547144    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 12:53:23.552974 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.547149    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 12:53:23.552974 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.547152    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 12:53:23.552974 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.547156    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 12:53:23.552974 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.547161    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 12:53:23.552974 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.547165    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 12:53:23.552974 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.547169    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 12:53:23.552974 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.547173    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 12:53:23.552974 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.547177    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 12:53:23.552974 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.547181    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 12:53:23.552974 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.547185    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 12:53:23.552974 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.547189    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 12:53:23.552974 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.547193    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 12:53:23.552974 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.547197    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 12:53:23.552974 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.547202    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 12:53:23.552974 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.547206    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 12:53:23.552974 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.547210    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 12:53:23.552974 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.547214    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 12:53:23.552974 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:23.547218    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 12:53:23.553534 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.547226    2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 12:53:23.553534 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.547423    2554 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 12:53:23.553534 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.550429    2554 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 12:53:23.553534 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.552228    2554 server.go:1019] "Starting client certificate rotation"
Jul 28 12:53:23.553534 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.552300    2554 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 12:53:23.553534 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.552337    2554 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 12:53:23.555113 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.555097    2554 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 12:53:23.559965 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.559943    2554 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 12:53:23.570776 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.570760    2554 log.go:25] "Validated CRI v1 runtime API"
Jul 28 12:53:23.573378 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.573365    2554 log.go:25] "Validated CRI v1 image API"
Jul 28 12:53:23.574153 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.574132    2554 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 12:53:23.576751 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.576727    2554 fs.go:135] Filesystem UUIDs: map[0930f720-e3a8-447f-96f4-6e102c9a67cd:/dev/nvme0n1p4 1a9c7bb1-3b5c-42f6-9346-324209c11551:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Jul 28 12:53:23.576802 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.576752    2554 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 12:53:23.582044 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.581923    2554 manager.go:216] Machine: {Timestamp:2026-07-28 12:53:23.580554813 +0000 UTC m=+0.366429915 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099800 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ba919a39af405d1f4ccc104d904b4 SystemUUID:ec2ba919-a39a-f405-d1f4-ccc104d904b4 BootID:77250672-0853-40e4-8b76-419ef851e306 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:35:0e:09:31:cb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:35:0e:09:31:cb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8a:5b:8c:8d:a7:b6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 12:53:23.582110 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.582046    2554 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 12:53:23.582165 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.582149    2554 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 12:53:23.582196 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.582163    2554 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 12:53:23.585851 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.585824    2554 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 12:53:23.585975 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.585854    2554 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-177.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 12:53:23.586663 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.586652    2554 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 12:53:23.586699 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.586665    2554 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 12:53:23.586699 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.586673    2554 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 12:53:23.586699 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.586686    2554 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 12:53:23.587933 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.587923    2554 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 12:53:23.590760 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.590749    2554 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 12:53:23.590867 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.590857    2554 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 12:53:23.590901 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.590885    2554 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 12:53:23.590901 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.590895    2554 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 12:53:23.590956 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.590910    2554 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 12:53:23.590956 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.590919    2554 kubelet.go:398] "Adding apiserver pod source"
Jul 28 12:53:23.590956 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.590929    2554 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 12:53:23.591324 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.591304    2554 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 12:53:23.591324 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.591326    2554 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 12:53:23.593861 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.593836    2554 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 12:53:23.596200 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.596185    2554 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 12:53:23.596252 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.596217    2554 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 12:53:23.598332 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.598320    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 12:53:23.598379 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.598337    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 12:53:23.598379 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.598343    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 12:53:23.598379 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.598349    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 12:53:23.598379 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.598357    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 12:53:23.598379 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.598362    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 12:53:23.598379 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.598373    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 12:53:23.598379 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.598380    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 12:53:23.598577 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.598390    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 12:53:23.598577 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.598395    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 12:53:23.598577 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.598403    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 12:53:23.598577 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.598408    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 12:53:23.600080 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.600067    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 12:53:23.600080 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.600077    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 12:53:23.603516 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:23.602855    2554 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-177.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 12:53:23.603596 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:23.602884    2554 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 12:53:23.604083 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.604069    2554 server.go:1268] "Started kubelet"
Jul 28 12:53:23.604220 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.604167    2554 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 12:53:23.604220 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.604204    2554 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 12:53:23.604393 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.604236    2554 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 12:53:23.604956 ip-10-0-129-177 systemd[1]: Started Kubernetes Kubelet.
Jul 28 12:53:23.606190 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.606177    2554 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 12:53:23.606296 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.606200    2554 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 12:53:23.610428 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:23.610403    2554 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 12:53:23.611475 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.611460    2554 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-177.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 12:53:23.612047 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.612028    2554 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qv9lt"
Jul 28 12:53:23.612658 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:23.611508    2554 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-177.ec2.internal.18c6753c2c7fd0a2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-177.ec2.internal,UID:ip-10-0-129-177.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-177.ec2.internal,},FirstTimestamp:2026-07-28 12:53:23.60404189 +0000 UTC m=+0.389916992,LastTimestamp:2026-07-28 12:53:23.60404189 +0000 UTC m=+0.389916992,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-177.ec2.internal,}"
Jul 28 12:53:23.616318 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.616305    2554 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 12:53:23.616405 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.616383    2554 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 12:53:23.616680 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:23.616602    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-177.ec2.internal\" not found"
Jul 28 12:53:23.617340 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.617129    2554 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 12:53:23.617425 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.617397    2554 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 12:53:23.617425 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.617410    2554 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 12:53:23.617519 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.617495    2554 factory.go:55] Registering systemd factory
Jul 28 12:53:23.617519 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.617509    2554 factory.go:223] Registration of the systemd container factory successfully
Jul 28 12:53:23.617603 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.617508    2554 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 12:53:23.617603 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.617557    2554 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 12:53:23.617751 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.617737    2554 factory.go:153] Registering CRI-O factory
Jul 28 12:53:23.617830 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.617752    2554 factory.go:223] Registration of the crio container factory successfully
Jul 28 12:53:23.617830 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.617816    2554 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 12:53:23.617891 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.617839    2554 factory.go:103] Registering Raw factory
Jul 28 12:53:23.617891 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.617877    2554 manager.go:1196] Started watching for new ooms in manager
Jul 28 12:53:23.618710 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:23.618681    2554 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-177.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 12:53:23.618826 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:23.618801    2554 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 12:53:23.618955 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.618937    2554 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qv9lt"
Jul 28 12:53:23.619032 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.619019    2554 manager.go:349] Starting recovery of all containers
Jul 28 12:53:23.626725 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.626706    2554 manager.go:354] Recovery completed
Jul 28 12:53:23.629025 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:23.628994    2554 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jul 28 12:53:23.632895 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.632881    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:53:23.636020 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.636005    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-177.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:53:23.636074 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.636033    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-177.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:53:23.636074 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.636044    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-177.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:53:23.636573 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.636556    2554 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 12:53:23.636573 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.636570    2554 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 12:53:23.636573 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.636575    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:53:23.636762 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.636584    2554 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 12:53:23.637238 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.637222    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-177.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:53:23.637329 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.637253    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-177.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:53:23.637329 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.637268    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-177.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:53:23.639259 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.639245    2554 policy_none.go:49] "None policy: Start"
Jul 28 12:53:23.639322 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.639265    2554 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 12:53:23.639322 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.639297    2554 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 12:53:23.641305 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.641291    2554 policy_none.go:47] "Start"
Jul 28 12:53:23.680930 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.680799    2554 manager.go:337] "Starting Device Plugin manager"
Jul 28 12:53:23.681017 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:23.680943    2554 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 12:53:23.681017 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.680959    2554 server.go:87] "Starting device plugin registration server"
Jul 28 12:53:23.681162 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.681151    2554 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 12:53:23.681192 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.681164    2554 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 12:53:23.681374 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.681353    2554 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 12:53:23.681464 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.681440    2554 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 12:53:23.681464 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.681450    2554 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 12:53:23.681926 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:23.681906    2554 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 12:53:23.681993 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:23.681960    2554 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-177.ec2.internal\" not found"
Jul 28 12:53:23.697498 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.697478    2554 container_manager_linux.go:676] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2585 into the system container \"/system.slice\": <nil>"
Jul 28 12:53:23.781816 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.781747    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:53:23.783964 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.782722    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-177.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:53:23.783964 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.782747    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-177.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:53:23.783964 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.782759    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-177.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:53:23.783964 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.782793    2554 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-177.ec2.internal"
Jul 28 12:53:23.791092 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.791078    2554 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-177.ec2.internal"
Jul 28 12:53:23.791149 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:23.791100    2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-177.ec2.internal\": node \"ip-10-0-129-177.ec2.internal\" not found"
Jul 28 12:53:23.794443 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.794410    2554 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 12:53:23.795893 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.795877    2554 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 12:53:23.795967 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.795914    2554 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 12:53:23.795967 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.795932    2554 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 12:53:23.796042 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:23.795966    2554 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 12:53:23.798251 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.798228    2554 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 12:53:23.804290 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:23.804261    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-177.ec2.internal\" not found"
Jul 28 12:53:23.896837 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.896802    2554 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-177.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-177.ec2.internal"]
Jul 28 12:53:23.896945 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.896890    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:53:23.900492 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.900475    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-177.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:53:23.900577 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.900509    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-177.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:53:23.900577 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.900523    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-177.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:53:23.902871 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.902857    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:53:23.902992 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.902978    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-177.ec2.internal"
Jul 28 12:53:23.903039 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.903008    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:53:23.903558 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.903545    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-177.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:53:23.903626 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.903569    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-177.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:53:23.903626 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.903582    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-177.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:53:23.903626 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.903596    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-177.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:53:23.903626 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.903620    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-177.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:53:23.903818 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.903632    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-177.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:53:23.905006 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:23.904993    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-177.ec2.internal\" not found"
Jul 28 12:53:23.905818 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.905806    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-177.ec2.internal"
Jul 28 12:53:23.905863 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.905833    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 12:53:23.906552 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.906529    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-177.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 12:53:23.906631 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.906564    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-177.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 12:53:23.906631 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.906577    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-177.ec2.internal" event="NodeHasSufficientPID"
Jul 28 12:53:23.919573 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.919551    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b0d874dd23a22efefe3523ff609d1a20-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-177.ec2.internal\" (UID: \"b0d874dd23a22efefe3523ff609d1a20\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-177.ec2.internal"
Jul 28 12:53:23.919682 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.919578    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0d874dd23a22efefe3523ff609d1a20-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-177.ec2.internal\" (UID: \"b0d874dd23a22efefe3523ff609d1a20\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-177.ec2.internal"
Jul 28 12:53:23.919682 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:23.919593    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/571f960b5c36dc73834816c1792a750a-config\") pod \"kube-apiserver-proxy-ip-10-0-129-177.ec2.internal\" (UID: \"571f960b5c36dc73834816c1792a750a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-177.ec2.internal"
Jul 28 12:53:23.923789 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:23.923772    2554 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-177.ec2.internal\" not found" node="ip-10-0-129-177.ec2.internal"
Jul 28 12:53:23.927067 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:23.927052    2554 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-177.ec2.internal\" not found" node="ip-10-0-129-177.ec2.internal"
Jul 28 12:53:24.006001 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:24.005978    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-177.ec2.internal\" not found"
Jul 28 12:53:24.020408 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.020387    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b0d874dd23a22efefe3523ff609d1a20-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-177.ec2.internal\" (UID: \"b0d874dd23a22efefe3523ff609d1a20\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-177.ec2.internal"
Jul 28 12:53:24.020460 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.020413    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0d874dd23a22efefe3523ff609d1a20-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-177.ec2.internal\" (UID: \"b0d874dd23a22efefe3523ff609d1a20\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-177.ec2.internal"
Jul 28 12:53:24.020460 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.020436    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/571f960b5c36dc73834816c1792a750a-config\") pod \"kube-apiserver-proxy-ip-10-0-129-177.ec2.internal\" (UID: \"571f960b5c36dc73834816c1792a750a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-177.ec2.internal"
Jul 28 12:53:24.020523 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.020477    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b0d874dd23a22efefe3523ff609d1a20-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-177.ec2.internal\" (UID: \"b0d874dd23a22efefe3523ff609d1a20\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-177.ec2.internal"
Jul 28 12:53:24.020523 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.020488    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/571f960b5c36dc73834816c1792a750a-config\") pod \"kube-apiserver-proxy-ip-10-0-129-177.ec2.internal\" (UID: \"571f960b5c36dc73834816c1792a750a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-177.ec2.internal"
Jul 28 12:53:24.020523 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.020495    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0d874dd23a22efefe3523ff609d1a20-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-177.ec2.internal\" (UID: \"b0d874dd23a22efefe3523ff609d1a20\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-177.ec2.internal"
Jul 28 12:53:24.106874 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:24.106807    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-177.ec2.internal\" not found"
Jul 28 12:53:24.207391 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:24.207349    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-177.ec2.internal\" not found"
Jul 28 12:53:24.226830 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.226809    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-177.ec2.internal"
Jul 28 12:53:24.229975 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.229959    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-177.ec2.internal"
Jul 28 12:53:24.308252 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:24.308197    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-177.ec2.internal\" not found"
Jul 28 12:53:24.408741 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:24.408670    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-177.ec2.internal\" not found"
Jul 28 12:53:24.509267 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:24.509228    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-177.ec2.internal\" not found"
Jul 28 12:53:24.551680 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.551655    2554 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 12:53:24.552078 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.551800    2554 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 12:53:24.590728 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.590701    2554 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 12:53:24.591906 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.591893    2554 apiserver.go:52] "Watching apiserver"
Jul 28 12:53:24.602392 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.601953    2554 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 12:53:24.602835 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.602815    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-x88hp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmd95","openshift-dns/node-resolver-sn9v6","openshift-multus/multus-additional-cni-plugins-ckrz7","openshift-multus/multus-jd2p2","openshift-network-diagnostics/network-check-target-4l2q4","openshift-network-operator/iptables-alerter-248g2","openshift-cluster-node-tuning-operator/tuned-tgjnv","openshift-image-registry/node-ca-68tbg","openshift-multus/network-metrics-daemon-vvbch","openshift-ovn-kubernetes/ovnkube-node-st9kv","kube-system/global-pull-secret-syncer-jcccx"]
Jul 28 12:53:24.605320 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.605306    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-x88hp"
Jul 28 12:53:24.607472 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.607455    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmd95"
Jul 28 12:53:24.608017 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.608003    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-j2lz2\""
Jul 28 12:53:24.608601 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.608590    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 12:53:24.609584 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.609569    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sn9v6"
Jul 28 12:53:24.609685 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.609667    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 12:53:24.610244 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.610226    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 12:53:24.610486 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.610472    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 12:53:24.610904 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.610890    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 12:53:24.611092 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.611080    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vwcrj\""
Jul 28 12:53:24.611794 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.611774    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ckrz7"
Jul 28 12:53:24.612062 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.612040    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 12:53:24.612137 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.612111    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 12:53:24.612405 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.612382    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2nw72\""
Jul 28 12:53:24.613934 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.613917    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jd2p2"
Jul 28 12:53:24.614765 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.614753    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 12:53:24.615065 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.615052    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 12:53:24.615131 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.615056    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 12:53:24.615660 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.615645    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-s7sd4\""
Jul 28 12:53:24.615706 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.615648    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 12:53:24.615706 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.615647    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 12:53:24.615902 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.615890    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 12:53:24.615977 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.615895    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-7nxlq\""
Jul 28 12:53:24.616106 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.616091    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4l2q4"
Jul 28 12:53:24.616182 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:24.616161    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4l2q4" podUID="78f39c2f-8193-46af-b59d-8dc730d2e61a"
Jul 28 12:53:24.616467 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.616454    2554 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 12:53:24.616850 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.616838    2554 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-177.ec2.internal"
Jul 28 12:53:24.618222 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.618205    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-248g2"
Jul 28 12:53:24.620468 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.620452    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tgjnv"
Jul 28 12:53:24.622622 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.622606    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-68tbg"
Jul 28 12:53:24.624070 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.624051    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1118726b-19e7-4111-9934-60e937e54642-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ckrz7\" (UID: \"1118726b-19e7-4111-9934-60e937e54642\") " pod="openshift-multus/multus-additional-cni-plugins-ckrz7"
Jul 28 12:53:24.624156 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.624079    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f02e9322-ad0d-4b41-b3e4-a4c6a71f566d-os-release\") pod \"multus-jd2p2\" (UID: \"f02e9322-ad0d-4b41-b3e4-a4c6a71f566d\") " pod="openshift-multus/multus-jd2p2"
Jul 28 12:53:24.624156 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.624102    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f02e9322-ad0d-4b41-b3e4-a4c6a71f566d-multus-socket-dir-parent\") pod \"multus-jd2p2\" (UID: \"f02e9322-ad0d-4b41-b3e4-a4c6a71f566d\") " pod="openshift-multus/multus-jd2p2"
Jul 28 12:53:24.624156 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.624124    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f02e9322-ad0d-4b41-b3e4-a4c6a71f566d-host-var-lib-cni-bin\") pod \"multus-jd2p2\" (UID: \"f02e9322-ad0d-4b41-b3e4-a4c6a71f566d\") " pod="openshift-multus/multus-jd2p2"
Jul 28 12:53:24.624156 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.624148    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f02e9322-ad0d-4b41-b3e4-a4c6a71f566d-host-run-multus-certs\") pod \"multus-jd2p2\" (UID: \"f02e9322-ad0d-4b41-b3e4-a4c6a71f566d\") " pod="openshift-multus/multus-jd2p2"
Jul 28 12:53:24.624347 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.624172    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f02e9322-ad0d-4b41-b3e4-a4c6a71f566d-system-cni-dir\") pod \"multus-jd2p2\" (UID: \"f02e9322-ad0d-4b41-b3e4-a4c6a71f566d\") " pod="openshift-multus/multus-jd2p2"
Jul 28 12:53:24.624347 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.624217    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1c64d1e9-a9c9-48e8-bf71-cc650e871002-registration-dir\") pod \"aws-ebs-csi-driver-node-jmd95\" (UID: \"1c64d1e9-a9c9-48e8-bf71-cc650e871002\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmd95"
Jul 28 12:53:24.624347 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.624246    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d336b5ed-db42-40f6-9e7a-a1882cb28484-hosts-file\") pod \"node-resolver-sn9v6\" (UID: \"d336b5ed-db42-40f6-9e7a-a1882cb28484\") " pod="openshift-dns/node-resolver-sn9v6"
Jul 28 12:53:24.624347 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.624289    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1118726b-19e7-4111-9934-60e937e54642-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ckrz7\" (UID: \"1118726b-19e7-4111-9934-60e937e54642\") " pod="openshift-multus/multus-additional-cni-plugins-ckrz7"
Jul 28 12:53:24.624347 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.624319    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f02e9322-ad0d-4b41-b3e4-a4c6a71f566d-multus-cni-dir\") pod \"multus-jd2p2\" (UID: \"f02e9322-ad0d-4b41-b3e4-a4c6a71f566d\") " pod="openshift-multus/multus-jd2p2"
Jul 28 12:53:24.624347 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.624334    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f02e9322-ad0d-4b41-b3e4-a4c6a71f566d-cnibin\") pod \"multus-jd2p2\" (UID: \"f02e9322-ad0d-4b41-b3e4-a4c6a71f566d\") " pod="openshift-multus/multus-jd2p2"
Jul 28 12:53:24.624347 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.624347    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f02e9322-ad0d-4b41-b3e4-a4c6a71f566d-host-var-lib-kubelet\") pod \"multus-jd2p2\" (UID: \"f02e9322-ad0d-4b41-b3e4-a4c6a71f566d\") " pod="openshift-multus/multus-jd2p2"
Jul 28 12:53:24.624549 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.624368    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn4tv\" (UniqueName: \"kubernetes.io/projected/f02e9322-ad0d-4b41-b3e4-a4c6a71f566d-kube-api-access-jn4tv\") pod \"multus-jd2p2\" (UID: \"f02e9322-ad0d-4b41-b3e4-a4c6a71f566d\") " pod="openshift-multus/multus-jd2p2"
Jul 28 12:53:24.624549 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.624412    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkn8s\" (UniqueName: \"kubernetes.io/projected/1118726b-19e7-4111-9934-60e937e54642-kube-api-access-pkn8s\") pod \"multus-additional-cni-plugins-ckrz7\" (UID: \"1118726b-19e7-4111-9934-60e937e54642\") " pod="openshift-multus/multus-additional-cni-plugins-ckrz7"
Jul 28 12:53:24.624549 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.624442    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f02e9322-ad0d-4b41-b3e4-a4c6a71f566d-host-run-netns\") pod \"multus-jd2p2\" (UID: \"f02e9322-ad0d-4b41-b3e4-a4c6a71f566d\") " pod="openshift-multus/multus-jd2p2"
Jul 28 12:53:24.624549 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.624459    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f02e9322-ad0d-4b41-b3e4-a4c6a71f566d-host-var-lib-cni-multus\") pod \"multus-jd2p2\" (UID: \"f02e9322-ad0d-4b41-b3e4-a4c6a71f566d\") " pod="openshift-multus/multus-jd2p2"
Jul 28 12:53:24.624549 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.624475    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1c64d1e9-a9c9-48e8-bf71-cc650e871002-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jmd95\" (UID: \"1c64d1e9-a9c9-48e8-bf71-cc650e871002\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmd95"
Jul 28 12:53:24.624549 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.624488    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1c64d1e9-a9c9-48e8-bf71-cc650e871002-socket-dir\") pod \"aws-ebs-csi-driver-node-jmd95\" (UID: \"1c64d1e9-a9c9-48e8-bf71-cc650e871002\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmd95"
Jul 28 12:53:24.624549 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.624509    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1c64d1e9-a9c9-48e8-bf71-cc650e871002-device-dir\") pod \"aws-ebs-csi-driver-node-jmd95\" (UID: \"1c64d1e9-a9c9-48e8-bf71-cc650e871002\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmd95"
Jul 28 12:53:24.624549 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.624531    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d336b5ed-db42-40f6-9e7a-a1882cb28484-tmp-dir\") pod \"node-resolver-sn9v6\" (UID: \"d336b5ed-db42-40f6-9e7a-a1882cb28484\") " pod="openshift-dns/node-resolver-sn9v6"
Jul 28 12:53:24.624868 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.624567    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5731c918-3db8-447b-bda3-7ac8427ab9df-iptables-alerter-script\") pod \"iptables-alerter-248g2\" (UID: \"5731c918-3db8-447b-bda3-7ac8427ab9df\") " pod="openshift-network-operator/iptables-alerter-248g2"
Jul 28 12:53:24.624868 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.624605    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f02e9322-ad0d-4b41-b3e4-a4c6a71f566d-host-run-k8s-cni-cncf-io\") pod \"multus-jd2p2\" (UID: \"f02e9322-ad0d-4b41-b3e4-a4c6a71f566d\") " pod="openshift-multus/multus-jd2p2"
Jul 28 12:53:24.624868 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.624636    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f02e9322-ad0d-4b41-b3e4-a4c6a71f566d-multus-daemon-config\") pod \"multus-jd2p2\" (UID: \"f02e9322-ad0d-4b41-b3e4-a4c6a71f566d\") " pod="openshift-multus/multus-jd2p2"
Jul 28 12:53:24.624868 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.624663    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1118726b-19e7-4111-9934-60e937e54642-os-release\") pod \"multus-additional-cni-plugins-ckrz7\" (UID: \"1118726b-19e7-4111-9934-60e937e54642\") " pod="openshift-multus/multus-additional-cni-plugins-ckrz7"
Jul 28 12:53:24.624868 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.624687    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f02e9322-ad0d-4b41-b3e4-a4c6a71f566d-hostroot\") pod \"multus-jd2p2\" (UID: \"f02e9322-ad0d-4b41-b3e4-a4c6a71f566d\") " pod="openshift-multus/multus-jd2p2"
Jul 28 12:53:24.624868 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.624724    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1c64d1e9-a9c9-48e8-bf71-cc650e871002-sys-fs\") pod \"aws-ebs-csi-driver-node-jmd95\" (UID: \"1c64d1e9-a9c9-48e8-bf71-cc650e871002\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmd95"
Jul 28 12:53:24.624868 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.624774    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1118726b-19e7-4111-9934-60e937e54642-cni-binary-copy\") pod \"multus-additional-cni-plugins-ckrz7\" (UID: \"1118726b-19e7-4111-9934-60e937e54642\") " pod="openshift-multus/multus-additional-cni-plugins-ckrz7"
Jul 28 12:53:24.624868 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.624790    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vvbch"
Jul 28 12:53:24.624868 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.624803    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vlhn\" (UniqueName: \"kubernetes.io/projected/78f39c2f-8193-46af-b59d-8dc730d2e61a-kube-api-access-7vlhn\") pod \"network-check-target-4l2q4\" (UID: \"78f39c2f-8193-46af-b59d-8dc730d2e61a\") " pod="openshift-network-diagnostics/network-check-target-4l2q4"
Jul 28 12:53:24.624868 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.624826    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5731c918-3db8-447b-bda3-7ac8427ab9df-host-slash\") pod \"iptables-alerter-248g2\" (UID: \"5731c918-3db8-447b-bda3-7ac8427ab9df\") " pod="openshift-network-operator/iptables-alerter-248g2"
Jul 28 12:53:24.624868 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:24.624845    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vvbch" podUID="d04af0e5-3e6d-4027-8509-0af92acc4035"
Jul 28 12:53:24.624868 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.624862    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4afbaf34-0b91-4704-9e53-39f24049c3c7-konnectivity-ca\") pod \"konnectivity-agent-x88hp\" (UID: \"4afbaf34-0b91-4704-9e53-39f24049c3c7\") " pod="kube-system/konnectivity-agent-x88hp"
Jul 28 12:53:24.625255 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.624885    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f02e9322-ad0d-4b41-b3e4-a4c6a71f566d-cni-binary-copy\") pod \"multus-jd2p2\" (UID: \"f02e9322-ad0d-4b41-b3e4-a4c6a71f566d\") " pod="openshift-multus/multus-jd2p2"
Jul 28 12:53:24.625255 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.624910    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1c64d1e9-a9c9-48e8-bf71-cc650e871002-etc-selinux\") pod \"aws-ebs-csi-driver-node-jmd95\" (UID: \"1c64d1e9-a9c9-48e8-bf71-cc650e871002\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmd95"
Jul 28 12:53:24.625255 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.624935    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzjxv\" (UniqueName: \"kubernetes.io/projected/1c64d1e9-a9c9-48e8-bf71-cc650e871002-kube-api-access-pzjxv\") pod \"aws-ebs-csi-driver-node-jmd95\" (UID: \"1c64d1e9-a9c9-48e8-bf71-cc650e871002\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmd95"
Jul 28 12:53:24.625255 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.624974    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz4tc\" (UniqueName: \"kubernetes.io/projected/d336b5ed-db42-40f6-9e7a-a1882cb28484-kube-api-access-mz4tc\") pod \"node-resolver-sn9v6\" (UID: \"d336b5ed-db42-40f6-9e7a-a1882cb28484\") " pod="openshift-dns/node-resolver-sn9v6"
Jul 28 12:53:24.625255 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.624997    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1118726b-19e7-4111-9934-60e937e54642-system-cni-dir\") pod \"multus-additional-cni-plugins-ckrz7\" (UID: \"1118726b-19e7-4111-9934-60e937e54642\") " pod="openshift-multus/multus-additional-cni-plugins-ckrz7"
Jul 28 12:53:24.625255 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.625019    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1118726b-19e7-4111-9934-60e937e54642-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ckrz7\" (UID: \"1118726b-19e7-4111-9934-60e937e54642\") " pod="openshift-multus/multus-additional-cni-plugins-ckrz7"
Jul 28 12:53:24.625255 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.625042    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vzsc\" (UniqueName: \"kubernetes.io/projected/5731c918-3db8-447b-bda3-7ac8427ab9df-kube-api-access-5vzsc\") pod \"iptables-alerter-248g2\" (UID: \"5731c918-3db8-447b-bda3-7ac8427ab9df\") " pod="openshift-network-operator/iptables-alerter-248g2"
Jul 28 12:53:24.625255 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.625065    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4afbaf34-0b91-4704-9e53-39f24049c3c7-agent-certs\") pod \"konnectivity-agent-x88hp\" (UID: \"4afbaf34-0b91-4704-9e53-39f24049c3c7\") " pod="kube-system/konnectivity-agent-x88hp"
Jul 28 12:53:24.625255 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.625089    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f02e9322-ad0d-4b41-b3e4-a4c6a71f566d-multus-conf-dir\") pod \"multus-jd2p2\" (UID: \"f02e9322-ad0d-4b41-b3e4-a4c6a71f566d\") " pod="openshift-multus/multus-jd2p2"
Jul 28 12:53:24.625255 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.625118    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1118726b-19e7-4111-9934-60e937e54642-cnibin\") pod \"multus-additional-cni-plugins-ckrz7\" (UID: \"1118726b-19e7-4111-9934-60e937e54642\") " pod="openshift-multus/multus-additional-cni-plugins-ckrz7"
Jul 28 12:53:24.625255 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.625145    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f02e9322-ad0d-4b41-b3e4-a4c6a71f566d-etc-kubernetes\") pod \"multus-jd2p2\" (UID: \"f02e9322-ad0d-4b41-b3e4-a4c6a71f566d\") " pod="openshift-multus/multus-jd2p2"
Jul 28 12:53:24.625766 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.625397    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 12:53:24.625766 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.625413    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 12:53:24.626219 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.626052    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 12:53:24.626219 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.626066    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-dgmgt\""
Jul 28 12:53:24.626219 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.626081    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 12:53:24.626219 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.626123    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 12:53:24.626219 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.626122    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 12:53:24.626219 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.626166    2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 12:48:24 +0000 UTC" deadline="2028-02-26 11:24:47.848774061 +0000 UTC"
Jul 28 12:53:24.626219 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.626219    2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13870h31m23.222558074s"
Jul 28 12:53:24.626496 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.626054    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 12:53:24.626496 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.626066    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sqwtd\""
Jul 28 12:53:24.626816 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.626803    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 12:53:24.626865 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.626838    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4tvpp\""
Jul 28 12:53:24.627185 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.627171    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.629572 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.629554    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jcccx"
Jul 28 12:53:24.629656 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:24.629612    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jcccx" podUID="9f6f2eb9-22ea-4119-b474-75c3d6ec08eb"
Jul 28 12:53:24.630478 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.630462    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 12:53:24.630633 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.630619    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 12:53:24.630700 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.630689    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 12:53:24.630764 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.630701    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 12:53:24.631614 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.631599    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cgp79\""
Jul 28 12:53:24.631695 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.631611    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 12:53:24.631695 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.631660    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 12:53:24.634048 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.634034    2554 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 12:53:24.634132 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.634057    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-177.ec2.internal"]
Jul 28 12:53:24.634187 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.634164    2554 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-177.ec2.internal"
Jul 28 12:53:24.643552 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.643531    2554 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 12:53:24.655519 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.655497    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-129-177.ec2.internal"]
Jul 28 12:53:24.655766 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.655753    2554 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 12:53:24.669207 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.669189    2554 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5sp2g"
Jul 28 12:53:24.678697 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.678681    2554 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5sp2g"
Jul 28 12:53:24.718257 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.718241    2554 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 12:53:24.725477 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.725458    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1118726b-19e7-4111-9934-60e937e54642-system-cni-dir\") pod \"multus-additional-cni-plugins-ckrz7\" (UID: \"1118726b-19e7-4111-9934-60e937e54642\") " pod="openshift-multus/multus-additional-cni-plugins-ckrz7"
Jul 28 12:53:24.725570 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.725493    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vzsc\" (UniqueName: \"kubernetes.io/projected/5731c918-3db8-447b-bda3-7ac8427ab9df-kube-api-access-5vzsc\") pod \"iptables-alerter-248g2\" (UID: \"5731c918-3db8-447b-bda3-7ac8427ab9df\") " pod="openshift-network-operator/iptables-alerter-248g2"
Jul 28 12:53:24.725570 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.725523    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4afbaf34-0b91-4704-9e53-39f24049c3c7-agent-certs\") pod \"konnectivity-agent-x88hp\" (UID: \"4afbaf34-0b91-4704-9e53-39f24049c3c7\") " pod="kube-system/konnectivity-agent-x88hp"
Jul 28 12:53:24.725570 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.725555    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a71336e9-5f7a-4c21-8a5f-6402450ed0fa-etc-systemd\") pod \"tuned-tgjnv\" (UID: \"a71336e9-5f7a-4c21-8a5f-6402450ed0fa\") " pod="openshift-cluster-node-tuning-operator/tuned-tgjnv"
Jul 28 12:53:24.725769 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.725573    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1118726b-19e7-4111-9934-60e937e54642-system-cni-dir\") pod \"multus-additional-cni-plugins-ckrz7\" (UID: \"1118726b-19e7-4111-9934-60e937e54642\") " pod="openshift-multus/multus-additional-cni-plugins-ckrz7"
Jul 28 12:53:24.725769 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.725582    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-etc-openvswitch\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.725769 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.725606    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a71336e9-5f7a-4c21-8a5f-6402450ed0fa-host\") pod \"tuned-tgjnv\" (UID: \"a71336e9-5f7a-4c21-8a5f-6402450ed0fa\") " pod="openshift-cluster-node-tuning-operator/tuned-tgjnv"
Jul 28 12:53:24.725769 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.725658    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9f6f2eb9-22ea-4119-b474-75c3d6ec08eb-original-pull-secret\") pod \"global-pull-secret-syncer-jcccx\" (UID: \"9f6f2eb9-22ea-4119-b474-75c3d6ec08eb\") " pod="kube-system/global-pull-secret-syncer-jcccx"
Jul 28 12:53:24.725769 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.725703    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f02e9322-ad0d-4b41-b3e4-a4c6a71f566d-host-run-multus-certs\") pod \"multus-jd2p2\" (UID: \"f02e9322-ad0d-4b41-b3e4-a4c6a71f566d\") " pod="openshift-multus/multus-jd2p2"
Jul 28 12:53:24.725769 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.725730    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a71336e9-5f7a-4c21-8a5f-6402450ed0fa-var-lib-kubelet\") pod \"tuned-tgjnv\" (UID: \"a71336e9-5f7a-4c21-8a5f-6402450ed0fa\") " pod="openshift-cluster-node-tuning-operator/tuned-tgjnv"
Jul 28 12:53:24.725769 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.725754    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f02e9322-ad0d-4b41-b3e4-a4c6a71f566d-system-cni-dir\") pod \"multus-jd2p2\" (UID: \"f02e9322-ad0d-4b41-b3e4-a4c6a71f566d\") " pod="openshift-multus/multus-jd2p2"
Jul 28 12:53:24.726044 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.725820    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1118726b-19e7-4111-9934-60e937e54642-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ckrz7\" (UID: \"1118726b-19e7-4111-9934-60e937e54642\") " pod="openshift-multus/multus-additional-cni-plugins-ckrz7"
Jul 28 12:53:24.726044 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.725868    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f02e9322-ad0d-4b41-b3e4-a4c6a71f566d-multus-cni-dir\") pod \"multus-jd2p2\" (UID: \"f02e9322-ad0d-4b41-b3e4-a4c6a71f566d\") " pod="openshift-multus/multus-jd2p2"
Jul 28 12:53:24.726044 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.725892    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f02e9322-ad0d-4b41-b3e4-a4c6a71f566d-cnibin\") pod \"multus-jd2p2\" (UID: \"f02e9322-ad0d-4b41-b3e4-a4c6a71f566d\") " pod="openshift-multus/multus-jd2p2"
Jul 28 12:53:24.726044 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.725897    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f02e9322-ad0d-4b41-b3e4-a4c6a71f566d-host-run-multus-certs\") pod \"multus-jd2p2\" (UID: \"f02e9322-ad0d-4b41-b3e4-a4c6a71f566d\") " pod="openshift-multus/multus-jd2p2"
Jul 28 12:53:24.726044 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.725913    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f02e9322-ad0d-4b41-b3e4-a4c6a71f566d-host-var-lib-kubelet\") pod \"multus-jd2p2\" (UID: \"f02e9322-ad0d-4b41-b3e4-a4c6a71f566d\") " pod="openshift-multus/multus-jd2p2"
Jul 28 12:53:24.726044 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.725866    2554 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 12:53:24.726044 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.725956    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f02e9322-ad0d-4b41-b3e4-a4c6a71f566d-host-var-lib-kubelet\") pod \"multus-jd2p2\" (UID: \"f02e9322-ad0d-4b41-b3e4-a4c6a71f566d\") " pod="openshift-multus/multus-jd2p2"
Jul 28 12:53:24.726044 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.725969    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f02e9322-ad0d-4b41-b3e4-a4c6a71f566d-system-cni-dir\") pod \"multus-jd2p2\" (UID: \"f02e9322-ad0d-4b41-b3e4-a4c6a71f566d\") " pod="openshift-multus/multus-jd2p2"
Jul 28 12:53:24.726044 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.726006    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-host-run-ovn-kubernetes\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.726044 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.726026    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f02e9322-ad0d-4b41-b3e4-a4c6a71f566d-multus-cni-dir\") pod \"multus-jd2p2\" (UID: \"f02e9322-ad0d-4b41-b3e4-a4c6a71f566d\") " pod="openshift-multus/multus-jd2p2"
Jul 28 12:53:24.726044 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.726033    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-env-overrides\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.726468 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.726007    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f02e9322-ad0d-4b41-b3e4-a4c6a71f566d-cnibin\") pod \"multus-jd2p2\" (UID: \"f02e9322-ad0d-4b41-b3e4-a4c6a71f566d\") " pod="openshift-multus/multus-jd2p2"
Jul 28 12:53:24.726468 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.726062    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkn8s\" (UniqueName: \"kubernetes.io/projected/1118726b-19e7-4111-9934-60e937e54642-kube-api-access-pkn8s\") pod \"multus-additional-cni-plugins-ckrz7\" (UID: \"1118726b-19e7-4111-9934-60e937e54642\") " pod="openshift-multus/multus-additional-cni-plugins-ckrz7"
Jul 28 12:53:24.726468 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.726088    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f02e9322-ad0d-4b41-b3e4-a4c6a71f566d-host-run-netns\") pod \"multus-jd2p2\" (UID: \"f02e9322-ad0d-4b41-b3e4-a4c6a71f566d\") " pod="openshift-multus/multus-jd2p2"
Jul 28 12:53:24.726468 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.726113    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-systemd-units\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.726468 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.726138    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/48e4bcde-dcb9-4e7e-8b15-e8682eb564d1-host\") pod \"node-ca-68tbg\" (UID: \"48e4bcde-dcb9-4e7e-8b15-e8682eb564d1\") " pod="openshift-image-registry/node-ca-68tbg"
Jul 28 12:53:24.726468 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.726167    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1c64d1e9-a9c9-48e8-bf71-cc650e871002-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jmd95\" (UID: \"1c64d1e9-a9c9-48e8-bf71-cc650e871002\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmd95"
Jul 28 12:53:24.726468 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.726178    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f02e9322-ad0d-4b41-b3e4-a4c6a71f566d-host-run-netns\") pod \"multus-jd2p2\" (UID: \"f02e9322-ad0d-4b41-b3e4-a4c6a71f566d\") " pod="openshift-multus/multus-jd2p2"
Jul 28 12:53:24.726468 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.726194    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1c64d1e9-a9c9-48e8-bf71-cc650e871002-socket-dir\") pod \"aws-ebs-csi-driver-node-jmd95\" (UID: \"1c64d1e9-a9c9-48e8-bf71-cc650e871002\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmd95"
Jul 28 12:53:24.726468 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.726234    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1c64d1e9-a9c9-48e8-bf71-cc650e871002-device-dir\") pod \"aws-ebs-csi-driver-node-jmd95\" (UID: \"1c64d1e9-a9c9-48e8-bf71-cc650e871002\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmd95"
Jul 28 12:53:24.726468 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.726250    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1c64d1e9-a9c9-48e8-bf71-cc650e871002-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jmd95\" (UID: \"1c64d1e9-a9c9-48e8-bf71-cc650e871002\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmd95"
Jul 28 12:53:24.726468 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.726258    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d336b5ed-db42-40f6-9e7a-a1882cb28484-tmp-dir\") pod \"node-resolver-sn9v6\" (UID: \"d336b5ed-db42-40f6-9e7a-a1882cb28484\") " pod="openshift-dns/node-resolver-sn9v6"
Jul 28 12:53:24.726468 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.726326    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1c64d1e9-a9c9-48e8-bf71-cc650e871002-socket-dir\") pod \"aws-ebs-csi-driver-node-jmd95\" (UID: \"1c64d1e9-a9c9-48e8-bf71-cc650e871002\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmd95"
Jul 28 12:53:24.726468 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.726327    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f02e9322-ad0d-4b41-b3e4-a4c6a71f566d-host-run-k8s-cni-cncf-io\") pod \"multus-jd2p2\" (UID: \"f02e9322-ad0d-4b41-b3e4-a4c6a71f566d\") " pod="openshift-multus/multus-jd2p2"
Jul 28 12:53:24.726468 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.726348    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1c64d1e9-a9c9-48e8-bf71-cc650e871002-device-dir\") pod \"aws-ebs-csi-driver-node-jmd95\" (UID: \"1c64d1e9-a9c9-48e8-bf71-cc650e871002\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmd95"
Jul 28 12:53:24.726468 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.726355    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f02e9322-ad0d-4b41-b3e4-a4c6a71f566d-host-run-k8s-cni-cncf-io\") pod \"multus-jd2p2\" (UID: \"f02e9322-ad0d-4b41-b3e4-a4c6a71f566d\") " pod="openshift-multus/multus-jd2p2"
Jul 28 12:53:24.726468 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.726374    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj77c\" (UniqueName: \"kubernetes.io/projected/d04af0e5-3e6d-4027-8509-0af92acc4035-kube-api-access-rj77c\") pod \"network-metrics-daemon-vvbch\" (UID: \"d04af0e5-3e6d-4027-8509-0af92acc4035\") " pod="openshift-multus/network-metrics-daemon-vvbch"
Jul 28 12:53:24.726468 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.726405    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1118726b-19e7-4111-9934-60e937e54642-os-release\") pod \"multus-additional-cni-plugins-ckrz7\" (UID: \"1118726b-19e7-4111-9934-60e937e54642\") " pod="openshift-multus/multus-additional-cni-plugins-ckrz7"
Jul 28 12:53:24.727312 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.726430    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f02e9322-ad0d-4b41-b3e4-a4c6a71f566d-hostroot\") pod \"multus-jd2p2\" (UID: \"f02e9322-ad0d-4b41-b3e4-a4c6a71f566d\") " pod="openshift-multus/multus-jd2p2"
Jul 28 12:53:24.727312 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.726456    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-node-log\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.727312 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.726480    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-ovn-node-metrics-cert\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.727312 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.726490    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f02e9322-ad0d-4b41-b3e4-a4c6a71f566d-hostroot\") pod \"multus-jd2p2\" (UID: \"f02e9322-ad0d-4b41-b3e4-a4c6a71f566d\") " pod="openshift-multus/multus-jd2p2"
Jul 28 12:53:24.727312 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.726505    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8txxr\" (UniqueName: \"kubernetes.io/projected/a71336e9-5f7a-4c21-8a5f-6402450ed0fa-kube-api-access-8txxr\") pod \"tuned-tgjnv\" (UID: \"a71336e9-5f7a-4c21-8a5f-6402450ed0fa\") " pod="openshift-cluster-node-tuning-operator/tuned-tgjnv"
Jul 28 12:53:24.727312 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.726510    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1118726b-19e7-4111-9934-60e937e54642-os-release\") pod \"multus-additional-cni-plugins-ckrz7\" (UID: \"1118726b-19e7-4111-9934-60e937e54642\") " pod="openshift-multus/multus-additional-cni-plugins-ckrz7"
Jul 28 12:53:24.727312 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.726522    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d336b5ed-db42-40f6-9e7a-a1882cb28484-tmp-dir\") pod \"node-resolver-sn9v6\" (UID: \"d336b5ed-db42-40f6-9e7a-a1882cb28484\") " pod="openshift-dns/node-resolver-sn9v6"
Jul 28 12:53:24.727312 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.726530    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1c64d1e9-a9c9-48e8-bf71-cc650e871002-sys-fs\") pod \"aws-ebs-csi-driver-node-jmd95\" (UID: \"1c64d1e9-a9c9-48e8-bf71-cc650e871002\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmd95"
Jul 28 12:53:24.727312 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.726556    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vlhn\" (UniqueName: \"kubernetes.io/projected/78f39c2f-8193-46af-b59d-8dc730d2e61a-kube-api-access-7vlhn\") pod \"network-check-target-4l2q4\" (UID: \"78f39c2f-8193-46af-b59d-8dc730d2e61a\") " pod="openshift-network-diagnostics/network-check-target-4l2q4"
Jul 28 12:53:24.727312 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.726594    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1c64d1e9-a9c9-48e8-bf71-cc650e871002-sys-fs\") pod \"aws-ebs-csi-driver-node-jmd95\" (UID: \"1c64d1e9-a9c9-48e8-bf71-cc650e871002\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmd95"
Jul 28 12:53:24.727312 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.726643    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f02e9322-ad0d-4b41-b3e4-a4c6a71f566d-cni-binary-copy\") pod \"multus-jd2p2\" (UID: \"f02e9322-ad0d-4b41-b3e4-a4c6a71f566d\") " pod="openshift-multus/multus-jd2p2"
Jul 28 12:53:24.727312 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.726769    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-ovnkube-config\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.727312 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.726823    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9f6f2eb9-22ea-4119-b474-75c3d6ec08eb-kubelet-config\") pod \"global-pull-secret-syncer-jcccx\" (UID: \"9f6f2eb9-22ea-4119-b474-75c3d6ec08eb\") " pod="kube-system/global-pull-secret-syncer-jcccx"
Jul 28 12:53:24.727312 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.726854    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1c64d1e9-a9c9-48e8-bf71-cc650e871002-etc-selinux\") pod \"aws-ebs-csi-driver-node-jmd95\" (UID: \"1c64d1e9-a9c9-48e8-bf71-cc650e871002\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmd95"
Jul 28 12:53:24.727312 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.726875    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzjxv\" (UniqueName: \"kubernetes.io/projected/1c64d1e9-a9c9-48e8-bf71-cc650e871002-kube-api-access-pzjxv\") pod \"aws-ebs-csi-driver-node-jmd95\" (UID: \"1c64d1e9-a9c9-48e8-bf71-cc650e871002\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmd95"
Jul 28 12:53:24.727312 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.726896    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz4tc\" (UniqueName: \"kubernetes.io/projected/d336b5ed-db42-40f6-9e7a-a1882cb28484-kube-api-access-mz4tc\") pod \"node-resolver-sn9v6\" (UID: \"d336b5ed-db42-40f6-9e7a-a1882cb28484\") " pod="openshift-dns/node-resolver-sn9v6"
Jul 28 12:53:24.727312 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.726917    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1118726b-19e7-4111-9934-60e937e54642-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ckrz7\" (UID: \"1118726b-19e7-4111-9934-60e937e54642\") " pod="openshift-multus/multus-additional-cni-plugins-ckrz7"
Jul 28 12:53:24.728099 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.726939    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f02e9322-ad0d-4b41-b3e4-a4c6a71f566d-multus-conf-dir\") pod \"multus-jd2p2\" (UID: \"f02e9322-ad0d-4b41-b3e4-a4c6a71f566d\") " pod="openshift-multus/multus-jd2p2"
Jul 28 12:53:24.728099 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.726987    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f02e9322-ad0d-4b41-b3e4-a4c6a71f566d-multus-conf-dir\") pod \"multus-jd2p2\" (UID: \"f02e9322-ad0d-4b41-b3e4-a4c6a71f566d\") " pod="openshift-multus/multus-jd2p2"
Jul 28 12:53:24.728099 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.726999    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1c64d1e9-a9c9-48e8-bf71-cc650e871002-etc-selinux\") pod \"aws-ebs-csi-driver-node-jmd95\" (UID: \"1c64d1e9-a9c9-48e8-bf71-cc650e871002\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmd95"
Jul 28 12:53:24.728099 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.727147    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a71336e9-5f7a-4c21-8a5f-6402450ed0fa-etc-sysctl-d\") pod \"tuned-tgjnv\" (UID: \"a71336e9-5f7a-4c21-8a5f-6402450ed0fa\") " pod="openshift-cluster-node-tuning-operator/tuned-tgjnv"
Jul 28 12:53:24.728099 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.727178    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9f6f2eb9-22ea-4119-b474-75c3d6ec08eb-dbus\") pod \"global-pull-secret-syncer-jcccx\" (UID: \"9f6f2eb9-22ea-4119-b474-75c3d6ec08eb\") " pod="kube-system/global-pull-secret-syncer-jcccx"
Jul 28 12:53:24.728099 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.727205    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1118726b-19e7-4111-9934-60e937e54642-cnibin\") pod \"multus-additional-cni-plugins-ckrz7\" (UID: \"1118726b-19e7-4111-9934-60e937e54642\") " pod="openshift-multus/multus-additional-cni-plugins-ckrz7"
Jul 28 12:53:24.728099 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.727232    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f02e9322-ad0d-4b41-b3e4-a4c6a71f566d-etc-kubernetes\") pod \"multus-jd2p2\" (UID: \"f02e9322-ad0d-4b41-b3e4-a4c6a71f566d\") " pod="openshift-multus/multus-jd2p2"
Jul 28 12:53:24.728099 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.727251    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f02e9322-ad0d-4b41-b3e4-a4c6a71f566d-cni-binary-copy\") pod \"multus-jd2p2\" (UID: \"f02e9322-ad0d-4b41-b3e4-a4c6a71f566d\") " pod="openshift-multus/multus-jd2p2"
Jul 28 12:53:24.728099 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.727289    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d04af0e5-3e6d-4027-8509-0af92acc4035-metrics-certs\") pod \"network-metrics-daemon-vvbch\" (UID: \"d04af0e5-3e6d-4027-8509-0af92acc4035\") " pod="openshift-multus/network-metrics-daemon-vvbch"
Jul 28 12:53:24.728099 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.727314    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1118726b-19e7-4111-9934-60e937e54642-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ckrz7\" (UID: \"1118726b-19e7-4111-9934-60e937e54642\") " pod="openshift-multus/multus-additional-cni-plugins-ckrz7"
Jul 28 12:53:24.728099 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.727547    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1118726b-19e7-4111-9934-60e937e54642-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ckrz7\" (UID: \"1118726b-19e7-4111-9934-60e937e54642\") " pod="openshift-multus/multus-additional-cni-plugins-ckrz7"
Jul 28 12:53:24.728099 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.727650    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f02e9322-ad0d-4b41-b3e4-a4c6a71f566d-etc-kubernetes\") pod \"multus-jd2p2\" (UID: \"f02e9322-ad0d-4b41-b3e4-a4c6a71f566d\") " pod="openshift-multus/multus-jd2p2"
Jul 28 12:53:24.728099 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.727690    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1118726b-19e7-4111-9934-60e937e54642-cnibin\") pod \"multus-additional-cni-plugins-ckrz7\" (UID: \"1118726b-19e7-4111-9934-60e937e54642\") " pod="openshift-multus/multus-additional-cni-plugins-ckrz7"
Jul 28 12:53:24.728099 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.727726    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-run-systemd\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.728099 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.727755    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-run-openvswitch\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.728099 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.727776    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-host-cni-bin\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.728099 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.727792    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a71336e9-5f7a-4c21-8a5f-6402450ed0fa-etc-modprobe-d\") pod \"tuned-tgjnv\" (UID: \"a71336e9-5f7a-4c21-8a5f-6402450ed0fa\") " pod="openshift-cluster-node-tuning-operator/tuned-tgjnv"
Jul 28 12:53:24.728812 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.727813    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a71336e9-5f7a-4c21-8a5f-6402450ed0fa-run\") pod \"tuned-tgjnv\" (UID: \"a71336e9-5f7a-4c21-8a5f-6402450ed0fa\") " pod="openshift-cluster-node-tuning-operator/tuned-tgjnv"
Jul 28 12:53:24.728812 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.727831    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1118726b-19e7-4111-9934-60e937e54642-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ckrz7\" (UID: \"1118726b-19e7-4111-9934-60e937e54642\") " pod="openshift-multus/multus-additional-cni-plugins-ckrz7"
Jul 28 12:53:24.728812 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.727848    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f02e9322-ad0d-4b41-b3e4-a4c6a71f566d-os-release\") pod \"multus-jd2p2\" (UID: \"f02e9322-ad0d-4b41-b3e4-a4c6a71f566d\") " pod="openshift-multus/multus-jd2p2"
Jul 28 12:53:24.728812 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.727897    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f02e9322-ad0d-4b41-b3e4-a4c6a71f566d-os-release\") pod \"multus-jd2p2\" (UID: \"f02e9322-ad0d-4b41-b3e4-a4c6a71f566d\") " pod="openshift-multus/multus-jd2p2"
Jul 28 12:53:24.728812 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.727917    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f02e9322-ad0d-4b41-b3e4-a4c6a71f566d-multus-socket-dir-parent\") pod \"multus-jd2p2\" (UID: \"f02e9322-ad0d-4b41-b3e4-a4c6a71f566d\") " pod="openshift-multus/multus-jd2p2"
Jul 28 12:53:24.728812 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.727932    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f02e9322-ad0d-4b41-b3e4-a4c6a71f566d-host-var-lib-cni-bin\") pod \"multus-jd2p2\" (UID: \"f02e9322-ad0d-4b41-b3e4-a4c6a71f566d\") " pod="openshift-multus/multus-jd2p2"
Jul 28 12:53:24.728812 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.727949    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-run-ovn\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.728812 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.727973    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.728812 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.727987    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f02e9322-ad0d-4b41-b3e4-a4c6a71f566d-host-var-lib-cni-bin\") pod \"multus-jd2p2\" (UID: \"f02e9322-ad0d-4b41-b3e4-a4c6a71f566d\") " pod="openshift-multus/multus-jd2p2"
Jul 28 12:53:24.728812 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.727998    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7b6k\" (UniqueName: \"kubernetes.io/projected/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-kube-api-access-w7b6k\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.728812 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.727989    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f02e9322-ad0d-4b41-b3e4-a4c6a71f566d-multus-socket-dir-parent\") pod \"multus-jd2p2\" (UID: \"f02e9322-ad0d-4b41-b3e4-a4c6a71f566d\") " pod="openshift-multus/multus-jd2p2"
Jul 28 12:53:24.728812 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.728029    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a71336e9-5f7a-4c21-8a5f-6402450ed0fa-lib-modules\") pod \"tuned-tgjnv\" (UID: \"a71336e9-5f7a-4c21-8a5f-6402450ed0fa\") " pod="openshift-cluster-node-tuning-operator/tuned-tgjnv"
Jul 28 12:53:24.728812 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.728059    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1c64d1e9-a9c9-48e8-bf71-cc650e871002-registration-dir\") pod \"aws-ebs-csi-driver-node-jmd95\" (UID: \"1c64d1e9-a9c9-48e8-bf71-cc650e871002\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmd95"
Jul 28 12:53:24.728812 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.728080    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d336b5ed-db42-40f6-9e7a-a1882cb28484-hosts-file\") pod \"node-resolver-sn9v6\" (UID: \"d336b5ed-db42-40f6-9e7a-a1882cb28484\") " pod="openshift-dns/node-resolver-sn9v6"
Jul 28 12:53:24.728812 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.728100    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn4tv\" (UniqueName: \"kubernetes.io/projected/f02e9322-ad0d-4b41-b3e4-a4c6a71f566d-kube-api-access-jn4tv\") pod \"multus-jd2p2\" (UID: \"f02e9322-ad0d-4b41-b3e4-a4c6a71f566d\") " pod="openshift-multus/multus-jd2p2"
Jul 28 12:53:24.728812 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.728122    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d336b5ed-db42-40f6-9e7a-a1882cb28484-hosts-file\") pod \"node-resolver-sn9v6\" (UID: \"d336b5ed-db42-40f6-9e7a-a1882cb28484\") " pod="openshift-dns/node-resolver-sn9v6"
Jul 28 12:53:24.728812 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.728120    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-host-cni-netd\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.729488 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.728146    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1c64d1e9-a9c9-48e8-bf71-cc650e871002-registration-dir\") pod \"aws-ebs-csi-driver-node-jmd95\" (UID: \"1c64d1e9-a9c9-48e8-bf71-cc650e871002\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmd95"
Jul 28 12:53:24.729488 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.728144    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a71336e9-5f7a-4c21-8a5f-6402450ed0fa-tmp\") pod \"tuned-tgjnv\" (UID: \"a71336e9-5f7a-4c21-8a5f-6402450ed0fa\") " pod="openshift-cluster-node-tuning-operator/tuned-tgjnv"
Jul 28 12:53:24.729488 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.728192    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/48e4bcde-dcb9-4e7e-8b15-e8682eb564d1-serviceca\") pod \"node-ca-68tbg\" (UID: \"48e4bcde-dcb9-4e7e-8b15-e8682eb564d1\") " pod="openshift-image-registry/node-ca-68tbg"
Jul 28 12:53:24.729488 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.728217    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f02e9322-ad0d-4b41-b3e4-a4c6a71f566d-host-var-lib-cni-multus\") pod \"multus-jd2p2\" (UID: \"f02e9322-ad0d-4b41-b3e4-a4c6a71f566d\") " pod="openshift-multus/multus-jd2p2"
Jul 28 12:53:24.729488 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.728241    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-ovnkube-script-lib\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.729488 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.728261    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a71336e9-5f7a-4c21-8a5f-6402450ed0fa-etc-sysconfig\") pod \"tuned-tgjnv\" (UID: \"a71336e9-5f7a-4c21-8a5f-6402450ed0fa\") " pod="openshift-cluster-node-tuning-operator/tuned-tgjnv"
Jul 28 12:53:24.729488 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.728266    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f02e9322-ad0d-4b41-b3e4-a4c6a71f566d-host-var-lib-cni-multus\") pod \"multus-jd2p2\" (UID: \"f02e9322-ad0d-4b41-b3e4-a4c6a71f566d\") " pod="openshift-multus/multus-jd2p2"
Jul 28 12:53:24.729488 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.728302    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a71336e9-5f7a-4c21-8a5f-6402450ed0fa-etc-kubernetes\") pod \"tuned-tgjnv\" (UID: \"a71336e9-5f7a-4c21-8a5f-6402450ed0fa\") " pod="openshift-cluster-node-tuning-operator/tuned-tgjnv"
Jul 28 12:53:24.729488 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.728321    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1118726b-19e7-4111-9934-60e937e54642-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ckrz7\" (UID: \"1118726b-19e7-4111-9934-60e937e54642\") " pod="openshift-multus/multus-additional-cni-plugins-ckrz7"
Jul 28 12:53:24.729488 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.728353    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a71336e9-5f7a-4c21-8a5f-6402450ed0fa-etc-tuned\") pod \"tuned-tgjnv\" (UID: \"a71336e9-5f7a-4c21-8a5f-6402450ed0fa\") " pod="openshift-cluster-node-tuning-operator/tuned-tgjnv"
Jul 28 12:53:24.729488 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.728389    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76zk8\" (UniqueName: \"kubernetes.io/projected/48e4bcde-dcb9-4e7e-8b15-e8682eb564d1-kube-api-access-76zk8\") pod \"node-ca-68tbg\" (UID: \"48e4bcde-dcb9-4e7e-8b15-e8682eb564d1\") " pod="openshift-image-registry/node-ca-68tbg"
Jul 28 12:53:24.729488 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.728406    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5731c918-3db8-447b-bda3-7ac8427ab9df-iptables-alerter-script\") pod \"iptables-alerter-248g2\" (UID: \"5731c918-3db8-447b-bda3-7ac8427ab9df\") " pod="openshift-network-operator/iptables-alerter-248g2"
Jul 28 12:53:24.729488 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.728440    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f02e9322-ad0d-4b41-b3e4-a4c6a71f566d-multus-daemon-config\") pod \"multus-jd2p2\" (UID: \"f02e9322-ad0d-4b41-b3e4-a4c6a71f566d\") " pod="openshift-multus/multus-jd2p2"
Jul 28 12:53:24.729488 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.728475    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-host-slash\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.729488 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.728502    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a71336e9-5f7a-4c21-8a5f-6402450ed0fa-etc-sysctl-conf\") pod \"tuned-tgjnv\" (UID: \"a71336e9-5f7a-4c21-8a5f-6402450ed0fa\") " pod="openshift-cluster-node-tuning-operator/tuned-tgjnv"
Jul 28 12:53:24.729488 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.728527    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a71336e9-5f7a-4c21-8a5f-6402450ed0fa-sys\") pod \"tuned-tgjnv\" (UID: \"a71336e9-5f7a-4c21-8a5f-6402450ed0fa\") " pod="openshift-cluster-node-tuning-operator/tuned-tgjnv"
Jul 28 12:53:24.729488 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.728552    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-var-lib-openvswitch\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.729948 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.728578    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1118726b-19e7-4111-9934-60e937e54642-cni-binary-copy\") pod \"multus-additional-cni-plugins-ckrz7\" (UID: \"1118726b-19e7-4111-9934-60e937e54642\") " pod="openshift-multus/multus-additional-cni-plugins-ckrz7"
Jul 28 12:53:24.729948 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.728614    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5731c918-3db8-447b-bda3-7ac8427ab9df-host-slash\") pod \"iptables-alerter-248g2\" (UID: \"5731c918-3db8-447b-bda3-7ac8427ab9df\") " pod="openshift-network-operator/iptables-alerter-248g2"
Jul 28 12:53:24.729948 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.728641    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4afbaf34-0b91-4704-9e53-39f24049c3c7-konnectivity-ca\") pod \"konnectivity-agent-x88hp\" (UID: \"4afbaf34-0b91-4704-9e53-39f24049c3c7\") " pod="kube-system/konnectivity-agent-x88hp"
Jul 28 12:53:24.729948 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.728676    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-host-kubelet\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.729948 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.728681    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5731c918-3db8-447b-bda3-7ac8427ab9df-host-slash\") pod \"iptables-alerter-248g2\" (UID: \"5731c918-3db8-447b-bda3-7ac8427ab9df\") " pod="openshift-network-operator/iptables-alerter-248g2"
Jul 28 12:53:24.729948 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.728717    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-host-run-netns\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.729948 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.728740    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-log-socket\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.729948 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.728949    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5731c918-3db8-447b-bda3-7ac8427ab9df-iptables-alerter-script\") pod \"iptables-alerter-248g2\" (UID: \"5731c918-3db8-447b-bda3-7ac8427ab9df\") " pod="openshift-network-operator/iptables-alerter-248g2"
Jul 28 12:53:24.729948 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.729034    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f02e9322-ad0d-4b41-b3e4-a4c6a71f566d-multus-daemon-config\") pod \"multus-jd2p2\" (UID: \"f02e9322-ad0d-4b41-b3e4-a4c6a71f566d\") " pod="openshift-multus/multus-jd2p2"
Jul 28 12:53:24.729948 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.729193    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4afbaf34-0b91-4704-9e53-39f24049c3c7-konnectivity-ca\") pod \"konnectivity-agent-x88hp\" (UID: \"4afbaf34-0b91-4704-9e53-39f24049c3c7\") " pod="kube-system/konnectivity-agent-x88hp"
Jul 28 12:53:24.729948 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.729452    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1118726b-19e7-4111-9934-60e937e54642-cni-binary-copy\") pod \"multus-additional-cni-plugins-ckrz7\" (UID: \"1118726b-19e7-4111-9934-60e937e54642\") " pod="openshift-multus/multus-additional-cni-plugins-ckrz7"
Jul 28 12:53:24.729948 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.729563    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4afbaf34-0b91-4704-9e53-39f24049c3c7-agent-certs\") pod \"konnectivity-agent-x88hp\" (UID: \"4afbaf34-0b91-4704-9e53-39f24049c3c7\") " pod="kube-system/konnectivity-agent-x88hp"
Jul 28 12:53:24.735512 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:24.735489    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:53:24.735621 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:24.735517    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:53:24.735621 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:24.735543    2554 projected.go:196] Error preparing data for projected volume kube-api-access-7vlhn for pod openshift-network-diagnostics/network-check-target-4l2q4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:53:24.735749 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:24.735627    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/78f39c2f-8193-46af-b59d-8dc730d2e61a-kube-api-access-7vlhn podName:78f39c2f-8193-46af-b59d-8dc730d2e61a nodeName:}" failed. No retries permitted until 2026-07-28 12:53:25.235586727 +0000 UTC m=+2.021461819 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7vlhn" (UniqueName: "kubernetes.io/projected/78f39c2f-8193-46af-b59d-8dc730d2e61a-kube-api-access-7vlhn") pod "network-check-target-4l2q4" (UID: "78f39c2f-8193-46af-b59d-8dc730d2e61a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:53:24.738414 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.738394    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vzsc\" (UniqueName: \"kubernetes.io/projected/5731c918-3db8-447b-bda3-7ac8427ab9df-kube-api-access-5vzsc\") pod \"iptables-alerter-248g2\" (UID: \"5731c918-3db8-447b-bda3-7ac8427ab9df\") " pod="openshift-network-operator/iptables-alerter-248g2"
Jul 28 12:53:24.738795 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.738775    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkn8s\" (UniqueName: \"kubernetes.io/projected/1118726b-19e7-4111-9934-60e937e54642-kube-api-access-pkn8s\") pod \"multus-additional-cni-plugins-ckrz7\" (UID: \"1118726b-19e7-4111-9934-60e937e54642\") " pod="openshift-multus/multus-additional-cni-plugins-ckrz7"
Jul 28 12:53:24.738892 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.738869    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz4tc\" (UniqueName: \"kubernetes.io/projected/d336b5ed-db42-40f6-9e7a-a1882cb28484-kube-api-access-mz4tc\") pod \"node-resolver-sn9v6\" (UID: \"d336b5ed-db42-40f6-9e7a-a1882cb28484\") " pod="openshift-dns/node-resolver-sn9v6"
Jul 28 12:53:24.738938 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.738881    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn4tv\" (UniqueName: \"kubernetes.io/projected/f02e9322-ad0d-4b41-b3e4-a4c6a71f566d-kube-api-access-jn4tv\") pod \"multus-jd2p2\" (UID: \"f02e9322-ad0d-4b41-b3e4-a4c6a71f566d\") " pod="openshift-multus/multus-jd2p2"
Jul 28 12:53:24.738938 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.738881    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzjxv\" (UniqueName: \"kubernetes.io/projected/1c64d1e9-a9c9-48e8-bf71-cc650e871002-kube-api-access-pzjxv\") pod \"aws-ebs-csi-driver-node-jmd95\" (UID: \"1c64d1e9-a9c9-48e8-bf71-cc650e871002\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmd95"
Jul 28 12:53:24.829557 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.829525    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a71336e9-5f7a-4c21-8a5f-6402450ed0fa-etc-kubernetes\") pod \"tuned-tgjnv\" (UID: \"a71336e9-5f7a-4c21-8a5f-6402450ed0fa\") " pod="openshift-cluster-node-tuning-operator/tuned-tgjnv"
Jul 28 12:53:24.829557 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.829559    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a71336e9-5f7a-4c21-8a5f-6402450ed0fa-etc-tuned\") pod \"tuned-tgjnv\" (UID: \"a71336e9-5f7a-4c21-8a5f-6402450ed0fa\") " pod="openshift-cluster-node-tuning-operator/tuned-tgjnv"
Jul 28 12:53:24.829754 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.829581    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76zk8\" (UniqueName: \"kubernetes.io/projected/48e4bcde-dcb9-4e7e-8b15-e8682eb564d1-kube-api-access-76zk8\") pod \"node-ca-68tbg\" (UID: \"48e4bcde-dcb9-4e7e-8b15-e8682eb564d1\") " pod="openshift-image-registry/node-ca-68tbg"
Jul 28 12:53:24.829754 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.829627    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-host-slash\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.829754 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.829648    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a71336e9-5f7a-4c21-8a5f-6402450ed0fa-etc-sysctl-conf\") pod \"tuned-tgjnv\" (UID: \"a71336e9-5f7a-4c21-8a5f-6402450ed0fa\") " pod="openshift-cluster-node-tuning-operator/tuned-tgjnv"
Jul 28 12:53:24.829754 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.829649    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a71336e9-5f7a-4c21-8a5f-6402450ed0fa-etc-kubernetes\") pod \"tuned-tgjnv\" (UID: \"a71336e9-5f7a-4c21-8a5f-6402450ed0fa\") " pod="openshift-cluster-node-tuning-operator/tuned-tgjnv"
Jul 28 12:53:24.829754 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.829669    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a71336e9-5f7a-4c21-8a5f-6402450ed0fa-sys\") pod \"tuned-tgjnv\" (UID: \"a71336e9-5f7a-4c21-8a5f-6402450ed0fa\") " pod="openshift-cluster-node-tuning-operator/tuned-tgjnv"
Jul 28 12:53:24.829754 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.829706    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-var-lib-openvswitch\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.829754 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.829717    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a71336e9-5f7a-4c21-8a5f-6402450ed0fa-sys\") pod \"tuned-tgjnv\" (UID: \"a71336e9-5f7a-4c21-8a5f-6402450ed0fa\") " pod="openshift-cluster-node-tuning-operator/tuned-tgjnv"
Jul 28 12:53:24.829754 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.829748    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-host-kubelet\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.829754 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.829754    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-var-lib-openvswitch\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.830184 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.829771    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-host-run-netns\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.830184 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.829794    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-host-slash\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.830184 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.829794    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-log-socket\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.830184 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.829826    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-log-socket\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.830184 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.829840    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a71336e9-5f7a-4c21-8a5f-6402450ed0fa-etc-systemd\") pod \"tuned-tgjnv\" (UID: \"a71336e9-5f7a-4c21-8a5f-6402450ed0fa\") " pod="openshift-cluster-node-tuning-operator/tuned-tgjnv"
Jul 28 12:53:24.830184 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.829865    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-etc-openvswitch\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.830184 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.829871    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-host-run-netns\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.830184 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.829890    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a71336e9-5f7a-4c21-8a5f-6402450ed0fa-host\") pod \"tuned-tgjnv\" (UID: \"a71336e9-5f7a-4c21-8a5f-6402450ed0fa\") " pod="openshift-cluster-node-tuning-operator/tuned-tgjnv"
Jul 28 12:53:24.830184 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.829889    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-host-kubelet\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.830184 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.829913    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9f6f2eb9-22ea-4119-b474-75c3d6ec08eb-original-pull-secret\") pod \"global-pull-secret-syncer-jcccx\" (UID: \"9f6f2eb9-22ea-4119-b474-75c3d6ec08eb\") " pod="kube-system/global-pull-secret-syncer-jcccx"
Jul 28 12:53:24.830184 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.829916    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a71336e9-5f7a-4c21-8a5f-6402450ed0fa-etc-systemd\") pod \"tuned-tgjnv\" (UID: \"a71336e9-5f7a-4c21-8a5f-6402450ed0fa\") " pod="openshift-cluster-node-tuning-operator/tuned-tgjnv"
Jul 28 12:53:24.830184 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.829915    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a71336e9-5f7a-4c21-8a5f-6402450ed0fa-etc-sysctl-conf\") pod \"tuned-tgjnv\" (UID: \"a71336e9-5f7a-4c21-8a5f-6402450ed0fa\") " pod="openshift-cluster-node-tuning-operator/tuned-tgjnv"
Jul 28 12:53:24.830184 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.829938    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a71336e9-5f7a-4c21-8a5f-6402450ed0fa-var-lib-kubelet\") pod \"tuned-tgjnv\" (UID: \"a71336e9-5f7a-4c21-8a5f-6402450ed0fa\") " pod="openshift-cluster-node-tuning-operator/tuned-tgjnv"
Jul 28 12:53:24.830184 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.829954    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-etc-openvswitch\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.830184 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.829965    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-host-run-ovn-kubernetes\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.830184 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.829971    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a71336e9-5f7a-4c21-8a5f-6402450ed0fa-host\") pod \"tuned-tgjnv\" (UID: \"a71336e9-5f7a-4c21-8a5f-6402450ed0fa\") " pod="openshift-cluster-node-tuning-operator/tuned-tgjnv"
Jul 28 12:53:24.830184 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:24.830019    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:53:24.830184 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.830026    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a71336e9-5f7a-4c21-8a5f-6402450ed0fa-var-lib-kubelet\") pod \"tuned-tgjnv\" (UID: \"a71336e9-5f7a-4c21-8a5f-6402450ed0fa\") " pod="openshift-cluster-node-tuning-operator/tuned-tgjnv"
Jul 28 12:53:24.831032 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.830034    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-env-overrides\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.831032 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.830076    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-host-run-ovn-kubernetes\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.831032 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:24.830091    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f6f2eb9-22ea-4119-b474-75c3d6ec08eb-original-pull-secret podName:9f6f2eb9-22ea-4119-b474-75c3d6ec08eb nodeName:}" failed. No retries permitted until 2026-07-28 12:53:25.33007043 +0000 UTC m=+2.115945533 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9f6f2eb9-22ea-4119-b474-75c3d6ec08eb-original-pull-secret") pod "global-pull-secret-syncer-jcccx" (UID: "9f6f2eb9-22ea-4119-b474-75c3d6ec08eb") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:53:24.831032 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.830106    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-systemd-units\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.831032 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.830138    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-systemd-units\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.831032 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.830156    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/48e4bcde-dcb9-4e7e-8b15-e8682eb564d1-host\") pod \"node-ca-68tbg\" (UID: \"48e4bcde-dcb9-4e7e-8b15-e8682eb564d1\") " pod="openshift-image-registry/node-ca-68tbg"
Jul 28 12:53:24.831032 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.830192    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj77c\" (UniqueName: \"kubernetes.io/projected/d04af0e5-3e6d-4027-8509-0af92acc4035-kube-api-access-rj77c\") pod \"network-metrics-daemon-vvbch\" (UID: \"d04af0e5-3e6d-4027-8509-0af92acc4035\") " pod="openshift-multus/network-metrics-daemon-vvbch"
Jul 28 12:53:24.831032 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.830220    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-node-log\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.831032 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.830237    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/48e4bcde-dcb9-4e7e-8b15-e8682eb564d1-host\") pod \"node-ca-68tbg\" (UID: \"48e4bcde-dcb9-4e7e-8b15-e8682eb564d1\") " pod="openshift-image-registry/node-ca-68tbg"
Jul 28 12:53:24.831032 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.830253    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-ovn-node-metrics-cert\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.831032 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.830339    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-node-log\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.831032 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.830382    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8txxr\" (UniqueName: \"kubernetes.io/projected/a71336e9-5f7a-4c21-8a5f-6402450ed0fa-kube-api-access-8txxr\") pod \"tuned-tgjnv\" (UID: \"a71336e9-5f7a-4c21-8a5f-6402450ed0fa\") " pod="openshift-cluster-node-tuning-operator/tuned-tgjnv"
Jul 28 12:53:24.831032 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.830484    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-ovnkube-config\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.831032 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.830516    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9f6f2eb9-22ea-4119-b474-75c3d6ec08eb-kubelet-config\") pod \"global-pull-secret-syncer-jcccx\" (UID: \"9f6f2eb9-22ea-4119-b474-75c3d6ec08eb\") " pod="kube-system/global-pull-secret-syncer-jcccx"
Jul 28 12:53:24.831032 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.830544    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a71336e9-5f7a-4c21-8a5f-6402450ed0fa-etc-sysctl-d\") pod \"tuned-tgjnv\" (UID: \"a71336e9-5f7a-4c21-8a5f-6402450ed0fa\") " pod="openshift-cluster-node-tuning-operator/tuned-tgjnv"
Jul 28 12:53:24.831032 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.830566    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9f6f2eb9-22ea-4119-b474-75c3d6ec08eb-dbus\") pod \"global-pull-secret-syncer-jcccx\" (UID: \"9f6f2eb9-22ea-4119-b474-75c3d6ec08eb\") " pod="kube-system/global-pull-secret-syncer-jcccx"
Jul 28 12:53:24.831032 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.830592    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d04af0e5-3e6d-4027-8509-0af92acc4035-metrics-certs\") pod \"network-metrics-daemon-vvbch\" (UID: \"d04af0e5-3e6d-4027-8509-0af92acc4035\") " pod="openshift-multus/network-metrics-daemon-vvbch"
Jul 28 12:53:24.831881 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.830616    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-run-systemd\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.831881 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.830629    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-env-overrides\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.831881 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.830641    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-run-openvswitch\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.831881 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.830643    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9f6f2eb9-22ea-4119-b474-75c3d6ec08eb-kubelet-config\") pod \"global-pull-secret-syncer-jcccx\" (UID: \"9f6f2eb9-22ea-4119-b474-75c3d6ec08eb\") " pod="kube-system/global-pull-secret-syncer-jcccx"
Jul 28 12:53:24.831881 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.830688    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-host-cni-bin\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.831881 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.830690    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a71336e9-5f7a-4c21-8a5f-6402450ed0fa-etc-sysctl-d\") pod \"tuned-tgjnv\" (UID: \"a71336e9-5f7a-4c21-8a5f-6402450ed0fa\") " pod="openshift-cluster-node-tuning-operator/tuned-tgjnv"
Jul 28 12:53:24.831881 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.830712    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a71336e9-5f7a-4c21-8a5f-6402450ed0fa-etc-modprobe-d\") pod \"tuned-tgjnv\" (UID: \"a71336e9-5f7a-4c21-8a5f-6402450ed0fa\") " pod="openshift-cluster-node-tuning-operator/tuned-tgjnv"
Jul 28 12:53:24.831881 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.830734    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a71336e9-5f7a-4c21-8a5f-6402450ed0fa-run\") pod \"tuned-tgjnv\" (UID: \"a71336e9-5f7a-4c21-8a5f-6402450ed0fa\") " pod="openshift-cluster-node-tuning-operator/tuned-tgjnv"
Jul 28 12:53:24.831881 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.830739    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-run-openvswitch\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.831881 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.830770    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-run-ovn\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.831881 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.830787    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-host-cni-bin\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.831881 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.830798    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.831881 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.830826    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7b6k\" (UniqueName: \"kubernetes.io/projected/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-kube-api-access-w7b6k\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.831881 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.830854    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a71336e9-5f7a-4c21-8a5f-6402450ed0fa-lib-modules\") pod \"tuned-tgjnv\" (UID: \"a71336e9-5f7a-4c21-8a5f-6402450ed0fa\") " pod="openshift-cluster-node-tuning-operator/tuned-tgjnv"
Jul 28 12:53:24.831881 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.830882    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a71336e9-5f7a-4c21-8a5f-6402450ed0fa-etc-modprobe-d\") pod \"tuned-tgjnv\" (UID: \"a71336e9-5f7a-4c21-8a5f-6402450ed0fa\") " pod="openshift-cluster-node-tuning-operator/tuned-tgjnv"
Jul 28 12:53:24.831881 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.830884    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-run-systemd\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.831881 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.830921    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-host-cni-netd\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.832460 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.830934    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9f6f2eb9-22ea-4119-b474-75c3d6ec08eb-dbus\") pod \"global-pull-secret-syncer-jcccx\" (UID: \"9f6f2eb9-22ea-4119-b474-75c3d6ec08eb\") " pod="kube-system/global-pull-secret-syncer-jcccx"
Jul 28 12:53:24.832460 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.830943    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a71336e9-5f7a-4c21-8a5f-6402450ed0fa-tmp\") pod \"tuned-tgjnv\" (UID: \"a71336e9-5f7a-4c21-8a5f-6402450ed0fa\") " pod="openshift-cluster-node-tuning-operator/tuned-tgjnv"
Jul 28 12:53:24.832460 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.830946    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.832460 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.830968    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/48e4bcde-dcb9-4e7e-8b15-e8682eb564d1-serviceca\") pod \"node-ca-68tbg\" (UID: \"48e4bcde-dcb9-4e7e-8b15-e8682eb564d1\") " pod="openshift-image-registry/node-ca-68tbg"
Jul 28 12:53:24.832460 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.830980    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-run-ovn\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.832460 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.830993    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-ovnkube-script-lib\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.832460 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.831018    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a71336e9-5f7a-4c21-8a5f-6402450ed0fa-etc-sysconfig\") pod \"tuned-tgjnv\" (UID: \"a71336e9-5f7a-4c21-8a5f-6402450ed0fa\") " pod="openshift-cluster-node-tuning-operator/tuned-tgjnv"
Jul 28 12:53:24.832460 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.831020    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-host-cni-netd\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.832460 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.831084    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-ovnkube-config\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.832460 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:24.831170    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:53:24.832460 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:24.831234    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d04af0e5-3e6d-4027-8509-0af92acc4035-metrics-certs podName:d04af0e5-3e6d-4027-8509-0af92acc4035 nodeName:}" failed. No retries permitted until 2026-07-28 12:53:25.331218129 +0000 UTC m=+2.117093221 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d04af0e5-3e6d-4027-8509-0af92acc4035-metrics-certs") pod "network-metrics-daemon-vvbch" (UID: "d04af0e5-3e6d-4027-8509-0af92acc4035") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:53:24.832460 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.831231    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a71336e9-5f7a-4c21-8a5f-6402450ed0fa-lib-modules\") pod \"tuned-tgjnv\" (UID: \"a71336e9-5f7a-4c21-8a5f-6402450ed0fa\") " pod="openshift-cluster-node-tuning-operator/tuned-tgjnv"
Jul 28 12:53:24.832460 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.831292    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a71336e9-5f7a-4c21-8a5f-6402450ed0fa-run\") pod \"tuned-tgjnv\" (UID: \"a71336e9-5f7a-4c21-8a5f-6402450ed0fa\") " pod="openshift-cluster-node-tuning-operator/tuned-tgjnv"
Jul 28 12:53:24.832460 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.831338    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a71336e9-5f7a-4c21-8a5f-6402450ed0fa-etc-sysconfig\") pod \"tuned-tgjnv\" (UID: \"a71336e9-5f7a-4c21-8a5f-6402450ed0fa\") " pod="openshift-cluster-node-tuning-operator/tuned-tgjnv"
Jul 28 12:53:24.832460 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.831487    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-ovnkube-script-lib\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.832460 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.831520    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/48e4bcde-dcb9-4e7e-8b15-e8682eb564d1-serviceca\") pod \"node-ca-68tbg\" (UID: \"48e4bcde-dcb9-4e7e-8b15-e8682eb564d1\") " pod="openshift-image-registry/node-ca-68tbg"
Jul 28 12:53:24.832460 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.832253    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a71336e9-5f7a-4c21-8a5f-6402450ed0fa-etc-tuned\") pod \"tuned-tgjnv\" (UID: \"a71336e9-5f7a-4c21-8a5f-6402450ed0fa\") " pod="openshift-cluster-node-tuning-operator/tuned-tgjnv"
Jul 28 12:53:24.833583 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.833567    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a71336e9-5f7a-4c21-8a5f-6402450ed0fa-tmp\") pod \"tuned-tgjnv\" (UID: \"a71336e9-5f7a-4c21-8a5f-6402450ed0fa\") " pod="openshift-cluster-node-tuning-operator/tuned-tgjnv"
Jul 28 12:53:24.833627 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.833618    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-ovn-node-metrics-cert\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.838787 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.838764    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76zk8\" (UniqueName: \"kubernetes.io/projected/48e4bcde-dcb9-4e7e-8b15-e8682eb564d1-kube-api-access-76zk8\") pod \"node-ca-68tbg\" (UID: \"48e4bcde-dcb9-4e7e-8b15-e8682eb564d1\") " pod="openshift-image-registry/node-ca-68tbg"
Jul 28 12:53:24.838924 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:24.838903    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0d874dd23a22efefe3523ff609d1a20.slice/crio-df26ea658552ddebda55c4ea95f1caa78769434bf770ce2e6c6ee5e696fcb570 WatchSource:0}: Error finding container df26ea658552ddebda55c4ea95f1caa78769434bf770ce2e6c6ee5e696fcb570: Status 404 returned error can't find the container with id df26ea658552ddebda55c4ea95f1caa78769434bf770ce2e6c6ee5e696fcb570
Jul 28 12:53:24.839618 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:24.839600    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod571f960b5c36dc73834816c1792a750a.slice/crio-300737d2071726dd0b15b3fa6fc0f2e23e145baa93bee3bc64c338eb4efb1447 WatchSource:0}: Error finding container 300737d2071726dd0b15b3fa6fc0f2e23e145baa93bee3bc64c338eb4efb1447: Status 404 returned error can't find the container with id 300737d2071726dd0b15b3fa6fc0f2e23e145baa93bee3bc64c338eb4efb1447
Jul 28 12:53:24.842095 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.842071    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8txxr\" (UniqueName: \"kubernetes.io/projected/a71336e9-5f7a-4c21-8a5f-6402450ed0fa-kube-api-access-8txxr\") pod \"tuned-tgjnv\" (UID: \"a71336e9-5f7a-4c21-8a5f-6402450ed0fa\") " pod="openshift-cluster-node-tuning-operator/tuned-tgjnv"
Jul 28 12:53:24.842456 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.842441    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj77c\" (UniqueName: \"kubernetes.io/projected/d04af0e5-3e6d-4027-8509-0af92acc4035-kube-api-access-rj77c\") pod \"network-metrics-daemon-vvbch\" (UID: \"d04af0e5-3e6d-4027-8509-0af92acc4035\") " pod="openshift-multus/network-metrics-daemon-vvbch"
Jul 28 12:53:24.842976 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.842951    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7b6k\" (UniqueName: \"kubernetes.io/projected/c549168d-d9be-4f07-8fbe-bf4d7c49d8c8-kube-api-access-w7b6k\") pod \"ovnkube-node-st9kv\" (UID: \"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:24.844090 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.844076    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 12:53:24.946746 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.946675    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-x88hp"
Jul 28 12:53:24.955607 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:24.955584    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4afbaf34_0b91_4704_9e53_39f24049c3c7.slice/crio-df47cc85951a8ceb1cf1c4a525c6987a8bff19c7e80da6c4323d2b2b3c8b35f3 WatchSource:0}: Error finding container df47cc85951a8ceb1cf1c4a525c6987a8bff19c7e80da6c4323d2b2b3c8b35f3: Status 404 returned error can't find the container with id df47cc85951a8ceb1cf1c4a525c6987a8bff19c7e80da6c4323d2b2b3c8b35f3
Jul 28 12:53:24.966087 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.966066    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmd95"
Jul 28 12:53:24.974110 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:24.974087    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c64d1e9_a9c9_48e8_bf71_cc650e871002.slice/crio-5e72e631642cc4f6e3bc47ec7cef706607d4e3df8c2b2982cd0e4703b4293be3 WatchSource:0}: Error finding container 5e72e631642cc4f6e3bc47ec7cef706607d4e3df8c2b2982cd0e4703b4293be3: Status 404 returned error can't find the container with id 5e72e631642cc4f6e3bc47ec7cef706607d4e3df8c2b2982cd0e4703b4293be3
Jul 28 12:53:24.977387 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.977368    2554 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 12:53:24.988768 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:24.988749    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sn9v6"
Jul 28 12:53:24.997004 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:24.996986    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd336b5ed_db42_40f6_9e7a_a1882cb28484.slice/crio-d5ded7a3ecd313414db8e2cd6052530b1bda8ee12ba2f247c6b9ddafd1c0d8f3 WatchSource:0}: Error finding container d5ded7a3ecd313414db8e2cd6052530b1bda8ee12ba2f247c6b9ddafd1c0d8f3: Status 404 returned error can't find the container with id d5ded7a3ecd313414db8e2cd6052530b1bda8ee12ba2f247c6b9ddafd1c0d8f3
Jul 28 12:53:25.001827 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:25.001813    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ckrz7"
Jul 28 12:53:25.008963 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:25.008931    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1118726b_19e7_4111_9934_60e937e54642.slice/crio-7cf1cdb9dd7c27bdd80d548920b7ce1e9c90a703abae03796800577d585994f7 WatchSource:0}: Error finding container 7cf1cdb9dd7c27bdd80d548920b7ce1e9c90a703abae03796800577d585994f7: Status 404 returned error can't find the container with id 7cf1cdb9dd7c27bdd80d548920b7ce1e9c90a703abae03796800577d585994f7
Jul 28 12:53:25.022116 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:25.022093    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jd2p2"
Jul 28 12:53:25.029475 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:25.029455    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf02e9322_ad0d_4b41_b3e4_a4c6a71f566d.slice/crio-5a22c89ca70f9a486f91514ad3e22e7349aa8493289715e0673b6da95a68a5b0 WatchSource:0}: Error finding container 5a22c89ca70f9a486f91514ad3e22e7349aa8493289715e0673b6da95a68a5b0: Status 404 returned error can't find the container with id 5a22c89ca70f9a486f91514ad3e22e7349aa8493289715e0673b6da95a68a5b0
Jul 28 12:53:25.034480 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:25.034464    2554 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 12:53:25.037342 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:25.037326    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-248g2"
Jul 28 12:53:25.044828 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:25.044807    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5731c918_3db8_447b_bda3_7ac8427ab9df.slice/crio-450226100bb85e1bdfbe44d62e6eaa2d4f7610615b6a836587064e87ee401e5d WatchSource:0}: Error finding container 450226100bb85e1bdfbe44d62e6eaa2d4f7610615b6a836587064e87ee401e5d: Status 404 returned error can't find the container with id 450226100bb85e1bdfbe44d62e6eaa2d4f7610615b6a836587064e87ee401e5d
Jul 28 12:53:25.051426 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:25.051409    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tgjnv"
Jul 28 12:53:25.058819 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:25.058803    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda71336e9_5f7a_4c21_8a5f_6402450ed0fa.slice/crio-0a74e83f1352d0263da7d2c1e5dfa0c0c0d9b0896cf53c0654d6fa43726df21a WatchSource:0}: Error finding container 0a74e83f1352d0263da7d2c1e5dfa0c0c0d9b0896cf53c0654d6fa43726df21a: Status 404 returned error can't find the container with id 0a74e83f1352d0263da7d2c1e5dfa0c0c0d9b0896cf53c0654d6fa43726df21a
Jul 28 12:53:25.066708 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:25.066692    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-68tbg"
Jul 28 12:53:25.071177 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:25.071160    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:25.074963 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:25.074941    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48e4bcde_dcb9_4e7e_8b15_e8682eb564d1.slice/crio-c306fe4c654390b6c972a1789719d02742b8a0bc7cd68c82f7ed96fce99cf0ac WatchSource:0}: Error finding container c306fe4c654390b6c972a1789719d02742b8a0bc7cd68c82f7ed96fce99cf0ac: Status 404 returned error can't find the container with id c306fe4c654390b6c972a1789719d02742b8a0bc7cd68c82f7ed96fce99cf0ac
Jul 28 12:53:25.078913 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:53:25.078896    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc549168d_d9be_4f07_8fbe_bf4d7c49d8c8.slice/crio-b0ef263c000f1c7692db06174f4947950d82637ebe6d267925dd7b91dc343193 WatchSource:0}: Error finding container b0ef263c000f1c7692db06174f4947950d82637ebe6d267925dd7b91dc343193: Status 404 returned error can't find the container with id b0ef263c000f1c7692db06174f4947950d82637ebe6d267925dd7b91dc343193
Jul 28 12:53:25.335467 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:25.335440    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9f6f2eb9-22ea-4119-b474-75c3d6ec08eb-original-pull-secret\") pod \"global-pull-secret-syncer-jcccx\" (UID: \"9f6f2eb9-22ea-4119-b474-75c3d6ec08eb\") " pod="kube-system/global-pull-secret-syncer-jcccx"
Jul 28 12:53:25.335606 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:25.335479    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vlhn\" (UniqueName: \"kubernetes.io/projected/78f39c2f-8193-46af-b59d-8dc730d2e61a-kube-api-access-7vlhn\") pod \"network-check-target-4l2q4\" (UID: \"78f39c2f-8193-46af-b59d-8dc730d2e61a\") " pod="openshift-network-diagnostics/network-check-target-4l2q4"
Jul 28 12:53:25.335606 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:25.335511    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d04af0e5-3e6d-4027-8509-0af92acc4035-metrics-certs\") pod \"network-metrics-daemon-vvbch\" (UID: \"d04af0e5-3e6d-4027-8509-0af92acc4035\") " pod="openshift-multus/network-metrics-daemon-vvbch"
Jul 28 12:53:25.335606 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:25.335604    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:53:25.335743 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:25.335606    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:53:25.335743 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:25.335621    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:53:25.335743 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:25.335638    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:53:25.335743 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:25.335647    2554 projected.go:196] Error preparing data for projected volume kube-api-access-7vlhn for pod openshift-network-diagnostics/network-check-target-4l2q4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:53:25.335743 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:25.335654    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d04af0e5-3e6d-4027-8509-0af92acc4035-metrics-certs podName:d04af0e5-3e6d-4027-8509-0af92acc4035 nodeName:}" failed. No retries permitted until 2026-07-28 12:53:26.335641132 +0000 UTC m=+3.121516223 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d04af0e5-3e6d-4027-8509-0af92acc4035-metrics-certs") pod "network-metrics-daemon-vvbch" (UID: "d04af0e5-3e6d-4027-8509-0af92acc4035") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:53:25.335743 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:25.335670    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f6f2eb9-22ea-4119-b474-75c3d6ec08eb-original-pull-secret podName:9f6f2eb9-22ea-4119-b474-75c3d6ec08eb nodeName:}" failed. No retries permitted until 2026-07-28 12:53:26.335660752 +0000 UTC m=+3.121535866 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9f6f2eb9-22ea-4119-b474-75c3d6ec08eb-original-pull-secret") pod "global-pull-secret-syncer-jcccx" (UID: "9f6f2eb9-22ea-4119-b474-75c3d6ec08eb") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:53:25.335743 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:25.335691    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/78f39c2f-8193-46af-b59d-8dc730d2e61a-kube-api-access-7vlhn podName:78f39c2f-8193-46af-b59d-8dc730d2e61a nodeName:}" failed. No retries permitted until 2026-07-28 12:53:26.335678028 +0000 UTC m=+3.121553122 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7vlhn" (UniqueName: "kubernetes.io/projected/78f39c2f-8193-46af-b59d-8dc730d2e61a-kube-api-access-7vlhn") pod "network-check-target-4l2q4" (UID: "78f39c2f-8193-46af-b59d-8dc730d2e61a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:53:25.682078 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:25.681988    2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 12:48:25 +0000 UTC" deadline="2028-03-19 21:07:53.99109763 +0000 UTC"
Jul 28 12:53:25.682078 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:25.682031    2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14408h14m28.309072166s"
Jul 28 12:53:25.736001 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:25.735966    2554 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 12:53:25.799893 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:25.799861    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vvbch"
Jul 28 12:53:25.800093 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:25.800015    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vvbch" podUID="d04af0e5-3e6d-4027-8509-0af92acc4035"
Jul 28 12:53:25.810989 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:25.810925    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-st9kv" event={"ID":"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8","Type":"ContainerStarted","Data":"b0ef263c000f1c7692db06174f4947950d82637ebe6d267925dd7b91dc343193"}
Jul 28 12:53:25.839371 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:25.834725    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-68tbg" event={"ID":"48e4bcde-dcb9-4e7e-8b15-e8682eb564d1","Type":"ContainerStarted","Data":"c306fe4c654390b6c972a1789719d02742b8a0bc7cd68c82f7ed96fce99cf0ac"}
Jul 28 12:53:25.848804 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:25.848762    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tgjnv" event={"ID":"a71336e9-5f7a-4c21-8a5f-6402450ed0fa","Type":"ContainerStarted","Data":"0a74e83f1352d0263da7d2c1e5dfa0c0c0d9b0896cf53c0654d6fa43726df21a"}
Jul 28 12:53:25.853421 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:25.853371    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-248g2" event={"ID":"5731c918-3db8-447b-bda3-7ac8427ab9df","Type":"ContainerStarted","Data":"450226100bb85e1bdfbe44d62e6eaa2d4f7610615b6a836587064e87ee401e5d"}
Jul 28 12:53:25.860544 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:25.860464    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ckrz7" event={"ID":"1118726b-19e7-4111-9934-60e937e54642","Type":"ContainerStarted","Data":"7cf1cdb9dd7c27bdd80d548920b7ce1e9c90a703abae03796800577d585994f7"}
Jul 28 12:53:25.877072 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:25.877047    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-177.ec2.internal" event={"ID":"b0d874dd23a22efefe3523ff609d1a20","Type":"ContainerStarted","Data":"df26ea658552ddebda55c4ea95f1caa78769434bf770ce2e6c6ee5e696fcb570"}
Jul 28 12:53:25.885393 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:25.885366    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jd2p2" event={"ID":"f02e9322-ad0d-4b41-b3e4-a4c6a71f566d","Type":"ContainerStarted","Data":"5a22c89ca70f9a486f91514ad3e22e7349aa8493289715e0673b6da95a68a5b0"}
Jul 28 12:53:25.898052 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:25.898022    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sn9v6" event={"ID":"d336b5ed-db42-40f6-9e7a-a1882cb28484","Type":"ContainerStarted","Data":"d5ded7a3ecd313414db8e2cd6052530b1bda8ee12ba2f247c6b9ddafd1c0d8f3"}
Jul 28 12:53:25.901260 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:25.901234    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmd95" event={"ID":"1c64d1e9-a9c9-48e8-bf71-cc650e871002","Type":"ContainerStarted","Data":"5e72e631642cc4f6e3bc47ec7cef706607d4e3df8c2b2982cd0e4703b4293be3"}
Jul 28 12:53:25.908722 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:25.908049    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-x88hp" event={"ID":"4afbaf34-0b91-4704-9e53-39f24049c3c7","Type":"ContainerStarted","Data":"df47cc85951a8ceb1cf1c4a525c6987a8bff19c7e80da6c4323d2b2b3c8b35f3"}
Jul 28 12:53:25.910057 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:25.910003    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-177.ec2.internal" event={"ID":"571f960b5c36dc73834816c1792a750a","Type":"ContainerStarted","Data":"300737d2071726dd0b15b3fa6fc0f2e23e145baa93bee3bc64c338eb4efb1447"}
Jul 28 12:53:26.343942 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:26.343892    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vlhn\" (UniqueName: \"kubernetes.io/projected/78f39c2f-8193-46af-b59d-8dc730d2e61a-kube-api-access-7vlhn\") pod \"network-check-target-4l2q4\" (UID: \"78f39c2f-8193-46af-b59d-8dc730d2e61a\") " pod="openshift-network-diagnostics/network-check-target-4l2q4"
Jul 28 12:53:26.344146 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:26.343955    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d04af0e5-3e6d-4027-8509-0af92acc4035-metrics-certs\") pod \"network-metrics-daemon-vvbch\" (UID: \"d04af0e5-3e6d-4027-8509-0af92acc4035\") " pod="openshift-multus/network-metrics-daemon-vvbch"
Jul 28 12:53:26.344146 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:26.344009    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9f6f2eb9-22ea-4119-b474-75c3d6ec08eb-original-pull-secret\") pod \"global-pull-secret-syncer-jcccx\" (UID: \"9f6f2eb9-22ea-4119-b474-75c3d6ec08eb\") " pod="kube-system/global-pull-secret-syncer-jcccx"
Jul 28 12:53:26.344281 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:26.344167    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:53:26.344281 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:26.344235    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f6f2eb9-22ea-4119-b474-75c3d6ec08eb-original-pull-secret podName:9f6f2eb9-22ea-4119-b474-75c3d6ec08eb nodeName:}" failed. No retries permitted until 2026-07-28 12:53:28.34421377 +0000 UTC m=+5.130088864 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9f6f2eb9-22ea-4119-b474-75c3d6ec08eb-original-pull-secret") pod "global-pull-secret-syncer-jcccx" (UID: "9f6f2eb9-22ea-4119-b474-75c3d6ec08eb") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:53:26.347728 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:26.347690    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:53:26.347880 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:26.347734    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:53:26.347880 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:26.347749    2554 projected.go:196] Error preparing data for projected volume kube-api-access-7vlhn for pod openshift-network-diagnostics/network-check-target-4l2q4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:53:26.347880 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:26.347837    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/78f39c2f-8193-46af-b59d-8dc730d2e61a-kube-api-access-7vlhn podName:78f39c2f-8193-46af-b59d-8dc730d2e61a nodeName:}" failed. No retries permitted until 2026-07-28 12:53:28.347791352 +0000 UTC m=+5.133666448 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7vlhn" (UniqueName: "kubernetes.io/projected/78f39c2f-8193-46af-b59d-8dc730d2e61a-kube-api-access-7vlhn") pod "network-check-target-4l2q4" (UID: "78f39c2f-8193-46af-b59d-8dc730d2e61a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:53:26.352310 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:26.348184    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:53:26.352310 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:26.348267    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d04af0e5-3e6d-4027-8509-0af92acc4035-metrics-certs podName:d04af0e5-3e6d-4027-8509-0af92acc4035 nodeName:}" failed. No retries permitted until 2026-07-28 12:53:28.348239463 +0000 UTC m=+5.134114559 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d04af0e5-3e6d-4027-8509-0af92acc4035-metrics-certs") pod "network-metrics-daemon-vvbch" (UID: "d04af0e5-3e6d-4027-8509-0af92acc4035") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:53:26.682469 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:26.682343    2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 12:48:25 +0000 UTC" deadline="2028-02-05 09:26:37.509122212 +0000 UTC"
Jul 28 12:53:26.682469 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:26.682412    2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13364h33m10.826713589s"
Jul 28 12:53:26.796344 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:26.796308    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jcccx"
Jul 28 12:53:26.796546 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:26.796455    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jcccx" podUID="9f6f2eb9-22ea-4119-b474-75c3d6ec08eb"
Jul 28 12:53:26.796889 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:26.796866    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4l2q4"
Jul 28 12:53:26.796995 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:26.796974    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4l2q4" podUID="78f39c2f-8193-46af-b59d-8dc730d2e61a"
Jul 28 12:53:27.799304 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:27.799210    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vvbch"
Jul 28 12:53:27.799769 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:27.799371    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vvbch" podUID="d04af0e5-3e6d-4027-8509-0af92acc4035"
Jul 28 12:53:28.367675 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:28.367631    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vlhn\" (UniqueName: \"kubernetes.io/projected/78f39c2f-8193-46af-b59d-8dc730d2e61a-kube-api-access-7vlhn\") pod \"network-check-target-4l2q4\" (UID: \"78f39c2f-8193-46af-b59d-8dc730d2e61a\") " pod="openshift-network-diagnostics/network-check-target-4l2q4"
Jul 28 12:53:28.367878 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:28.367703    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d04af0e5-3e6d-4027-8509-0af92acc4035-metrics-certs\") pod \"network-metrics-daemon-vvbch\" (UID: \"d04af0e5-3e6d-4027-8509-0af92acc4035\") " pod="openshift-multus/network-metrics-daemon-vvbch"
Jul 28 12:53:28.367878 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:28.367762    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9f6f2eb9-22ea-4119-b474-75c3d6ec08eb-original-pull-secret\") pod \"global-pull-secret-syncer-jcccx\" (UID: \"9f6f2eb9-22ea-4119-b474-75c3d6ec08eb\") " pod="kube-system/global-pull-secret-syncer-jcccx"
Jul 28 12:53:28.368012 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:28.367927    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:53:28.368012 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:28.367993    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f6f2eb9-22ea-4119-b474-75c3d6ec08eb-original-pull-secret podName:9f6f2eb9-22ea-4119-b474-75c3d6ec08eb nodeName:}" failed. No retries permitted until 2026-07-28 12:53:32.367972191 +0000 UTC m=+9.153847307 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9f6f2eb9-22ea-4119-b474-75c3d6ec08eb-original-pull-secret") pod "global-pull-secret-syncer-jcccx" (UID: "9f6f2eb9-22ea-4119-b474-75c3d6ec08eb") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:53:28.368443 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:28.368424    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:53:28.368443 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:28.368447    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:53:28.368661 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:28.368458    2554 projected.go:196] Error preparing data for projected volume kube-api-access-7vlhn for pod openshift-network-diagnostics/network-check-target-4l2q4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:53:28.368661 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:28.368504    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/78f39c2f-8193-46af-b59d-8dc730d2e61a-kube-api-access-7vlhn podName:78f39c2f-8193-46af-b59d-8dc730d2e61a nodeName:}" failed. No retries permitted until 2026-07-28 12:53:32.368490311 +0000 UTC m=+9.154365402 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7vlhn" (UniqueName: "kubernetes.io/projected/78f39c2f-8193-46af-b59d-8dc730d2e61a-kube-api-access-7vlhn") pod "network-check-target-4l2q4" (UID: "78f39c2f-8193-46af-b59d-8dc730d2e61a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:53:28.368661 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:28.368560    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:53:28.368661 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:28.368592    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d04af0e5-3e6d-4027-8509-0af92acc4035-metrics-certs podName:d04af0e5-3e6d-4027-8509-0af92acc4035 nodeName:}" failed. No retries permitted until 2026-07-28 12:53:32.368581649 +0000 UTC m=+9.154456743 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d04af0e5-3e6d-4027-8509-0af92acc4035-metrics-certs") pod "network-metrics-daemon-vvbch" (UID: "d04af0e5-3e6d-4027-8509-0af92acc4035") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:53:28.796185 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:28.796107    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jcccx"
Jul 28 12:53:28.796351 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:28.796230    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jcccx" podUID="9f6f2eb9-22ea-4119-b474-75c3d6ec08eb"
Jul 28 12:53:28.796351 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:28.796325    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4l2q4"
Jul 28 12:53:28.796488 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:28.796440    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4l2q4" podUID="78f39c2f-8193-46af-b59d-8dc730d2e61a"
Jul 28 12:53:29.796333 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:29.796272    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vvbch"
Jul 28 12:53:29.796827 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:29.796441    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vvbch" podUID="d04af0e5-3e6d-4027-8509-0af92acc4035"
Jul 28 12:53:30.796915 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:30.796879    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jcccx"
Jul 28 12:53:30.797404 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:30.797017    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4l2q4"
Jul 28 12:53:30.797404 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:30.797024    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jcccx" podUID="9f6f2eb9-22ea-4119-b474-75c3d6ec08eb"
Jul 28 12:53:30.797404 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:30.797093    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4l2q4" podUID="78f39c2f-8193-46af-b59d-8dc730d2e61a"
Jul 28 12:53:31.799808 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:31.799774    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vvbch"
Jul 28 12:53:31.800319 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:31.799916    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vvbch" podUID="d04af0e5-3e6d-4027-8509-0af92acc4035"
Jul 28 12:53:32.407726 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:32.407631    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9f6f2eb9-22ea-4119-b474-75c3d6ec08eb-original-pull-secret\") pod \"global-pull-secret-syncer-jcccx\" (UID: \"9f6f2eb9-22ea-4119-b474-75c3d6ec08eb\") " pod="kube-system/global-pull-secret-syncer-jcccx"
Jul 28 12:53:32.407726 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:32.407698    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vlhn\" (UniqueName: \"kubernetes.io/projected/78f39c2f-8193-46af-b59d-8dc730d2e61a-kube-api-access-7vlhn\") pod \"network-check-target-4l2q4\" (UID: \"78f39c2f-8193-46af-b59d-8dc730d2e61a\") " pod="openshift-network-diagnostics/network-check-target-4l2q4"
Jul 28 12:53:32.407971 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:32.407836    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d04af0e5-3e6d-4027-8509-0af92acc4035-metrics-certs\") pod \"network-metrics-daemon-vvbch\" (UID: \"d04af0e5-3e6d-4027-8509-0af92acc4035\") " pod="openshift-multus/network-metrics-daemon-vvbch"
Jul 28 12:53:32.407971 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:32.407844    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:53:32.407971 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:32.407952    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:53:32.407971 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:32.407967    2554 projected.go:196] Error preparing data for projected volume kube-api-access-7vlhn for pod openshift-network-diagnostics/network-check-target-4l2q4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:53:32.408156 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:32.408025    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/78f39c2f-8193-46af-b59d-8dc730d2e61a-kube-api-access-7vlhn podName:78f39c2f-8193-46af-b59d-8dc730d2e61a nodeName:}" failed. No retries permitted until 2026-07-28 12:53:40.408011401 +0000 UTC m=+17.193886492 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7vlhn" (UniqueName: "kubernetes.io/projected/78f39c2f-8193-46af-b59d-8dc730d2e61a-kube-api-access-7vlhn") pod "network-check-target-4l2q4" (UID: "78f39c2f-8193-46af-b59d-8dc730d2e61a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:53:32.408156 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:32.407861    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:53:32.408156 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:32.408063    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f6f2eb9-22ea-4119-b474-75c3d6ec08eb-original-pull-secret podName:9f6f2eb9-22ea-4119-b474-75c3d6ec08eb nodeName:}" failed. No retries permitted until 2026-07-28 12:53:40.408053257 +0000 UTC m=+17.193928348 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9f6f2eb9-22ea-4119-b474-75c3d6ec08eb-original-pull-secret") pod "global-pull-secret-syncer-jcccx" (UID: "9f6f2eb9-22ea-4119-b474-75c3d6ec08eb") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:53:32.408156 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:32.407910    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:53:32.408156 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:32.408098    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d04af0e5-3e6d-4027-8509-0af92acc4035-metrics-certs podName:d04af0e5-3e6d-4027-8509-0af92acc4035 nodeName:}" failed. No retries permitted until 2026-07-28 12:53:40.408089836 +0000 UTC m=+17.193964926 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d04af0e5-3e6d-4027-8509-0af92acc4035-metrics-certs") pod "network-metrics-daemon-vvbch" (UID: "d04af0e5-3e6d-4027-8509-0af92acc4035") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:53:32.796518 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:32.796440    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4l2q4"
Jul 28 12:53:32.796676 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:32.796571    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4l2q4" podUID="78f39c2f-8193-46af-b59d-8dc730d2e61a"
Jul 28 12:53:32.796935 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:32.796914    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jcccx"
Jul 28 12:53:32.797059 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:32.796982    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jcccx" podUID="9f6f2eb9-22ea-4119-b474-75c3d6ec08eb"
Jul 28 12:53:33.797228 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:33.797199    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vvbch"
Jul 28 12:53:33.797686 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:33.797329    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vvbch" podUID="d04af0e5-3e6d-4027-8509-0af92acc4035"
Jul 28 12:53:34.797190 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:34.797156    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4l2q4"
Jul 28 12:53:34.797386 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:34.797312    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4l2q4" podUID="78f39c2f-8193-46af-b59d-8dc730d2e61a"
Jul 28 12:53:34.797386 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:34.797331    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jcccx"
Jul 28 12:53:34.797816 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:34.797415    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jcccx" podUID="9f6f2eb9-22ea-4119-b474-75c3d6ec08eb"
Jul 28 12:53:35.796913 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:35.796877    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vvbch"
Jul 28 12:53:35.797069 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:35.797018    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vvbch" podUID="d04af0e5-3e6d-4027-8509-0af92acc4035"
Jul 28 12:53:36.796870 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:36.796828    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jcccx"
Jul 28 12:53:36.796870 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:36.796851    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4l2q4"
Jul 28 12:53:36.797260 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:36.796943    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jcccx" podUID="9f6f2eb9-22ea-4119-b474-75c3d6ec08eb"
Jul 28 12:53:36.797260 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:36.797078    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4l2q4" podUID="78f39c2f-8193-46af-b59d-8dc730d2e61a"
Jul 28 12:53:37.796807 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:37.796735    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vvbch"
Jul 28 12:53:37.796941 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:37.796876    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vvbch" podUID="d04af0e5-3e6d-4027-8509-0af92acc4035"
Jul 28 12:53:38.796683 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:38.796651    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jcccx"
Jul 28 12:53:38.796851 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:38.796781    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jcccx" podUID="9f6f2eb9-22ea-4119-b474-75c3d6ec08eb"
Jul 28 12:53:38.796851 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:38.796833    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4l2q4"
Jul 28 12:53:38.796952 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:38.796934    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4l2q4" podUID="78f39c2f-8193-46af-b59d-8dc730d2e61a"
Jul 28 12:53:39.796497 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:39.796466    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vvbch"
Jul 28 12:53:39.796684 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:39.796605    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vvbch" podUID="d04af0e5-3e6d-4027-8509-0af92acc4035"
Jul 28 12:53:40.470036 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:40.469112    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9f6f2eb9-22ea-4119-b474-75c3d6ec08eb-original-pull-secret\") pod \"global-pull-secret-syncer-jcccx\" (UID: \"9f6f2eb9-22ea-4119-b474-75c3d6ec08eb\") " pod="kube-system/global-pull-secret-syncer-jcccx"
Jul 28 12:53:40.470036 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:40.469177    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vlhn\" (UniqueName: \"kubernetes.io/projected/78f39c2f-8193-46af-b59d-8dc730d2e61a-kube-api-access-7vlhn\") pod \"network-check-target-4l2q4\" (UID: \"78f39c2f-8193-46af-b59d-8dc730d2e61a\") " pod="openshift-network-diagnostics/network-check-target-4l2q4"
Jul 28 12:53:40.470036 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:40.469204    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d04af0e5-3e6d-4027-8509-0af92acc4035-metrics-certs\") pod \"network-metrics-daemon-vvbch\" (UID: \"d04af0e5-3e6d-4027-8509-0af92acc4035\") " pod="openshift-multus/network-metrics-daemon-vvbch"
Jul 28 12:53:40.470036 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:40.469351    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:53:40.470036 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:40.469416    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d04af0e5-3e6d-4027-8509-0af92acc4035-metrics-certs podName:d04af0e5-3e6d-4027-8509-0af92acc4035 nodeName:}" failed. No retries permitted until 2026-07-28 12:53:56.469397843 +0000 UTC m=+33.255272935 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d04af0e5-3e6d-4027-8509-0af92acc4035-metrics-certs") pod "network-metrics-daemon-vvbch" (UID: "d04af0e5-3e6d-4027-8509-0af92acc4035") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:53:40.470036 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:40.469814    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:53:40.470036 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:40.469862    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f6f2eb9-22ea-4119-b474-75c3d6ec08eb-original-pull-secret podName:9f6f2eb9-22ea-4119-b474-75c3d6ec08eb nodeName:}" failed. No retries permitted until 2026-07-28 12:53:56.469847877 +0000 UTC m=+33.255722968 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9f6f2eb9-22ea-4119-b474-75c3d6ec08eb-original-pull-secret") pod "global-pull-secret-syncer-jcccx" (UID: "9f6f2eb9-22ea-4119-b474-75c3d6ec08eb") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:53:40.470036 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:40.469945    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:53:40.470036 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:40.469960    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:53:40.470036 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:40.469969    2554 projected.go:196] Error preparing data for projected volume kube-api-access-7vlhn for pod openshift-network-diagnostics/network-check-target-4l2q4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:53:40.470036 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:40.470000    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/78f39c2f-8193-46af-b59d-8dc730d2e61a-kube-api-access-7vlhn podName:78f39c2f-8193-46af-b59d-8dc730d2e61a nodeName:}" failed. No retries permitted until 2026-07-28 12:53:56.469986575 +0000 UTC m=+33.255861664 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7vlhn" (UniqueName: "kubernetes.io/projected/78f39c2f-8193-46af-b59d-8dc730d2e61a-kube-api-access-7vlhn") pod "network-check-target-4l2q4" (UID: "78f39c2f-8193-46af-b59d-8dc730d2e61a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:53:40.796630 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:40.796552    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jcccx"
Jul 28 12:53:40.796824 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:40.796563    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4l2q4"
Jul 28 12:53:40.796824 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:40.796690    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jcccx" podUID="9f6f2eb9-22ea-4119-b474-75c3d6ec08eb"
Jul 28 12:53:40.796824 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:40.796751    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4l2q4" podUID="78f39c2f-8193-46af-b59d-8dc730d2e61a"
Jul 28 12:53:41.796815 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:41.796776    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vvbch"
Jul 28 12:53:41.797334 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:41.796921    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vvbch" podUID="d04af0e5-3e6d-4027-8509-0af92acc4035"
Jul 28 12:53:42.796836 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:42.796801    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4l2q4"
Jul 28 12:53:42.797243 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:42.796841    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jcccx"
Jul 28 12:53:42.797243 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:42.796967    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4l2q4" podUID="78f39c2f-8193-46af-b59d-8dc730d2e61a"
Jul 28 12:53:42.797243 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:42.797093    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jcccx" podUID="9f6f2eb9-22ea-4119-b474-75c3d6ec08eb"
Jul 28 12:53:43.797653 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:43.797619    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vvbch"
Jul 28 12:53:43.798107 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:43.797738    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vvbch" podUID="d04af0e5-3e6d-4027-8509-0af92acc4035"
Jul 28 12:53:44.796547 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:44.796511    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4l2q4"
Jul 28 12:53:44.796731 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:44.796626    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4l2q4" podUID="78f39c2f-8193-46af-b59d-8dc730d2e61a"
Jul 28 12:53:44.796731 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:44.796686    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jcccx"
Jul 28 12:53:44.796847 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:44.796775    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jcccx" podUID="9f6f2eb9-22ea-4119-b474-75c3d6ec08eb"
Jul 28 12:53:45.796232 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:45.796199    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vvbch"
Jul 28 12:53:45.796697 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:45.796355    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vvbch" podUID="d04af0e5-3e6d-4027-8509-0af92acc4035"
Jul 28 12:53:46.796686 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:46.796507    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jcccx"
Jul 28 12:53:46.797546 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:46.796572    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4l2q4"
Jul 28 12:53:46.797546 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:46.796759    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jcccx" podUID="9f6f2eb9-22ea-4119-b474-75c3d6ec08eb"
Jul 28 12:53:46.797546 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:46.796824    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4l2q4" podUID="78f39c2f-8193-46af-b59d-8dc730d2e61a"
Jul 28 12:53:46.964766 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:46.964690    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmd95" event={"ID":"1c64d1e9-a9c9-48e8-bf71-cc650e871002","Type":"ContainerStarted","Data":"23800b388c795718d93c63d4eb61dd8f42e08fa52ea38c306ed97e2a563c45ce"}
Jul 28 12:53:46.966111 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:46.966078    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-x88hp" event={"ID":"4afbaf34-0b91-4704-9e53-39f24049c3c7","Type":"ContainerStarted","Data":"a1ecddd47bc53f975d0a9ff2bfdd98db25b0c4e6c0fb15701400c186d6b003bf"}
Jul 28 12:53:46.967398 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:46.967373    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-177.ec2.internal" event={"ID":"571f960b5c36dc73834816c1792a750a","Type":"ContainerStarted","Data":"8e05b726c1f1afb5f743cfda71525e3227a0e075c72c57071f663f6df16e27c1"}
Jul 28 12:53:46.973043 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:46.973022    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-st9kv" event={"ID":"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8","Type":"ContainerStarted","Data":"454131716d939c929bf5c23c34ddd9a69fdc95bd6c245ae4c634a4f23c8b9578"}
Jul 28 12:53:46.973156 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:46.973088    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-st9kv" event={"ID":"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8","Type":"ContainerStarted","Data":"e8386a08a651ad4a7f083a372a7df6f980c9a4666772cc4604db92d87424488f"}
Jul 28 12:53:46.973156 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:46.973137    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-st9kv" event={"ID":"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8","Type":"ContainerStarted","Data":"8e5eb6f2a3d17890ea08ec0194a4778ca08a73c603f937882f2139be1656ad8a"}
Jul 28 12:53:46.973238 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:46.973188    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-st9kv" event={"ID":"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8","Type":"ContainerStarted","Data":"2a5e1d87de3f824c636d9671090e4717ad1a7b5fc4bd7b26cc7cad018f723b57"}
Jul 28 12:53:46.973238 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:46.973233    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-st9kv" event={"ID":"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8","Type":"ContainerStarted","Data":"cc1920f25472c98f1d1de2f0d21a065fae27e0e0b9d35a86b20fc77d8959f53d"}
Jul 28 12:53:46.973354 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:46.973265    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-st9kv" event={"ID":"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8","Type":"ContainerStarted","Data":"03eb95b72ce35302941928063263aaf4f335043186ac5fc6ff64125aa6c375ab"}
Jul 28 12:53:46.974468 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:46.974443    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-68tbg" event={"ID":"48e4bcde-dcb9-4e7e-8b15-e8682eb564d1","Type":"ContainerStarted","Data":"9dcd56ea22c42e8b130525c5aa95814762a020fda9231a4a5c605a330715c3f2"}
Jul 28 12:53:46.975902 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:46.975877    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tgjnv" event={"ID":"a71336e9-5f7a-4c21-8a5f-6402450ed0fa","Type":"ContainerStarted","Data":"29c0b5f3426517aa251a49f2bf0ef53a83efa80a162506545effa6aa3ef61673"}
Jul 28 12:53:46.977310 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:46.977290    2554 generic.go:356] "Generic (PLEG): container finished" podID="1118726b-19e7-4111-9934-60e937e54642" containerID="8b35784da34ea39f3d88933177f64eda8a50321c24d04a6457a3e47c57e416c4" exitCode=0
Jul 28 12:53:46.977383 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:46.977350    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ckrz7" event={"ID":"1118726b-19e7-4111-9934-60e937e54642","Type":"ContainerDied","Data":"8b35784da34ea39f3d88933177f64eda8a50321c24d04a6457a3e47c57e416c4"}
Jul 28 12:53:46.978723 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:46.978706    2554 generic.go:356] "Generic (PLEG): container finished" podID="b0d874dd23a22efefe3523ff609d1a20" containerID="f960870193ba88ee968b043359f0dff7bb9cb988e534c091c9aa7b3080dc2738" exitCode=0
Jul 28 12:53:46.978779 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:46.978730    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-177.ec2.internal" event={"ID":"b0d874dd23a22efefe3523ff609d1a20","Type":"ContainerDied","Data":"f960870193ba88ee968b043359f0dff7bb9cb988e534c091c9aa7b3080dc2738"}
Jul 28 12:53:46.980331 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:46.980307    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jd2p2" event={"ID":"f02e9322-ad0d-4b41-b3e4-a4c6a71f566d","Type":"ContainerStarted","Data":"0a571c8a6239bb4e54e91c8b8cad221bd2bac664e67b0156612edb9b4495ffe6"}
Jul 28 12:53:46.981652 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:46.981635    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sn9v6" event={"ID":"d336b5ed-db42-40f6-9e7a-a1882cb28484","Type":"ContainerStarted","Data":"2587f5a8ff0664e392f60bed3c843bc3a978210c07f25a28bb5984e0c196d577"}
Jul 28 12:53:46.984938 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:46.984902    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-x88hp" podStartSLOduration=5.920421956 podStartE2EDuration="22.984891791s" totalImagesPullingTime="17.064469835s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:53:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:53:46.984464613 +0000 UTC m=+23.770339734" watchObservedRunningTime="2026-07-28 12:53:46.984891791 +0000 UTC m=+23.770766901"
Jul 28 12:53:46.998579 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:46.998547    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-177.ec2.internal" podStartSLOduration=21.998536312 podStartE2EDuration="21.998536312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:53:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:53:46.998186747 +0000 UTC m=+23.784061858" watchObservedRunningTime="2026-07-28 12:53:46.998536312 +0000 UTC m=+23.784411423"
Jul 28 12:53:47.016466 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:47.016426    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-sn9v6" podStartSLOduration=2.039999297 podStartE2EDuration="23.016410921s" totalImagesPullingTime="20.976411624s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:53:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:53:47.016113067 +0000 UTC m=+23.801988189" watchObservedRunningTime="2026-07-28 12:53:47.016410921 +0000 UTC m=+23.802286032"
Jul 28 12:53:47.055701 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:47.055656    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-tgjnv" podStartSLOduration=2.131604032 podStartE2EDuration="23.055636011s" totalImagesPullingTime="20.924031979s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:53:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:53:47.055018059 +0000 UTC m=+23.840893182" watchObservedRunningTime="2026-07-28 12:53:47.055636011 +0000 UTC m=+23.841511102"
Jul 28 12:53:47.069443 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:47.069400    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-68tbg" podStartSLOduration=6.124124152 podStartE2EDuration="23.069387541s" totalImagesPullingTime="16.945263389s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:53:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:53:47.068977758 +0000 UTC m=+23.854852868" watchObservedRunningTime="2026-07-28 12:53:47.069387541 +0000 UTC m=+23.855262651"
Jul 28 12:53:47.104400 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:47.104358    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-jd2p2" podStartSLOduration=2.103943033 podStartE2EDuration="23.104343288s" totalImagesPullingTime="21.000400255s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:53:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:53:47.104142969 +0000 UTC m=+23.890018081" watchObservedRunningTime="2026-07-28 12:53:47.104343288 +0000 UTC m=+23.890218399"
Jul 28 12:53:47.797233 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:47.797065    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vvbch"
Jul 28 12:53:47.797649 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:47.797332    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vvbch" podUID="d04af0e5-3e6d-4027-8509-0af92acc4035"
Jul 28 12:53:47.985348 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:47.985314    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-248g2" event={"ID":"5731c918-3db8-447b-bda3-7ac8427ab9df","Type":"ContainerStarted","Data":"65ecb5a7e998563c7f66047f20a5f5eeeb4a5f45e8826d6f17418d01095ef502"}
Jul 28 12:53:47.988177 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:47.988149    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-177.ec2.internal" event={"ID":"b0d874dd23a22efefe3523ff609d1a20","Type":"ContainerStarted","Data":"c607fd49a66277699d3f9ec620ee0e675800e216febe1b98cbe1ac6d529edbbd"}
Jul 28 12:53:48.001837 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:48.001791    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-248g2" podStartSLOduration=3.075428912 podStartE2EDuration="24.001777793s" totalImagesPullingTime="20.926348881s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:53:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:53:48.001546356 +0000 UTC m=+24.787421467" watchObservedRunningTime="2026-07-28 12:53:48.001777793 +0000 UTC m=+24.787652906"
Jul 28 12:53:48.015715 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:48.015670    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-177.ec2.internal" podStartSLOduration=23.015653071 podStartE2EDuration="23.015653071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:53:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:53:48.015501494 +0000 UTC m=+24.801376606" watchObservedRunningTime="2026-07-28 12:53:48.015653071 +0000 UTC m=+24.801528182"
Jul 28 12:53:48.111171 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:48.111142    2554 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 12:53:48.702361 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:48.702212    2554 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T12:53:48.111168102Z","UUID":"be514cbe-b9da-4a87-8317-1836c324dcd8","Handler":null,"Name":"","Endpoint":""}
Jul 28 12:53:48.703983 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:48.703960    2554 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 12:53:48.704096 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:48.704005    2554 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 12:53:48.796593 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:48.796361    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jcccx"
Jul 28 12:53:48.796593 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:48.796378    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4l2q4"
Jul 28 12:53:48.796593 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:48.796487    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jcccx" podUID="9f6f2eb9-22ea-4119-b474-75c3d6ec08eb"
Jul 28 12:53:48.796593 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:48.796568    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4l2q4" podUID="78f39c2f-8193-46af-b59d-8dc730d2e61a"
Jul 28 12:53:48.992101 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:48.992068    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmd95" event={"ID":"1c64d1e9-a9c9-48e8-bf71-cc650e871002","Type":"ContainerStarted","Data":"59b52e0b28539043c1de4377a65988ebd7de7ba2415222b2d1695cdbcdb75572"}
Jul 28 12:53:49.607194 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:49.607111    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-x88hp"
Jul 28 12:53:49.607869 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:49.607853    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-x88hp"
Jul 28 12:53:49.796177 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:49.796147    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vvbch"
Jul 28 12:53:49.796434 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:49.796353    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vvbch" podUID="d04af0e5-3e6d-4027-8509-0af92acc4035"
Jul 28 12:53:49.995582 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:49.995505    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmd95" event={"ID":"1c64d1e9-a9c9-48e8-bf71-cc650e871002","Type":"ContainerStarted","Data":"261107b18751104e85dac038fc4ba6f19b6cea4aac6ddbddc1ca63a702d78d93"}
Jul 28 12:53:49.998651 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:49.998629    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-st9kv" event={"ID":"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8","Type":"ContainerStarted","Data":"8668b3805db6a11173e6a8928bc1e4e6403da27364b29c2d039f93d5674423aa"}
Jul 28 12:53:49.998752 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:49.998707    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-x88hp"
Jul 28 12:53:49.999254 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:49.999225    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-x88hp"
Jul 28 12:53:50.013804 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:50.013764    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jmd95" podStartSLOduration=10.190140499 podStartE2EDuration="26.013749256s" totalImagesPullingTime="15.823608757s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:53:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:53:50.013224511 +0000 UTC m=+26.799099624" watchObservedRunningTime="2026-07-28 12:53:50.013749256 +0000 UTC m=+26.799624366"
Jul 28 12:53:50.796735 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:50.796699    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4l2q4"
Jul 28 12:53:50.796735 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:50.796723    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jcccx"
Jul 28 12:53:50.796963 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:50.796820    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jcccx" podUID="9f6f2eb9-22ea-4119-b474-75c3d6ec08eb"
Jul 28 12:53:50.796963 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:50.796932    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4l2q4" podUID="78f39c2f-8193-46af-b59d-8dc730d2e61a"
Jul 28 12:53:51.796496 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:51.796341    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vvbch"
Jul 28 12:53:51.796916 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:51.796602    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vvbch" podUID="d04af0e5-3e6d-4027-8509-0af92acc4035"
Jul 28 12:53:52.004639 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:52.004610    2554 generic.go:356] "Generic (PLEG): container finished" podID="1118726b-19e7-4111-9934-60e937e54642" containerID="9a9d75ad3267b9bcab31f7f5855fc2abac68f234a3de53000e16437b973da423" exitCode=0
Jul 28 12:53:52.004746 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:52.004698    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ckrz7" event={"ID":"1118726b-19e7-4111-9934-60e937e54642","Type":"ContainerDied","Data":"9a9d75ad3267b9bcab31f7f5855fc2abac68f234a3de53000e16437b973da423"}
Jul 28 12:53:52.008601 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:52.008572    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-st9kv" event={"ID":"c549168d-d9be-4f07-8fbe-bf4d7c49d8c8","Type":"ContainerStarted","Data":"fe8326c4db6bb3626840ac463a10b737d3e5e8f9b162a543a050f38b67fce33f"}
Jul 28 12:53:52.052432 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:52.052392    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-st9kv" podStartSLOduration=7.086966876 podStartE2EDuration="28.052379064s" totalImagesPullingTime="20.965412188s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:53:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:53:52.051054102 +0000 UTC m=+28.836929213" watchObservedRunningTime="2026-07-28 12:53:52.052379064 +0000 UTC m=+28.838254175"
Jul 28 12:53:52.796547 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:52.796517    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4l2q4"
Jul 28 12:53:52.797156 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:52.796632    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4l2q4" podUID="78f39c2f-8193-46af-b59d-8dc730d2e61a"
Jul 28 12:53:52.797156 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:52.796667    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jcccx"
Jul 28 12:53:52.797156 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:52.796724    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jcccx" podUID="9f6f2eb9-22ea-4119-b474-75c3d6ec08eb"
Jul 28 12:53:53.010691 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:53.010662    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:53.010922 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:53.010736    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:53.010922 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:53.010776    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:53.025448 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:53.025422    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:53.025594 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:53.025541    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:53:53.731324 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:53.731083    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4l2q4"]
Jul 28 12:53:53.731486 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:53.731471    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4l2q4"
Jul 28 12:53:53.731614 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:53.731583    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4l2q4" podUID="78f39c2f-8193-46af-b59d-8dc730d2e61a"
Jul 28 12:53:53.734396 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:53.734374    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vvbch"]
Jul 28 12:53:53.734520 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:53.734485    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vvbch"
Jul 28 12:53:53.734633 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:53.734604    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vvbch" podUID="d04af0e5-3e6d-4027-8509-0af92acc4035"
Jul 28 12:53:53.734983 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:53.734963    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jcccx"]
Jul 28 12:53:53.735071 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:53.735057    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jcccx"
Jul 28 12:53:53.735169 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:53.735152    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jcccx" podUID="9f6f2eb9-22ea-4119-b474-75c3d6ec08eb"
Jul 28 12:53:54.014007 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:54.013978    2554 generic.go:356] "Generic (PLEG): container finished" podID="1118726b-19e7-4111-9934-60e937e54642" containerID="569856b7509521741a05e768a4554dfff15f6971614f7520d1e31c0cf1af88c5" exitCode=0
Jul 28 12:53:54.014612 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:54.014066    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ckrz7" event={"ID":"1118726b-19e7-4111-9934-60e937e54642","Type":"ContainerDied","Data":"569856b7509521741a05e768a4554dfff15f6971614f7520d1e31c0cf1af88c5"}
Jul 28 12:53:55.799983 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:55.799944    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4l2q4"
Jul 28 12:53:55.800483 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:55.800050    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vvbch"
Jul 28 12:53:55.800483 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:55.800062    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4l2q4" podUID="78f39c2f-8193-46af-b59d-8dc730d2e61a"
Jul 28 12:53:55.800483 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:55.800083    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jcccx"
Jul 28 12:53:55.800483 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:55.800165    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vvbch" podUID="d04af0e5-3e6d-4027-8509-0af92acc4035"
Jul 28 12:53:55.800483 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:55.800228    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jcccx" podUID="9f6f2eb9-22ea-4119-b474-75c3d6ec08eb"
Jul 28 12:53:56.021405 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:56.021326    2554 generic.go:356] "Generic (PLEG): container finished" podID="1118726b-19e7-4111-9934-60e937e54642" containerID="b55bb9bbdf496898bcd1e50604c21fe3d20a20f0fa94f9467efe17582ce97f1b" exitCode=0
Jul 28 12:53:56.021405 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:56.021384    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ckrz7" event={"ID":"1118726b-19e7-4111-9934-60e937e54642","Type":"ContainerDied","Data":"b55bb9bbdf496898bcd1e50604c21fe3d20a20f0fa94f9467efe17582ce97f1b"}
Jul 28 12:53:56.490124 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:56.490050    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vlhn\" (UniqueName: \"kubernetes.io/projected/78f39c2f-8193-46af-b59d-8dc730d2e61a-kube-api-access-7vlhn\") pod \"network-check-target-4l2q4\" (UID: \"78f39c2f-8193-46af-b59d-8dc730d2e61a\") " pod="openshift-network-diagnostics/network-check-target-4l2q4"
Jul 28 12:53:56.490124 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:56.490085    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d04af0e5-3e6d-4027-8509-0af92acc4035-metrics-certs\") pod \"network-metrics-daemon-vvbch\" (UID: \"d04af0e5-3e6d-4027-8509-0af92acc4035\") " pod="openshift-multus/network-metrics-daemon-vvbch"
Jul 28 12:53:56.490124 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:56.490113    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9f6f2eb9-22ea-4119-b474-75c3d6ec08eb-original-pull-secret\") pod \"global-pull-secret-syncer-jcccx\" (UID: \"9f6f2eb9-22ea-4119-b474-75c3d6ec08eb\") " pod="kube-system/global-pull-secret-syncer-jcccx"
Jul 28 12:53:56.490389 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:56.490187    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 12:53:56.490389 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:56.490196    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 12:53:56.490389 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:56.490206    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:53:56.490389 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:56.490216    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 12:53:56.490389 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:56.490228    2554 projected.go:196] Error preparing data for projected volume kube-api-access-7vlhn for pod openshift-network-diagnostics/network-check-target-4l2q4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:53:56.490389 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:56.490233    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f6f2eb9-22ea-4119-b474-75c3d6ec08eb-original-pull-secret podName:9f6f2eb9-22ea-4119-b474-75c3d6ec08eb nodeName:}" failed. No retries permitted until 2026-07-28 12:54:28.4902216 +0000 UTC m=+65.276096688 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9f6f2eb9-22ea-4119-b474-75c3d6ec08eb-original-pull-secret") pod "global-pull-secret-syncer-jcccx" (UID: "9f6f2eb9-22ea-4119-b474-75c3d6ec08eb") : object "kube-system"/"original-pull-secret" not registered
Jul 28 12:53:56.490389 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:56.490267    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/78f39c2f-8193-46af-b59d-8dc730d2e61a-kube-api-access-7vlhn podName:78f39c2f-8193-46af-b59d-8dc730d2e61a nodeName:}" failed. No retries permitted until 2026-07-28 12:54:28.490254183 +0000 UTC m=+65.276129272 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-7vlhn" (UniqueName: "kubernetes.io/projected/78f39c2f-8193-46af-b59d-8dc730d2e61a-kube-api-access-7vlhn") pod "network-check-target-4l2q4" (UID: "78f39c2f-8193-46af-b59d-8dc730d2e61a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 12:53:56.490389 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:56.490307    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d04af0e5-3e6d-4027-8509-0af92acc4035-metrics-certs podName:d04af0e5-3e6d-4027-8509-0af92acc4035 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:28.490296727 +0000 UTC m=+65.276171819 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d04af0e5-3e6d-4027-8509-0af92acc4035-metrics-certs") pod "network-metrics-daemon-vvbch" (UID: "d04af0e5-3e6d-4027-8509-0af92acc4035") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 12:53:57.799729 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:57.799698    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4l2q4"
Jul 28 12:53:57.800162 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:57.799821    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4l2q4" podUID="78f39c2f-8193-46af-b59d-8dc730d2e61a"
Jul 28 12:53:57.800162 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:57.799698    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vvbch"
Jul 28 12:53:57.800162 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:57.799930    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vvbch" podUID="d04af0e5-3e6d-4027-8509-0af92acc4035"
Jul 28 12:53:57.800162 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:57.799698    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jcccx"
Jul 28 12:53:57.800162 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:57.800011    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jcccx" podUID="9f6f2eb9-22ea-4119-b474-75c3d6ec08eb"
Jul 28 12:53:58.982201 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:58.982130    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-177.ec2.internal" event="NodeReady"
Jul 28 12:53:58.982618 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:58.982298    2554 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 12:53:59.035148 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:59.035118    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-29778"]
Jul 28 12:53:59.040060 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:59.040039    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-t4cwf"]
Jul 28 12:53:59.040182 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:59.040165    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-29778"
Jul 28 12:53:59.043088 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:59.042908    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 12:53:59.043088 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:59.042975    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 12:53:59.043219 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:59.043175    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 12:53:59.043533 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:59.043511    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vb57q\""
Jul 28 12:53:59.044047 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:59.044028    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-t4cwf"
Jul 28 12:53:59.048352 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:59.048332    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 12:53:59.049362 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:59.049339    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-t4cwf"]
Jul 28 12:53:59.049916 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:59.049896    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4l68m\""
Jul 28 12:53:59.050175 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:59.050157    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 12:53:59.054293 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:59.054254    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-29778"]
Jul 28 12:53:59.212477 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:59.212444    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zzpx\" (UniqueName: \"kubernetes.io/projected/41f6b60b-caf3-4c4c-92b0-386a82bcfc4a-kube-api-access-9zzpx\") pod \"ingress-canary-29778\" (UID: \"41f6b60b-caf3-4c4c-92b0-386a82bcfc4a\") " pod="openshift-ingress-canary/ingress-canary-29778"
Jul 28 12:53:59.212652 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:59.212494    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7-metrics-tls\") pod \"dns-default-t4cwf\" (UID: \"0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7\") " pod="openshift-dns/dns-default-t4cwf"
Jul 28 12:53:59.212652 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:59.212544    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t2xc\" (UniqueName: \"kubernetes.io/projected/0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7-kube-api-access-6t2xc\") pod \"dns-default-t4cwf\" (UID: \"0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7\") " pod="openshift-dns/dns-default-t4cwf"
Jul 28 12:53:59.212652 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:59.212565    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7-config-volume\") pod \"dns-default-t4cwf\" (UID: \"0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7\") " pod="openshift-dns/dns-default-t4cwf"
Jul 28 12:53:59.212652 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:59.212582    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7-tmp-dir\") pod \"dns-default-t4cwf\" (UID: \"0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7\") " pod="openshift-dns/dns-default-t4cwf"
Jul 28 12:53:59.212652 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:59.212626    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/41f6b60b-caf3-4c4c-92b0-386a82bcfc4a-cert\") pod \"ingress-canary-29778\" (UID: \"41f6b60b-caf3-4c4c-92b0-386a82bcfc4a\") " pod="openshift-ingress-canary/ingress-canary-29778"
Jul 28 12:53:59.313242 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:59.313163    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zzpx\" (UniqueName: \"kubernetes.io/projected/41f6b60b-caf3-4c4c-92b0-386a82bcfc4a-kube-api-access-9zzpx\") pod \"ingress-canary-29778\" (UID: \"41f6b60b-caf3-4c4c-92b0-386a82bcfc4a\") " pod="openshift-ingress-canary/ingress-canary-29778"
Jul 28 12:53:59.313242 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:59.313201    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7-metrics-tls\") pod \"dns-default-t4cwf\" (UID: \"0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7\") " pod="openshift-dns/dns-default-t4cwf"
Jul 28 12:53:59.313436 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:59.313324    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 12:53:59.313436 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:59.313349    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t2xc\" (UniqueName: \"kubernetes.io/projected/0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7-kube-api-access-6t2xc\") pod \"dns-default-t4cwf\" (UID: \"0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7\") " pod="openshift-dns/dns-default-t4cwf"
Jul 28 12:53:59.313436 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:59.313374    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7-metrics-tls podName:0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7 nodeName:}" failed. No retries permitted until 2026-07-28 12:53:59.813356296 +0000 UTC m=+36.599231386 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7-metrics-tls") pod "dns-default-t4cwf" (UID: "0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7") : secret "dns-default-metrics-tls" not found
Jul 28 12:53:59.313560 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:59.313486    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7-config-volume\") pod \"dns-default-t4cwf\" (UID: \"0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7\") " pod="openshift-dns/dns-default-t4cwf"
Jul 28 12:53:59.313560 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:59.313527    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7-tmp-dir\") pod \"dns-default-t4cwf\" (UID: \"0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7\") " pod="openshift-dns/dns-default-t4cwf"
Jul 28 12:53:59.313649 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:59.313563    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/41f6b60b-caf3-4c4c-92b0-386a82bcfc4a-cert\") pod \"ingress-canary-29778\" (UID: \"41f6b60b-caf3-4c4c-92b0-386a82bcfc4a\") " pod="openshift-ingress-canary/ingress-canary-29778"
Jul 28 12:53:59.313975 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:59.313952    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7-tmp-dir\") pod \"dns-default-t4cwf\" (UID: \"0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7\") " pod="openshift-dns/dns-default-t4cwf"
Jul 28 12:53:59.314127 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:59.314105    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7-config-volume\") pod \"dns-default-t4cwf\" (UID: \"0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7\") " pod="openshift-dns/dns-default-t4cwf"
Jul 28 12:53:59.318435 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:59.318409    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/41f6b60b-caf3-4c4c-92b0-386a82bcfc4a-cert\") pod \"ingress-canary-29778\" (UID: \"41f6b60b-caf3-4c4c-92b0-386a82bcfc4a\") " pod="openshift-ingress-canary/ingress-canary-29778"
Jul 28 12:53:59.326560 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:59.326535    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t2xc\" (UniqueName: \"kubernetes.io/projected/0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7-kube-api-access-6t2xc\") pod \"dns-default-t4cwf\" (UID: \"0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7\") " pod="openshift-dns/dns-default-t4cwf"
Jul 28 12:53:59.327003 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:59.326983    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zzpx\" (UniqueName: \"kubernetes.io/projected/41f6b60b-caf3-4c4c-92b0-386a82bcfc4a-kube-api-access-9zzpx\") pod \"ingress-canary-29778\" (UID: \"41f6b60b-caf3-4c4c-92b0-386a82bcfc4a\") " pod="openshift-ingress-canary/ingress-canary-29778"
Jul 28 12:53:59.353762 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:59.353737    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-29778"
Jul 28 12:53:59.800542 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:59.800313    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jcccx"
Jul 28 12:53:59.800687 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:59.800355    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4l2q4"
Jul 28 12:53:59.800687 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:59.800358    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vvbch"
Jul 28 12:53:59.803578 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:59.803555    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 12:53:59.803713 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:59.803584    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 12:53:59.803713 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:59.803557    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 12:53:59.804567 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:59.804545    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sxfnt\""
Jul 28 12:53:59.804650 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:59.804588    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 12:53:59.804650 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:59.804624    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wvgpn\""
Jul 28 12:53:59.815757 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:53:59.815732    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7-metrics-tls\") pod \"dns-default-t4cwf\" (UID: \"0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7\") " pod="openshift-dns/dns-default-t4cwf"
Jul 28 12:53:59.815896 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:59.815882    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 12:53:59.815959 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:53:59.815951    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7-metrics-tls podName:0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:00.815931302 +0000 UTC m=+37.601806398 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7-metrics-tls") pod "dns-default-t4cwf" (UID: "0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7") : secret "dns-default-metrics-tls" not found
Jul 28 12:54:00.241506 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:00.241481    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-29778"]
Jul 28 12:54:00.255399 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:54:00.255375    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6b60b_caf3_4c4c_92b0_386a82bcfc4a.slice/crio-e994bf472b4ddf8d46fd57cae9a726a8fb1675aa065171575100d5c8da1253c7 WatchSource:0}: Error finding container e994bf472b4ddf8d46fd57cae9a726a8fb1675aa065171575100d5c8da1253c7: Status 404 returned error can't find the container with id e994bf472b4ddf8d46fd57cae9a726a8fb1675aa065171575100d5c8da1253c7
Jul 28 12:54:00.822779 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:00.822695    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7-metrics-tls\") pod \"dns-default-t4cwf\" (UID: \"0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7\") " pod="openshift-dns/dns-default-t4cwf"
Jul 28 12:54:00.822935 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:00.822824    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 12:54:00.822935 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:00.822896    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7-metrics-tls podName:0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:02.822877209 +0000 UTC m=+39.608752300 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7-metrics-tls") pod "dns-default-t4cwf" (UID: "0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7") : secret "dns-default-metrics-tls" not found
Jul 28 12:54:01.036613 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:01.036578    2554 generic.go:356] "Generic (PLEG): container finished" podID="1118726b-19e7-4111-9934-60e937e54642" containerID="40c2c450c6041a11e1501be96ffe60f6f4d86f0f069ba2180fceffeea7863b7f" exitCode=0
Jul 28 12:54:01.036806 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:01.036629    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ckrz7" event={"ID":"1118726b-19e7-4111-9934-60e937e54642","Type":"ContainerDied","Data":"40c2c450c6041a11e1501be96ffe60f6f4d86f0f069ba2180fceffeea7863b7f"}
Jul 28 12:54:01.037969 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:01.037943    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-29778" event={"ID":"41f6b60b-caf3-4c4c-92b0-386a82bcfc4a","Type":"ContainerStarted","Data":"e994bf472b4ddf8d46fd57cae9a726a8fb1675aa065171575100d5c8da1253c7"}
Jul 28 12:54:02.049229 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:02.049191    2554 generic.go:356] "Generic (PLEG): container finished" podID="1118726b-19e7-4111-9934-60e937e54642" containerID="424cabe0eae3092589bfa3f11b615ce6b464171b5f559aef5524d41d718feae5" exitCode=0
Jul 28 12:54:02.049838 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:02.049247    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ckrz7" event={"ID":"1118726b-19e7-4111-9934-60e937e54642","Type":"ContainerDied","Data":"424cabe0eae3092589bfa3f11b615ce6b464171b5f559aef5524d41d718feae5"}
Jul 28 12:54:02.839937 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:02.839781    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7-metrics-tls\") pod \"dns-default-t4cwf\" (UID: \"0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7\") " pod="openshift-dns/dns-default-t4cwf"
Jul 28 12:54:02.840001 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:02.839917    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 12:54:02.840037 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:02.840028    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7-metrics-tls podName:0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:06.840014912 +0000 UTC m=+43.625890001 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7-metrics-tls") pod "dns-default-t4cwf" (UID: "0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7") : secret "dns-default-metrics-tls" not found
Jul 28 12:54:03.055626 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:03.055587    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ckrz7" event={"ID":"1118726b-19e7-4111-9934-60e937e54642","Type":"ContainerStarted","Data":"898dd560843a31e00e58fdd5714f97a444280c2a99f5cf2f56b17e9ab28c56e1"}
Jul 28 12:54:03.057255 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:03.057222    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-29778" event={"ID":"41f6b60b-caf3-4c4c-92b0-386a82bcfc4a","Type":"ContainerStarted","Data":"660d6594fa940aa9fc835a50de54663f483e4ba88417aed73d91efcafdd25222"}
Jul 28 12:54:03.083772 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:03.083700    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ckrz7" podStartSLOduration=7.134391694 podStartE2EDuration="39.083679925s" totalImagesPullingTime="31.949288231s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:53:24 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:54:03.082844686 +0000 UTC m=+39.868719809" watchObservedRunningTime="2026-07-28 12:54:03.083679925 +0000 UTC m=+39.869555035"
Jul 28 12:54:03.101238 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:03.101155    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-29778" podStartSLOduration=1.519777225 podStartE2EDuration="4.101140656s" totalImagesPullingTime="2.581363431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:53:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:54:03.101132668 +0000 UTC m=+39.887007778" watchObservedRunningTime="2026-07-28 12:54:03.101140656 +0000 UTC m=+39.887015967"
Jul 28 12:54:06.869052 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:06.869009    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7-metrics-tls\") pod \"dns-default-t4cwf\" (UID: \"0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7\") " pod="openshift-dns/dns-default-t4cwf"
Jul 28 12:54:06.869446 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:06.869143    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 12:54:06.869446 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:06.869199    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7-metrics-tls podName:0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:14.869183051 +0000 UTC m=+51.655058160 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7-metrics-tls") pod "dns-default-t4cwf" (UID: "0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7") : secret "dns-default-metrics-tls" not found
Jul 28 12:54:14.921077 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:14.921034    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7-metrics-tls\") pod \"dns-default-t4cwf\" (UID: \"0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7\") " pod="openshift-dns/dns-default-t4cwf"
Jul 28 12:54:14.921558 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:14.921182    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 12:54:14.921558 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:14.921257    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7-metrics-tls podName:0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:30.921241147 +0000 UTC m=+67.707116236 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7-metrics-tls") pod "dns-default-t4cwf" (UID: "0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7") : secret "dns-default-metrics-tls" not found
Jul 28 12:54:25.036850 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:25.036823    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-st9kv"
Jul 28 12:54:28.510812 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:28.510766    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9f6f2eb9-22ea-4119-b474-75c3d6ec08eb-original-pull-secret\") pod \"global-pull-secret-syncer-jcccx\" (UID: \"9f6f2eb9-22ea-4119-b474-75c3d6ec08eb\") " pod="kube-system/global-pull-secret-syncer-jcccx"
Jul 28 12:54:28.510812 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:28.510822    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vlhn\" (UniqueName: \"kubernetes.io/projected/78f39c2f-8193-46af-b59d-8dc730d2e61a-kube-api-access-7vlhn\") pod \"network-check-target-4l2q4\" (UID: \"78f39c2f-8193-46af-b59d-8dc730d2e61a\") " pod="openshift-network-diagnostics/network-check-target-4l2q4"
Jul 28 12:54:28.511240 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:28.510945    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d04af0e5-3e6d-4027-8509-0af92acc4035-metrics-certs\") pod \"network-metrics-daemon-vvbch\" (UID: \"d04af0e5-3e6d-4027-8509-0af92acc4035\") " pod="openshift-multus/network-metrics-daemon-vvbch"
Jul 28 12:54:28.513823 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:28.513802    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 12:54:28.513981 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:28.513964    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 12:54:28.514026 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:28.513987    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 12:54:28.521160 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:28.521140    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 28 12:54:28.521243 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:28.521208    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d04af0e5-3e6d-4027-8509-0af92acc4035-metrics-certs podName:d04af0e5-3e6d-4027-8509-0af92acc4035 nodeName:}" failed. No retries permitted until 2026-07-28 12:55:32.521187003 +0000 UTC m=+129.307062096 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d04af0e5-3e6d-4027-8509-0af92acc4035-metrics-certs") pod "network-metrics-daemon-vvbch" (UID: "d04af0e5-3e6d-4027-8509-0af92acc4035") : secret "metrics-daemon-secret" not found
Jul 28 12:54:28.523562 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:28.523547    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 12:54:28.524391 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:28.524375    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9f6f2eb9-22ea-4119-b474-75c3d6ec08eb-original-pull-secret\") pod \"global-pull-secret-syncer-jcccx\" (UID: \"9f6f2eb9-22ea-4119-b474-75c3d6ec08eb\") " pod="kube-system/global-pull-secret-syncer-jcccx"
Jul 28 12:54:28.533566 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:28.533547    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vlhn\" (UniqueName: \"kubernetes.io/projected/78f39c2f-8193-46af-b59d-8dc730d2e61a-kube-api-access-7vlhn\") pod \"network-check-target-4l2q4\" (UID: \"78f39c2f-8193-46af-b59d-8dc730d2e61a\") " pod="openshift-network-diagnostics/network-check-target-4l2q4"
Jul 28 12:54:28.612061 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:28.612034    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jcccx"
Jul 28 12:54:28.620931 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:28.620907    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sxfnt\""
Jul 28 12:54:28.629156 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:28.629136    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4l2q4"
Jul 28 12:54:28.747969 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:28.747940    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jcccx"]
Jul 28 12:54:28.750778 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:54:28.750735    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f6f2eb9_22ea_4119_b474_75c3d6ec08eb.slice/crio-d2712e4f0fff1eb39d8d0d7a419e89d966bd8f7568d05fcb61fa9c3dfc0e7793 WatchSource:0}: Error finding container d2712e4f0fff1eb39d8d0d7a419e89d966bd8f7568d05fcb61fa9c3dfc0e7793: Status 404 returned error can't find the container with id d2712e4f0fff1eb39d8d0d7a419e89d966bd8f7568d05fcb61fa9c3dfc0e7793
Jul 28 12:54:28.758465 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:28.758438    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4l2q4"]
Jul 28 12:54:28.761433 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:54:28.761392    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78f39c2f_8193_46af_b59d_8dc730d2e61a.slice/crio-1d0a9ba9b926bcfdec83189b3b36c937d2f55546bb0230feeea7479f5aefda02 WatchSource:0}: Error finding container 1d0a9ba9b926bcfdec83189b3b36c937d2f55546bb0230feeea7479f5aefda02: Status 404 returned error can't find the container with id 1d0a9ba9b926bcfdec83189b3b36c937d2f55546bb0230feeea7479f5aefda02
Jul 28 12:54:29.118172 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:29.118088    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4l2q4" event={"ID":"78f39c2f-8193-46af-b59d-8dc730d2e61a","Type":"ContainerStarted","Data":"1d0a9ba9b926bcfdec83189b3b36c937d2f55546bb0230feeea7479f5aefda02"}
Jul 28 12:54:29.119201 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:29.119180    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jcccx" event={"ID":"9f6f2eb9-22ea-4119-b474-75c3d6ec08eb","Type":"ContainerStarted","Data":"d2712e4f0fff1eb39d8d0d7a419e89d966bd8f7568d05fcb61fa9c3dfc0e7793"}
Jul 28 12:54:30.925797 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:30.925751    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7-metrics-tls\") pod \"dns-default-t4cwf\" (UID: \"0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7\") " pod="openshift-dns/dns-default-t4cwf"
Jul 28 12:54:30.926349 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:30.925899    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 12:54:30.926349 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:30.925970    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7-metrics-tls podName:0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7 nodeName:}" failed. No retries permitted until 2026-07-28 12:55:02.925950301 +0000 UTC m=+99.711825390 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7-metrics-tls") pod "dns-default-t4cwf" (UID: "0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7") : secret "dns-default-metrics-tls" not found
Jul 28 12:54:34.135451 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:34.135408    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4l2q4" event={"ID":"78f39c2f-8193-46af-b59d-8dc730d2e61a","Type":"ContainerStarted","Data":"99d40ce977e027d2cb8ef6ef95275ad6f3f347db6eb853398d2d6323d02274b3"}
Jul 28 12:54:34.135871 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:34.135483    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4l2q4"
Jul 28 12:54:34.136849 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:34.136829    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jcccx" event={"ID":"9f6f2eb9-22ea-4119-b474-75c3d6ec08eb","Type":"ContainerStarted","Data":"e3a997329e6ba9f332fa32ec3ae42afd991bc253ebc0f49fbb227d0dae61732f"}
Jul 28 12:54:34.175811 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:34.175760    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-jcccx" podStartSLOduration=65.295711462 podStartE2EDuration="1m10.175743787s" totalImagesPullingTime="4.880032325s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:53:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:54:34.175389029 +0000 UTC m=+70.961264139" watchObservedRunningTime="2026-07-28 12:54:34.175743787 +0000 UTC m=+70.961618902"
Jul 28 12:54:34.175960 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:34.175914    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-4l2q4" podStartSLOduration=65.318447019 podStartE2EDuration="1m10.175908005s" totalImagesPullingTime="4.857460986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:53:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:54:34.157723056 +0000 UTC m=+70.943598166" watchObservedRunningTime="2026-07-28 12:54:34.175908005 +0000 UTC m=+70.961783116"
Jul 28 12:54:38.824812 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:38.824778    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-mpgfw"]
Jul 28 12:54:38.826811 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:38.826793    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-mpgfw"
Jul 28 12:54:38.829146 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:38.829124    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 28 12:54:38.829920 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:38.829905    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 12:54:38.830016 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:38.829995    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 28 12:54:38.830730 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:38.830714    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 12:54:38.832496 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:38.832477    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-767866489f-t5k98"]
Jul 28 12:54:38.834674 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:38.834656    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-767866489f-t5k98"
Jul 28 12:54:38.839659 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:38.839619    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 28 12:54:38.839659 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:38.839629    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 28 12:54:38.839809 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:38.839674    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-2d7fd\""
Jul 28 12:54:38.839809 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:38.839719    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 28 12:54:38.840069 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:38.840025    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-bcxmp\""
Jul 28 12:54:38.840265 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:38.840227    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 28 12:54:38.840265 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:38.840244    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 28 12:54:38.840415 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:38.840361    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 28 12:54:38.844393 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:38.844372    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-mpgfw"]
Jul 28 12:54:38.845539 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:38.845520    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 28 12:54:38.855676 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:38.855652    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-767866489f-t5k98"]
Jul 28 12:54:38.876125 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:38.876105    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/324a14fe-0e26-4cea-b17f-5efda6b502de-service-ca-bundle\") pod \"insights-operator-5885fdb759-mpgfw\" (UID: \"324a14fe-0e26-4cea-b17f-5efda6b502de\") " pod="openshift-insights/insights-operator-5885fdb759-mpgfw"
Jul 28 12:54:38.876234 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:38.876134    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/128b78d0-c26a-4fb6-b914-1cafd9565ff7-default-certificate\") pod \"router-default-767866489f-t5k98\" (UID: \"128b78d0-c26a-4fb6-b914-1cafd9565ff7\") " pod="openshift-ingress/router-default-767866489f-t5k98"
Jul 28 12:54:38.876234 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:38.876152    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/128b78d0-c26a-4fb6-b914-1cafd9565ff7-service-ca-bundle\") pod \"router-default-767866489f-t5k98\" (UID: \"128b78d0-c26a-4fb6-b914-1cafd9565ff7\") " pod="openshift-ingress/router-default-767866489f-t5k98"
Jul 28 12:54:38.876234 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:38.876168    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/128b78d0-c26a-4fb6-b914-1cafd9565ff7-stats-auth\") pod \"router-default-767866489f-t5k98\" (UID: \"128b78d0-c26a-4fb6-b914-1cafd9565ff7\") " pod="openshift-ingress/router-default-767866489f-t5k98"
Jul 28 12:54:38.876234 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:38.876221    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/128b78d0-c26a-4fb6-b914-1cafd9565ff7-metrics-certs\") pod \"router-default-767866489f-t5k98\" (UID: \"128b78d0-c26a-4fb6-b914-1cafd9565ff7\") " pod="openshift-ingress/router-default-767866489f-t5k98"
Jul 28 12:54:38.876381 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:38.876248    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blhtp\" (UniqueName: \"kubernetes.io/projected/128b78d0-c26a-4fb6-b914-1cafd9565ff7-kube-api-access-blhtp\") pod \"router-default-767866489f-t5k98\" (UID: \"128b78d0-c26a-4fb6-b914-1cafd9565ff7\") " pod="openshift-ingress/router-default-767866489f-t5k98"
Jul 28 12:54:38.876381 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:38.876336    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shd47\" (UniqueName: \"kubernetes.io/projected/324a14fe-0e26-4cea-b17f-5efda6b502de-kube-api-access-shd47\") pod \"insights-operator-5885fdb759-mpgfw\" (UID: \"324a14fe-0e26-4cea-b17f-5efda6b502de\") " pod="openshift-insights/insights-operator-5885fdb759-mpgfw"
Jul 28 12:54:38.876381 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:38.876358    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/324a14fe-0e26-4cea-b17f-5efda6b502de-serving-cert\") pod \"insights-operator-5885fdb759-mpgfw\" (UID: \"324a14fe-0e26-4cea-b17f-5efda6b502de\") " pod="openshift-insights/insights-operator-5885fdb759-mpgfw"
Jul 28 12:54:38.876470 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:38.876387    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/324a14fe-0e26-4cea-b17f-5efda6b502de-snapshots\") pod \"insights-operator-5885fdb759-mpgfw\" (UID: \"324a14fe-0e26-4cea-b17f-5efda6b502de\") " pod="openshift-insights/insights-operator-5885fdb759-mpgfw"
Jul 28 12:54:38.876470 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:38.876415    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/324a14fe-0e26-4cea-b17f-5efda6b502de-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-mpgfw\" (UID: \"324a14fe-0e26-4cea-b17f-5efda6b502de\") " pod="openshift-insights/insights-operator-5885fdb759-mpgfw"
Jul 28 12:54:38.876470 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:38.876434    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/324a14fe-0e26-4cea-b17f-5efda6b502de-tmp\") pod \"insights-operator-5885fdb759-mpgfw\" (UID: \"324a14fe-0e26-4cea-b17f-5efda6b502de\") " pod="openshift-insights/insights-operator-5885fdb759-mpgfw"
Jul 28 12:54:38.918078 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:38.918051    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-lnd2f"]
Jul 28 12:54:38.920094 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:38.920080    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-lnd2f"
Jul 28 12:54:38.922955 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:38.922932    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-d4nf5\""
Jul 28 12:54:38.923098 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:38.922962    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 28 12:54:38.923171 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:38.923145    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 28 12:54:38.930239 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:38.930220    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-lnd2f"]
Jul 28 12:54:38.977362 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:38.977335    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/324a14fe-0e26-4cea-b17f-5efda6b502de-snapshots\") pod \"insights-operator-5885fdb759-mpgfw\" (UID: \"324a14fe-0e26-4cea-b17f-5efda6b502de\") " pod="openshift-insights/insights-operator-5885fdb759-mpgfw"
Jul 28 12:54:38.977534 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:38.977367    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/324a14fe-0e26-4cea-b17f-5efda6b502de-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-mpgfw\" (UID: \"324a14fe-0e26-4cea-b17f-5efda6b502de\") " pod="openshift-insights/insights-operator-5885fdb759-mpgfw"
Jul 28 12:54:38.977534 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:38.977386    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/324a14fe-0e26-4cea-b17f-5efda6b502de-tmp\") pod \"insights-operator-5885fdb759-mpgfw\" (UID: \"324a14fe-0e26-4cea-b17f-5efda6b502de\") " pod="openshift-insights/insights-operator-5885fdb759-mpgfw"
Jul 28 12:54:38.977534 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:38.977421    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/324a14fe-0e26-4cea-b17f-5efda6b502de-service-ca-bundle\") pod \"insights-operator-5885fdb759-mpgfw\" (UID: \"324a14fe-0e26-4cea-b17f-5efda6b502de\") " pod="openshift-insights/insights-operator-5885fdb759-mpgfw"
Jul 28 12:54:38.977534 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:38.977445    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/128b78d0-c26a-4fb6-b914-1cafd9565ff7-default-certificate\") pod \"router-default-767866489f-t5k98\" (UID: \"128b78d0-c26a-4fb6-b914-1cafd9565ff7\") " pod="openshift-ingress/router-default-767866489f-t5k98"
Jul 28 12:54:38.977705 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:38.977563    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/128b78d0-c26a-4fb6-b914-1cafd9565ff7-service-ca-bundle\") pod \"router-default-767866489f-t5k98\" (UID: \"128b78d0-c26a-4fb6-b914-1cafd9565ff7\") " pod="openshift-ingress/router-default-767866489f-t5k98"
Jul 28 12:54:38.977705 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:38.977592    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/128b78d0-c26a-4fb6-b914-1cafd9565ff7-stats-auth\") pod \"router-default-767866489f-t5k98\" (UID: \"128b78d0-c26a-4fb6-b914-1cafd9565ff7\") " pod="openshift-ingress/router-default-767866489f-t5k98"
Jul 28 12:54:38.977705 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:38.977615    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/128b78d0-c26a-4fb6-b914-1cafd9565ff7-metrics-certs\") pod \"router-default-767866489f-t5k98\" (UID: \"128b78d0-c26a-4fb6-b914-1cafd9565ff7\") " pod="openshift-ingress/router-default-767866489f-t5k98"
Jul 28 12:54:38.977705 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:38.977642    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blhtp\" (UniqueName: \"kubernetes.io/projected/128b78d0-c26a-4fb6-b914-1cafd9565ff7-kube-api-access-blhtp\") pod \"router-default-767866489f-t5k98\" (UID: \"128b78d0-c26a-4fb6-b914-1cafd9565ff7\") " pod="openshift-ingress/router-default-767866489f-t5k98"
Jul 28 12:54:38.977705 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:38.977686    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zdnm\" (UniqueName: \"kubernetes.io/projected/d0e527a0-8a32-4bff-bd54-c3fb3293d699-kube-api-access-2zdnm\") pod \"volume-data-source-validator-549c6d54cf-lnd2f\" (UID: \"d0e527a0-8a32-4bff-bd54-c3fb3293d699\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-lnd2f"
Jul 28 12:54:38.977938 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:38.977722    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shd47\" (UniqueName: \"kubernetes.io/projected/324a14fe-0e26-4cea-b17f-5efda6b502de-kube-api-access-shd47\") pod \"insights-operator-5885fdb759-mpgfw\" (UID: \"324a14fe-0e26-4cea-b17f-5efda6b502de\") " pod="openshift-insights/insights-operator-5885fdb759-mpgfw"
Jul 28 12:54:38.977938 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:38.977750    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/324a14fe-0e26-4cea-b17f-5efda6b502de-serving-cert\") pod \"insights-operator-5885fdb759-mpgfw\" (UID: \"324a14fe-0e26-4cea-b17f-5efda6b502de\") " pod="openshift-insights/insights-operator-5885fdb759-mpgfw"
Jul 28 12:54:38.977938 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:38.977786    2554 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 12:54:38.978141 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:38.977770    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/128b78d0-c26a-4fb6-b914-1cafd9565ff7-service-ca-bundle podName:128b78d0-c26a-4fb6-b914-1cafd9565ff7 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:39.477749503 +0000 UTC m=+76.263624612 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/128b78d0-c26a-4fb6-b914-1cafd9565ff7-service-ca-bundle") pod "router-default-767866489f-t5k98" (UID: "128b78d0-c26a-4fb6-b914-1cafd9565ff7") : configmap references non-existent config key: service-ca.crt
Jul 28 12:54:38.978307 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:38.978050    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/324a14fe-0e26-4cea-b17f-5efda6b502de-snapshots\") pod \"insights-operator-5885fdb759-mpgfw\" (UID: \"324a14fe-0e26-4cea-b17f-5efda6b502de\") " pod="openshift-insights/insights-operator-5885fdb759-mpgfw"
Jul 28 12:54:38.978396 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:38.978330    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/128b78d0-c26a-4fb6-b914-1cafd9565ff7-metrics-certs podName:128b78d0-c26a-4fb6-b914-1cafd9565ff7 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:39.478304991 +0000 UTC m=+76.264180085 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/128b78d0-c26a-4fb6-b914-1cafd9565ff7-metrics-certs") pod "router-default-767866489f-t5k98" (UID: "128b78d0-c26a-4fb6-b914-1cafd9565ff7") : secret "router-metrics-certs-default" not found
Jul 28 12:54:38.978396 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:38.978060    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/324a14fe-0e26-4cea-b17f-5efda6b502de-service-ca-bundle\") pod \"insights-operator-5885fdb759-mpgfw\" (UID: \"324a14fe-0e26-4cea-b17f-5efda6b502de\") " pod="openshift-insights/insights-operator-5885fdb759-mpgfw"
Jul 28 12:54:38.978396 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:38.978388    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/324a14fe-0e26-4cea-b17f-5efda6b502de-tmp\") pod \"insights-operator-5885fdb759-mpgfw\" (UID: \"324a14fe-0e26-4cea-b17f-5efda6b502de\") " pod="openshift-insights/insights-operator-5885fdb759-mpgfw"
Jul 28 12:54:38.978521 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:38.978448    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/324a14fe-0e26-4cea-b17f-5efda6b502de-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-mpgfw\" (UID: \"324a14fe-0e26-4cea-b17f-5efda6b502de\") " pod="openshift-insights/insights-operator-5885fdb759-mpgfw"
Jul 28 12:54:38.980183 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:38.980162    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/128b78d0-c26a-4fb6-b914-1cafd9565ff7-stats-auth\") pod \"router-default-767866489f-t5k98\" (UID: \"128b78d0-c26a-4fb6-b914-1cafd9565ff7\") " pod="openshift-ingress/router-default-767866489f-t5k98"
Jul 28 12:54:38.980323 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:38.980213    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/324a14fe-0e26-4cea-b17f-5efda6b502de-serving-cert\") pod \"insights-operator-5885fdb759-mpgfw\" (UID: \"324a14fe-0e26-4cea-b17f-5efda6b502de\") " pod="openshift-insights/insights-operator-5885fdb759-mpgfw"
Jul 28 12:54:38.980427 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:38.980409    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/128b78d0-c26a-4fb6-b914-1cafd9565ff7-default-certificate\") pod \"router-default-767866489f-t5k98\" (UID: \"128b78d0-c26a-4fb6-b914-1cafd9565ff7\") " pod="openshift-ingress/router-default-767866489f-t5k98"
Jul 28 12:54:38.996369 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:38.995838    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shd47\" (UniqueName: \"kubernetes.io/projected/324a14fe-0e26-4cea-b17f-5efda6b502de-kube-api-access-shd47\") pod \"insights-operator-5885fdb759-mpgfw\" (UID: \"324a14fe-0e26-4cea-b17f-5efda6b502de\") " pod="openshift-insights/insights-operator-5885fdb759-mpgfw"
Jul 28 12:54:38.997811 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:38.997773    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blhtp\" (UniqueName: \"kubernetes.io/projected/128b78d0-c26a-4fb6-b914-1cafd9565ff7-kube-api-access-blhtp\") pod \"router-default-767866489f-t5k98\" (UID: \"128b78d0-c26a-4fb6-b914-1cafd9565ff7\") " pod="openshift-ingress/router-default-767866489f-t5k98"
Jul 28 12:54:39.021367 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:39.021337    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-kc5t6"]
Jul 28 12:54:39.024308 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:39.024292    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-kc5t6"
Jul 28 12:54:39.031100 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:39.031075    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 28 12:54:39.031258 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:39.031100    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-dhpdw\""
Jul 28 12:54:39.031258 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:39.031155    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 12:54:39.031515 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:39.031368    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 12:54:39.032512 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:39.032496    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 28 12:54:39.039348 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:39.039327    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-kc5t6"]
Jul 28 12:54:39.078878 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:39.078806    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zdnm\" (UniqueName: \"kubernetes.io/projected/d0e527a0-8a32-4bff-bd54-c3fb3293d699-kube-api-access-2zdnm\") pod \"volume-data-source-validator-549c6d54cf-lnd2f\" (UID: \"d0e527a0-8a32-4bff-bd54-c3fb3293d699\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-lnd2f"
Jul 28 12:54:39.078878 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:39.078835    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1a00af5e-6526-4eb0-951d-1ac554676d9e-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-kc5t6\" (UID: \"1a00af5e-6526-4eb0-951d-1ac554676d9e\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-kc5t6"
Jul 28 12:54:39.078878 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:39.078852    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1a00af5e-6526-4eb0-951d-1ac554676d9e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-kc5t6\" (UID: \"1a00af5e-6526-4eb0-951d-1ac554676d9e\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-kc5t6"
Jul 28 12:54:39.078878 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:39.078868    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljv7n\" (UniqueName: \"kubernetes.io/projected/1a00af5e-6526-4eb0-951d-1ac554676d9e-kube-api-access-ljv7n\") pod \"cluster-monitoring-operator-548b489cb8-kc5t6\" (UID: \"1a00af5e-6526-4eb0-951d-1ac554676d9e\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-kc5t6"
Jul 28 12:54:39.087240 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:39.087214    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zdnm\" (UniqueName: \"kubernetes.io/projected/d0e527a0-8a32-4bff-bd54-c3fb3293d699-kube-api-access-2zdnm\") pod \"volume-data-source-validator-549c6d54cf-lnd2f\" (UID: \"d0e527a0-8a32-4bff-bd54-c3fb3293d699\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-lnd2f"
Jul 28 12:54:39.135802 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:39.135760    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-mpgfw"
Jul 28 12:54:39.179699 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:39.179661    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1a00af5e-6526-4eb0-951d-1ac554676d9e-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-kc5t6\" (UID: \"1a00af5e-6526-4eb0-951d-1ac554676d9e\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-kc5t6"
Jul 28 12:54:39.179857 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:39.179705    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1a00af5e-6526-4eb0-951d-1ac554676d9e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-kc5t6\" (UID: \"1a00af5e-6526-4eb0-951d-1ac554676d9e\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-kc5t6"
Jul 28 12:54:39.179857 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:39.179727    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljv7n\" (UniqueName: \"kubernetes.io/projected/1a00af5e-6526-4eb0-951d-1ac554676d9e-kube-api-access-ljv7n\") pod \"cluster-monitoring-operator-548b489cb8-kc5t6\" (UID: \"1a00af5e-6526-4eb0-951d-1ac554676d9e\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-kc5t6"
Jul 28 12:54:39.179857 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:39.179840    2554 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 12:54:39.180012 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:39.179892    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a00af5e-6526-4eb0-951d-1ac554676d9e-cluster-monitoring-operator-tls podName:1a00af5e-6526-4eb0-951d-1ac554676d9e nodeName:}" failed. No retries permitted until 2026-07-28 12:54:39.679877247 +0000 UTC m=+76.465752337 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1a00af5e-6526-4eb0-951d-1ac554676d9e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-kc5t6" (UID: "1a00af5e-6526-4eb0-951d-1ac554676d9e") : secret "cluster-monitoring-operator-tls" not found
Jul 28 12:54:39.180462 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:39.180428    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1a00af5e-6526-4eb0-951d-1ac554676d9e-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-kc5t6\" (UID: \"1a00af5e-6526-4eb0-951d-1ac554676d9e\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-kc5t6"
Jul 28 12:54:39.188744 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:39.188718    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljv7n\" (UniqueName: \"kubernetes.io/projected/1a00af5e-6526-4eb0-951d-1ac554676d9e-kube-api-access-ljv7n\") pod \"cluster-monitoring-operator-548b489cb8-kc5t6\" (UID: \"1a00af5e-6526-4eb0-951d-1ac554676d9e\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-kc5t6"
Jul 28 12:54:39.228576 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:39.228548    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-lnd2f"
Jul 28 12:54:39.256738 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:39.256712    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-mpgfw"]
Jul 28 12:54:39.259189 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:54:39.259162    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod324a14fe_0e26_4cea_b17f_5efda6b502de.slice/crio-ae0f21c771f97265a6e64bd57ebc48d340f7e8454bf54535b2efb3db207e4cb3 WatchSource:0}: Error finding container ae0f21c771f97265a6e64bd57ebc48d340f7e8454bf54535b2efb3db207e4cb3: Status 404 returned error can't find the container with id ae0f21c771f97265a6e64bd57ebc48d340f7e8454bf54535b2efb3db207e4cb3
Jul 28 12:54:39.348770 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:39.348695    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-lnd2f"]
Jul 28 12:54:39.351337 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:54:39.351314    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0e527a0_8a32_4bff_bd54_c3fb3293d699.slice/crio-ec5795fa0d83473ca1d3c30a561dab4ef73f1ff9ee4b36c953b34fc743b7ec72 WatchSource:0}: Error finding container ec5795fa0d83473ca1d3c30a561dab4ef73f1ff9ee4b36c953b34fc743b7ec72: Status 404 returned error can't find the container with id ec5795fa0d83473ca1d3c30a561dab4ef73f1ff9ee4b36c953b34fc743b7ec72
Jul 28 12:54:39.482151 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:39.482115    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/128b78d0-c26a-4fb6-b914-1cafd9565ff7-service-ca-bundle\") pod \"router-default-767866489f-t5k98\" (UID: \"128b78d0-c26a-4fb6-b914-1cafd9565ff7\") " pod="openshift-ingress/router-default-767866489f-t5k98"
Jul 28 12:54:39.482151 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:39.482151    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/128b78d0-c26a-4fb6-b914-1cafd9565ff7-metrics-certs\") pod \"router-default-767866489f-t5k98\" (UID: \"128b78d0-c26a-4fb6-b914-1cafd9565ff7\") " pod="openshift-ingress/router-default-767866489f-t5k98"
Jul 28 12:54:39.482429 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:39.482269    2554 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 12:54:39.482429 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:39.482328    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/128b78d0-c26a-4fb6-b914-1cafd9565ff7-service-ca-bundle podName:128b78d0-c26a-4fb6-b914-1cafd9565ff7 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:40.482306098 +0000 UTC m=+77.268181197 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/128b78d0-c26a-4fb6-b914-1cafd9565ff7-service-ca-bundle") pod "router-default-767866489f-t5k98" (UID: "128b78d0-c26a-4fb6-b914-1cafd9565ff7") : configmap references non-existent config key: service-ca.crt
Jul 28 12:54:39.482429 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:39.482354    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/128b78d0-c26a-4fb6-b914-1cafd9565ff7-metrics-certs podName:128b78d0-c26a-4fb6-b914-1cafd9565ff7 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:40.482345483 +0000 UTC m=+77.268220572 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/128b78d0-c26a-4fb6-b914-1cafd9565ff7-metrics-certs") pod "router-default-767866489f-t5k98" (UID: "128b78d0-c26a-4fb6-b914-1cafd9565ff7") : secret "router-metrics-certs-default" not found
Jul 28 12:54:39.684108 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:39.684024    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1a00af5e-6526-4eb0-951d-1ac554676d9e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-kc5t6\" (UID: \"1a00af5e-6526-4eb0-951d-1ac554676d9e\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-kc5t6"
Jul 28 12:54:39.684243 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:39.684168    2554 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 12:54:39.684243 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:39.684226    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a00af5e-6526-4eb0-951d-1ac554676d9e-cluster-monitoring-operator-tls podName:1a00af5e-6526-4eb0-951d-1ac554676d9e nodeName:}" failed. No retries permitted until 2026-07-28 12:54:40.684210599 +0000 UTC m=+77.470085687 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1a00af5e-6526-4eb0-951d-1ac554676d9e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-kc5t6" (UID: "1a00af5e-6526-4eb0-951d-1ac554676d9e") : secret "cluster-monitoring-operator-tls" not found
Jul 28 12:54:40.151704 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:40.151661    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-mpgfw" event={"ID":"324a14fe-0e26-4cea-b17f-5efda6b502de","Type":"ContainerStarted","Data":"ae0f21c771f97265a6e64bd57ebc48d340f7e8454bf54535b2efb3db207e4cb3"}
Jul 28 12:54:40.152950 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:40.152911    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-lnd2f" event={"ID":"d0e527a0-8a32-4bff-bd54-c3fb3293d699","Type":"ContainerStarted","Data":"ec5795fa0d83473ca1d3c30a561dab4ef73f1ff9ee4b36c953b34fc743b7ec72"}
Jul 28 12:54:40.490954 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:40.490913    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/128b78d0-c26a-4fb6-b914-1cafd9565ff7-service-ca-bundle\") pod \"router-default-767866489f-t5k98\" (UID: \"128b78d0-c26a-4fb6-b914-1cafd9565ff7\") " pod="openshift-ingress/router-default-767866489f-t5k98"
Jul 28 12:54:40.491132 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:40.490964    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/128b78d0-c26a-4fb6-b914-1cafd9565ff7-metrics-certs\") pod \"router-default-767866489f-t5k98\" (UID: \"128b78d0-c26a-4fb6-b914-1cafd9565ff7\") " pod="openshift-ingress/router-default-767866489f-t5k98"
Jul 28 12:54:40.491132 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:40.491092    2554 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 12:54:40.491261 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:40.491127    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/128b78d0-c26a-4fb6-b914-1cafd9565ff7-service-ca-bundle podName:128b78d0-c26a-4fb6-b914-1cafd9565ff7 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:42.491100584 +0000 UTC m=+79.276975677 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/128b78d0-c26a-4fb6-b914-1cafd9565ff7-service-ca-bundle") pod "router-default-767866489f-t5k98" (UID: "128b78d0-c26a-4fb6-b914-1cafd9565ff7") : configmap references non-existent config key: service-ca.crt
Jul 28 12:54:40.491261 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:40.491222    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/128b78d0-c26a-4fb6-b914-1cafd9565ff7-metrics-certs podName:128b78d0-c26a-4fb6-b914-1cafd9565ff7 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:42.491210611 +0000 UTC m=+79.277085700 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/128b78d0-c26a-4fb6-b914-1cafd9565ff7-metrics-certs") pod "router-default-767866489f-t5k98" (UID: "128b78d0-c26a-4fb6-b914-1cafd9565ff7") : secret "router-metrics-certs-default" not found
Jul 28 12:54:40.692504 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:40.692466    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1a00af5e-6526-4eb0-951d-1ac554676d9e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-kc5t6\" (UID: \"1a00af5e-6526-4eb0-951d-1ac554676d9e\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-kc5t6"
Jul 28 12:54:40.692690 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:40.692642    2554 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 12:54:40.692757 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:40.692724    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a00af5e-6526-4eb0-951d-1ac554676d9e-cluster-monitoring-operator-tls podName:1a00af5e-6526-4eb0-951d-1ac554676d9e nodeName:}" failed. No retries permitted until 2026-07-28 12:54:42.692703241 +0000 UTC m=+79.478578331 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1a00af5e-6526-4eb0-951d-1ac554676d9e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-kc5t6" (UID: "1a00af5e-6526-4eb0-951d-1ac554676d9e") : secret "cluster-monitoring-operator-tls" not found
Jul 28 12:54:42.161500 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:42.161459    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-mpgfw" event={"ID":"324a14fe-0e26-4cea-b17f-5efda6b502de","Type":"ContainerStarted","Data":"c8640666ec9d2afa1ca79022a648415bd1e1be3efe3b4a10405f419cae972516"}
Jul 28 12:54:42.162980 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:42.162958    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-lnd2f" event={"ID":"d0e527a0-8a32-4bff-bd54-c3fb3293d699","Type":"ContainerStarted","Data":"f01990e146d7beeedb9a4b20dc798e180ed4db0c8cc95611ae5fb08e07f32e6b"}
Jul 28 12:54:42.179581 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:42.179533    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-mpgfw" podStartSLOduration=2.332659769 podStartE2EDuration="4.179520004s" totalImagesPullingTime="1.846860235s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:54:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:54:42.179047436 +0000 UTC m=+78.964922548" watchObservedRunningTime="2026-07-28 12:54:42.179520004 +0000 UTC m=+78.965395115"
Jul 28 12:54:42.193938 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:42.193887    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-lnd2f" podStartSLOduration=2.4438472669999998 podStartE2EDuration="4.193868868s" totalImagesPullingTime="1.750021601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:54:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:54:42.192994676 +0000 UTC m=+78.978869789" watchObservedRunningTime="2026-07-28 12:54:42.193868868 +0000 UTC m=+78.979743980"
Jul 28 12:54:42.395791 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:42.395750    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rgh7t"]
Jul 28 12:54:42.399334 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:42.399312    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rgh7t"
Jul 28 12:54:42.401648 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:42.401628    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 12:54:42.401786 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:42.401766    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 12:54:42.401838 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:42.401806    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dl59l\""
Jul 28 12:54:42.408821 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:42.408798    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rgh7t"]
Jul 28 12:54:42.504507 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:42.504475    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmmms\" (UniqueName: \"kubernetes.io/projected/fbd7f2cb-b409-4e32-854e-043f12d72d21-kube-api-access-hmmms\") pod \"insights-runtime-extractor-rgh7t\" (UID: \"fbd7f2cb-b409-4e32-854e-043f12d72d21\") " pod="openshift-insights/insights-runtime-extractor-rgh7t"
Jul 28 12:54:42.504507 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:42.504505    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fbd7f2cb-b409-4e32-854e-043f12d72d21-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rgh7t\" (UID: \"fbd7f2cb-b409-4e32-854e-043f12d72d21\") " pod="openshift-insights/insights-runtime-extractor-rgh7t"
Jul 28 12:54:42.504715 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:42.504554    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/128b78d0-c26a-4fb6-b914-1cafd9565ff7-service-ca-bundle\") pod \"router-default-767866489f-t5k98\" (UID: \"128b78d0-c26a-4fb6-b914-1cafd9565ff7\") " pod="openshift-ingress/router-default-767866489f-t5k98"
Jul 28 12:54:42.504715 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:42.504572    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/128b78d0-c26a-4fb6-b914-1cafd9565ff7-metrics-certs\") pod \"router-default-767866489f-t5k98\" (UID: \"128b78d0-c26a-4fb6-b914-1cafd9565ff7\") " pod="openshift-ingress/router-default-767866489f-t5k98"
Jul 28 12:54:42.504715 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:42.504606    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fbd7f2cb-b409-4e32-854e-043f12d72d21-crio-socket\") pod \"insights-runtime-extractor-rgh7t\" (UID: \"fbd7f2cb-b409-4e32-854e-043f12d72d21\") " pod="openshift-insights/insights-runtime-extractor-rgh7t"
Jul 28 12:54:42.504715 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:42.504625    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fbd7f2cb-b409-4e32-854e-043f12d72d21-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rgh7t\" (UID: \"fbd7f2cb-b409-4e32-854e-043f12d72d21\") " pod="openshift-insights/insights-runtime-extractor-rgh7t"
Jul 28 12:54:42.504715 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:42.504646    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fbd7f2cb-b409-4e32-854e-043f12d72d21-data-volume\") pod \"insights-runtime-extractor-rgh7t\" (UID: \"fbd7f2cb-b409-4e32-854e-043f12d72d21\") " pod="openshift-insights/insights-runtime-extractor-rgh7t"
Jul 28 12:54:42.504715 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:42.504658    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/128b78d0-c26a-4fb6-b914-1cafd9565ff7-service-ca-bundle podName:128b78d0-c26a-4fb6-b914-1cafd9565ff7 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:46.504640842 +0000 UTC m=+83.290515930 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/128b78d0-c26a-4fb6-b914-1cafd9565ff7-service-ca-bundle") pod "router-default-767866489f-t5k98" (UID: "128b78d0-c26a-4fb6-b914-1cafd9565ff7") : configmap references non-existent config key: service-ca.crt
Jul 28 12:54:42.504715 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:42.504685    2554 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 12:54:42.504933 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:42.504738    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/128b78d0-c26a-4fb6-b914-1cafd9565ff7-metrics-certs podName:128b78d0-c26a-4fb6-b914-1cafd9565ff7 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:46.504726906 +0000 UTC m=+83.290602000 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/128b78d0-c26a-4fb6-b914-1cafd9565ff7-metrics-certs") pod "router-default-767866489f-t5k98" (UID: "128b78d0-c26a-4fb6-b914-1cafd9565ff7") : secret "router-metrics-certs-default" not found
Jul 28 12:54:42.605413 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:42.605372    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fbd7f2cb-b409-4e32-854e-043f12d72d21-crio-socket\") pod \"insights-runtime-extractor-rgh7t\" (UID: \"fbd7f2cb-b409-4e32-854e-043f12d72d21\") " pod="openshift-insights/insights-runtime-extractor-rgh7t"
Jul 28 12:54:42.605413 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:42.605416    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fbd7f2cb-b409-4e32-854e-043f12d72d21-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rgh7t\" (UID: \"fbd7f2cb-b409-4e32-854e-043f12d72d21\") " pod="openshift-insights/insights-runtime-extractor-rgh7t"
Jul 28 12:54:42.605625 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:42.605447    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fbd7f2cb-b409-4e32-854e-043f12d72d21-data-volume\") pod \"insights-runtime-extractor-rgh7t\" (UID: \"fbd7f2cb-b409-4e32-854e-043f12d72d21\") " pod="openshift-insights/insights-runtime-extractor-rgh7t"
Jul 28 12:54:42.605625 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:42.605489    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmmms\" (UniqueName: \"kubernetes.io/projected/fbd7f2cb-b409-4e32-854e-043f12d72d21-kube-api-access-hmmms\") pod \"insights-runtime-extractor-rgh7t\" (UID: \"fbd7f2cb-b409-4e32-854e-043f12d72d21\") " pod="openshift-insights/insights-runtime-extractor-rgh7t"
Jul 28 12:54:42.605625 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:42.605511    2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 12:54:42.605625 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:42.605522    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fbd7f2cb-b409-4e32-854e-043f12d72d21-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rgh7t\" (UID: \"fbd7f2cb-b409-4e32-854e-043f12d72d21\") " pod="openshift-insights/insights-runtime-extractor-rgh7t"
Jul 28 12:54:42.605625 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:42.605492    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fbd7f2cb-b409-4e32-854e-043f12d72d21-crio-socket\") pod \"insights-runtime-extractor-rgh7t\" (UID: \"fbd7f2cb-b409-4e32-854e-043f12d72d21\") " pod="openshift-insights/insights-runtime-extractor-rgh7t"
Jul 28 12:54:42.605625 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:42.605574    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fbd7f2cb-b409-4e32-854e-043f12d72d21-kube-rbac-proxy-cm podName:fbd7f2cb-b409-4e32-854e-043f12d72d21 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:43.105555028 +0000 UTC m=+79.891430140 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fbd7f2cb-b409-4e32-854e-043f12d72d21-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rgh7t" (UID: "fbd7f2cb-b409-4e32-854e-043f12d72d21") : configmap "kube-rbac-proxy" not found
Jul 28 12:54:42.605843 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:42.605636    2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 12:54:42.605843 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:42.605680    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbd7f2cb-b409-4e32-854e-043f12d72d21-insights-runtime-extractor-tls podName:fbd7f2cb-b409-4e32-854e-043f12d72d21 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:43.10566843 +0000 UTC m=+79.891543520 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fbd7f2cb-b409-4e32-854e-043f12d72d21-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rgh7t" (UID: "fbd7f2cb-b409-4e32-854e-043f12d72d21") : secret "insights-runtime-extractor-tls" not found
Jul 28 12:54:42.605843 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:42.605742    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fbd7f2cb-b409-4e32-854e-043f12d72d21-data-volume\") pod \"insights-runtime-extractor-rgh7t\" (UID: \"fbd7f2cb-b409-4e32-854e-043f12d72d21\") " pod="openshift-insights/insights-runtime-extractor-rgh7t"
Jul 28 12:54:42.614608 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:42.614579    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmmms\" (UniqueName: \"kubernetes.io/projected/fbd7f2cb-b409-4e32-854e-043f12d72d21-kube-api-access-hmmms\") pod \"insights-runtime-extractor-rgh7t\" (UID: \"fbd7f2cb-b409-4e32-854e-043f12d72d21\") " pod="openshift-insights/insights-runtime-extractor-rgh7t"
Jul 28 12:54:42.706418 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:42.706379    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1a00af5e-6526-4eb0-951d-1ac554676d9e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-kc5t6\" (UID: \"1a00af5e-6526-4eb0-951d-1ac554676d9e\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-kc5t6"
Jul 28 12:54:42.706558 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:42.706526    2554 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 12:54:42.706662 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:42.706594    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a00af5e-6526-4eb0-951d-1ac554676d9e-cluster-monitoring-operator-tls podName:1a00af5e-6526-4eb0-951d-1ac554676d9e nodeName:}" failed. No retries permitted until 2026-07-28 12:54:46.706573121 +0000 UTC m=+83.492448211 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1a00af5e-6526-4eb0-951d-1ac554676d9e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-kc5t6" (UID: "1a00af5e-6526-4eb0-951d-1ac554676d9e") : secret "cluster-monitoring-operator-tls" not found
Jul 28 12:54:43.109130 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:43.109091    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fbd7f2cb-b409-4e32-854e-043f12d72d21-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rgh7t\" (UID: \"fbd7f2cb-b409-4e32-854e-043f12d72d21\") " pod="openshift-insights/insights-runtime-extractor-rgh7t"
Jul 28 12:54:43.109372 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:43.109201    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fbd7f2cb-b409-4e32-854e-043f12d72d21-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rgh7t\" (UID: \"fbd7f2cb-b409-4e32-854e-043f12d72d21\") " pod="openshift-insights/insights-runtime-extractor-rgh7t"
Jul 28 12:54:43.109372 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:43.109247    2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 12:54:43.109372 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:43.109269    2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 12:54:43.109372 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:43.109332    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbd7f2cb-b409-4e32-854e-043f12d72d21-insights-runtime-extractor-tls podName:fbd7f2cb-b409-4e32-854e-043f12d72d21 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:44.109316465 +0000 UTC m=+80.895191554 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fbd7f2cb-b409-4e32-854e-043f12d72d21-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rgh7t" (UID: "fbd7f2cb-b409-4e32-854e-043f12d72d21") : secret "insights-runtime-extractor-tls" not found
Jul 28 12:54:43.109372 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:43.109347    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fbd7f2cb-b409-4e32-854e-043f12d72d21-kube-rbac-proxy-cm podName:fbd7f2cb-b409-4e32-854e-043f12d72d21 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:44.109340373 +0000 UTC m=+80.895215462 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fbd7f2cb-b409-4e32-854e-043f12d72d21-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rgh7t" (UID: "fbd7f2cb-b409-4e32-854e-043f12d72d21") : configmap "kube-rbac-proxy" not found
Jul 28 12:54:44.116897 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:44.116843    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fbd7f2cb-b409-4e32-854e-043f12d72d21-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rgh7t\" (UID: \"fbd7f2cb-b409-4e32-854e-043f12d72d21\") " pod="openshift-insights/insights-runtime-extractor-rgh7t"
Jul 28 12:54:44.117354 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:44.116937    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fbd7f2cb-b409-4e32-854e-043f12d72d21-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rgh7t\" (UID: \"fbd7f2cb-b409-4e32-854e-043f12d72d21\") " pod="openshift-insights/insights-runtime-extractor-rgh7t"
Jul 28 12:54:44.117354 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:44.116974    2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 12:54:44.117354 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:44.117047    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fbd7f2cb-b409-4e32-854e-043f12d72d21-kube-rbac-proxy-cm podName:fbd7f2cb-b409-4e32-854e-043f12d72d21 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:46.117028529 +0000 UTC m=+82.902903621 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fbd7f2cb-b409-4e32-854e-043f12d72d21-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rgh7t" (UID: "fbd7f2cb-b409-4e32-854e-043f12d72d21") : configmap "kube-rbac-proxy" not found
Jul 28 12:54:44.117354 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:44.117092    2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 12:54:44.117354 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:44.117150    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbd7f2cb-b409-4e32-854e-043f12d72d21-insights-runtime-extractor-tls podName:fbd7f2cb-b409-4e32-854e-043f12d72d21 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:46.117136834 +0000 UTC m=+82.903011923 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fbd7f2cb-b409-4e32-854e-043f12d72d21-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rgh7t" (UID: "fbd7f2cb-b409-4e32-854e-043f12d72d21") : secret "insights-runtime-extractor-tls" not found
Jul 28 12:54:44.564491 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:44.564463    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-sn9v6_d336b5ed-db42-40f6-9e7a-a1882cb28484/dns-node-resolver/0.log"
Jul 28 12:54:44.754349 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:44.754205    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-mw6cm"]
Jul 28 12:54:44.757135 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:44.757115    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-mw6cm"
Jul 28 12:54:44.762879 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:44.762849    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 28 12:54:44.762879 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:44.762850    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 28 12:54:44.763347 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:44.763088    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 28 12:54:44.763778 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:44.763763    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-fwkgm\""
Jul 28 12:54:44.763926 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:44.763907    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 28 12:54:44.775767 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:44.775750    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-mw6cm"]
Jul 28 12:54:44.821903 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:44.821833    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ecfec774-88e6-4647-be3f-7946bc703a4f-signing-key\") pod \"service-ca-658d7b7c78-mw6cm\" (UID: \"ecfec774-88e6-4647-be3f-7946bc703a4f\") " pod="openshift-service-ca/service-ca-658d7b7c78-mw6cm"
Jul 28 12:54:44.821903 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:44.821901    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ecfec774-88e6-4647-be3f-7946bc703a4f-signing-cabundle\") pod \"service-ca-658d7b7c78-mw6cm\" (UID: \"ecfec774-88e6-4647-be3f-7946bc703a4f\") " pod="openshift-service-ca/service-ca-658d7b7c78-mw6cm"
Jul 28 12:54:44.822045 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:44.821969    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twfd6\" (UniqueName: \"kubernetes.io/projected/ecfec774-88e6-4647-be3f-7946bc703a4f-kube-api-access-twfd6\") pod \"service-ca-658d7b7c78-mw6cm\" (UID: \"ecfec774-88e6-4647-be3f-7946bc703a4f\") " pod="openshift-service-ca/service-ca-658d7b7c78-mw6cm"
Jul 28 12:54:44.922483 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:44.922437    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ecfec774-88e6-4647-be3f-7946bc703a4f-signing-cabundle\") pod \"service-ca-658d7b7c78-mw6cm\" (UID: \"ecfec774-88e6-4647-be3f-7946bc703a4f\") " pod="openshift-service-ca/service-ca-658d7b7c78-mw6cm"
Jul 28 12:54:44.922580 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:44.922496    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twfd6\" (UniqueName: \"kubernetes.io/projected/ecfec774-88e6-4647-be3f-7946bc703a4f-kube-api-access-twfd6\") pod \"service-ca-658d7b7c78-mw6cm\" (UID: \"ecfec774-88e6-4647-be3f-7946bc703a4f\") " pod="openshift-service-ca/service-ca-658d7b7c78-mw6cm"
Jul 28 12:54:44.922616 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:44.922580    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ecfec774-88e6-4647-be3f-7946bc703a4f-signing-key\") pod \"service-ca-658d7b7c78-mw6cm\" (UID: \"ecfec774-88e6-4647-be3f-7946bc703a4f\") " pod="openshift-service-ca/service-ca-658d7b7c78-mw6cm"
Jul 28 12:54:44.923196 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:44.923169    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ecfec774-88e6-4647-be3f-7946bc703a4f-signing-cabundle\") pod \"service-ca-658d7b7c78-mw6cm\" (UID: \"ecfec774-88e6-4647-be3f-7946bc703a4f\") " pod="openshift-service-ca/service-ca-658d7b7c78-mw6cm"
Jul 28 12:54:44.925038 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:44.925014    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ecfec774-88e6-4647-be3f-7946bc703a4f-signing-key\") pod \"service-ca-658d7b7c78-mw6cm\" (UID: \"ecfec774-88e6-4647-be3f-7946bc703a4f\") " pod="openshift-service-ca/service-ca-658d7b7c78-mw6cm"
Jul 28 12:54:44.932870 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:44.932849    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twfd6\" (UniqueName: \"kubernetes.io/projected/ecfec774-88e6-4647-be3f-7946bc703a4f-kube-api-access-twfd6\") pod \"service-ca-658d7b7c78-mw6cm\" (UID: \"ecfec774-88e6-4647-be3f-7946bc703a4f\") " pod="openshift-service-ca/service-ca-658d7b7c78-mw6cm"
Jul 28 12:54:45.065446 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:45.065405    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-mw6cm"
Jul 28 12:54:45.164170 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:45.164149    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-68tbg_48e4bcde-dcb9-4e7e-8b15-e8682eb564d1/node-ca/0.log"
Jul 28 12:54:45.187410 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:45.187384    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-mw6cm"]
Jul 28 12:54:45.190296 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:54:45.190259    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecfec774_88e6_4647_be3f_7946bc703a4f.slice/crio-22a881a8d02147af5dbd4751136912821aaa9da1477d09755422494701b2503e WatchSource:0}: Error finding container 22a881a8d02147af5dbd4751136912821aaa9da1477d09755422494701b2503e: Status 404 returned error can't find the container with id 22a881a8d02147af5dbd4751136912821aaa9da1477d09755422494701b2503e
Jul 28 12:54:45.964802 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:45.964772    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-29778_41f6b60b-caf3-4c4c-92b0-386a82bcfc4a/serve-healthcheck-canary/0.log"
Jul 28 12:54:46.131116 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:46.131078    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fbd7f2cb-b409-4e32-854e-043f12d72d21-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rgh7t\" (UID: \"fbd7f2cb-b409-4e32-854e-043f12d72d21\") " pod="openshift-insights/insights-runtime-extractor-rgh7t"
Jul 28 12:54:46.131306 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:46.131190    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fbd7f2cb-b409-4e32-854e-043f12d72d21-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rgh7t\" (UID: \"fbd7f2cb-b409-4e32-854e-043f12d72d21\") " pod="openshift-insights/insights-runtime-extractor-rgh7t"
Jul 28 12:54:46.131306 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:46.131254    2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 12:54:46.131441 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:46.131311    2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 12:54:46.131441 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:46.131348    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbd7f2cb-b409-4e32-854e-043f12d72d21-insights-runtime-extractor-tls podName:fbd7f2cb-b409-4e32-854e-043f12d72d21 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:50.131323652 +0000 UTC m=+86.917198757 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fbd7f2cb-b409-4e32-854e-043f12d72d21-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rgh7t" (UID: "fbd7f2cb-b409-4e32-854e-043f12d72d21") : secret "insights-runtime-extractor-tls" not found
Jul 28 12:54:46.131441 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:46.131366    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fbd7f2cb-b409-4e32-854e-043f12d72d21-kube-rbac-proxy-cm podName:fbd7f2cb-b409-4e32-854e-043f12d72d21 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:50.131359215 +0000 UTC m=+86.917234303 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fbd7f2cb-b409-4e32-854e-043f12d72d21-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rgh7t" (UID: "fbd7f2cb-b409-4e32-854e-043f12d72d21") : configmap "kube-rbac-proxy" not found
Jul 28 12:54:46.174824 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:46.174798    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-mw6cm" event={"ID":"ecfec774-88e6-4647-be3f-7946bc703a4f","Type":"ContainerStarted","Data":"22a881a8d02147af5dbd4751136912821aaa9da1477d09755422494701b2503e"}
Jul 28 12:54:46.534453 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:46.534406    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/128b78d0-c26a-4fb6-b914-1cafd9565ff7-service-ca-bundle\") pod \"router-default-767866489f-t5k98\" (UID: \"128b78d0-c26a-4fb6-b914-1cafd9565ff7\") " pod="openshift-ingress/router-default-767866489f-t5k98"
Jul 28 12:54:46.534641 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:46.534521    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/128b78d0-c26a-4fb6-b914-1cafd9565ff7-metrics-certs\") pod \"router-default-767866489f-t5k98\" (UID: \"128b78d0-c26a-4fb6-b914-1cafd9565ff7\") " pod="openshift-ingress/router-default-767866489f-t5k98"
Jul 28 12:54:46.534641 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:46.534597    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/128b78d0-c26a-4fb6-b914-1cafd9565ff7-service-ca-bundle podName:128b78d0-c26a-4fb6-b914-1cafd9565ff7 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:54.534579936 +0000 UTC m=+91.320455024 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/128b78d0-c26a-4fb6-b914-1cafd9565ff7-service-ca-bundle") pod "router-default-767866489f-t5k98" (UID: "128b78d0-c26a-4fb6-b914-1cafd9565ff7") : configmap references non-existent config key: service-ca.crt
Jul 28 12:54:46.534641 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:46.534635    2554 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 12:54:46.534819 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:46.534686    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/128b78d0-c26a-4fb6-b914-1cafd9565ff7-metrics-certs podName:128b78d0-c26a-4fb6-b914-1cafd9565ff7 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:54.534670244 +0000 UTC m=+91.320545334 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/128b78d0-c26a-4fb6-b914-1cafd9565ff7-metrics-certs") pod "router-default-767866489f-t5k98" (UID: "128b78d0-c26a-4fb6-b914-1cafd9565ff7") : secret "router-metrics-certs-default" not found
Jul 28 12:54:46.735898 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:46.735860    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1a00af5e-6526-4eb0-951d-1ac554676d9e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-kc5t6\" (UID: \"1a00af5e-6526-4eb0-951d-1ac554676d9e\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-kc5t6"
Jul 28 12:54:46.736085 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:46.736003    2554 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 12:54:46.736085 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:46.736060    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a00af5e-6526-4eb0-951d-1ac554676d9e-cluster-monitoring-operator-tls podName:1a00af5e-6526-4eb0-951d-1ac554676d9e nodeName:}" failed. No retries permitted until 2026-07-28 12:54:54.736045849 +0000 UTC m=+91.521920939 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1a00af5e-6526-4eb0-951d-1ac554676d9e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-kc5t6" (UID: "1a00af5e-6526-4eb0-951d-1ac554676d9e") : secret "cluster-monitoring-operator-tls" not found
Jul 28 12:54:47.181591 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:47.181553    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-mw6cm" event={"ID":"ecfec774-88e6-4647-be3f-7946bc703a4f","Type":"ContainerStarted","Data":"25ccda315328ea2f767864f9afc2918a405f05f3ab2c3b73ee5897be21c5e878"}
Jul 28 12:54:47.200255 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:47.200206    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-mw6cm" podStartSLOduration=1.415624175 podStartE2EDuration="3.200191722s" totalImagesPullingTime="1.784567547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:54:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:54:47.198882725 +0000 UTC m=+83.984757841" watchObservedRunningTime="2026-07-28 12:54:47.200191722 +0000 UTC m=+83.986066832"
Jul 28 12:54:50.163350 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:50.163315    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fbd7f2cb-b409-4e32-854e-043f12d72d21-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rgh7t\" (UID: \"fbd7f2cb-b409-4e32-854e-043f12d72d21\") " pod="openshift-insights/insights-runtime-extractor-rgh7t"
Jul 28 12:54:50.163779 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:50.163430    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fbd7f2cb-b409-4e32-854e-043f12d72d21-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rgh7t\" (UID: \"fbd7f2cb-b409-4e32-854e-043f12d72d21\") " pod="openshift-insights/insights-runtime-extractor-rgh7t"
Jul 28 12:54:50.163779 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:50.163475    2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 12:54:50.163779 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:50.163513    2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 12:54:50.163779 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:50.163546    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbd7f2cb-b409-4e32-854e-043f12d72d21-insights-runtime-extractor-tls podName:fbd7f2cb-b409-4e32-854e-043f12d72d21 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:58.163530547 +0000 UTC m=+94.949405636 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fbd7f2cb-b409-4e32-854e-043f12d72d21-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rgh7t" (UID: "fbd7f2cb-b409-4e32-854e-043f12d72d21") : secret "insights-runtime-extractor-tls" not found
Jul 28 12:54:50.163779 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:50.163587    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fbd7f2cb-b409-4e32-854e-043f12d72d21-kube-rbac-proxy-cm podName:fbd7f2cb-b409-4e32-854e-043f12d72d21 nodeName:}" failed. No retries permitted until 2026-07-28 12:54:58.163569008 +0000 UTC m=+94.949444104 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fbd7f2cb-b409-4e32-854e-043f12d72d21-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rgh7t" (UID: "fbd7f2cb-b409-4e32-854e-043f12d72d21") : configmap "kube-rbac-proxy" not found
Jul 28 12:54:54.599039 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:54.599000    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/128b78d0-c26a-4fb6-b914-1cafd9565ff7-metrics-certs\") pod \"router-default-767866489f-t5k98\" (UID: \"128b78d0-c26a-4fb6-b914-1cafd9565ff7\") " pod="openshift-ingress/router-default-767866489f-t5k98"
Jul 28 12:54:54.599443 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:54.599097    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/128b78d0-c26a-4fb6-b914-1cafd9565ff7-service-ca-bundle\") pod \"router-default-767866489f-t5k98\" (UID: \"128b78d0-c26a-4fb6-b914-1cafd9565ff7\") " pod="openshift-ingress/router-default-767866489f-t5k98"
Jul 28 12:54:54.599749 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:54.599731    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/128b78d0-c26a-4fb6-b914-1cafd9565ff7-service-ca-bundle\") pod \"router-default-767866489f-t5k98\" (UID: \"128b78d0-c26a-4fb6-b914-1cafd9565ff7\") " pod="openshift-ingress/router-default-767866489f-t5k98"
Jul 28 12:54:54.601557 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:54.601528    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/128b78d0-c26a-4fb6-b914-1cafd9565ff7-metrics-certs\") pod \"router-default-767866489f-t5k98\" (UID: \"128b78d0-c26a-4fb6-b914-1cafd9565ff7\") " pod="openshift-ingress/router-default-767866489f-t5k98"
Jul 28 12:54:54.746483 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:54.746446    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-767866489f-t5k98"
Jul 28 12:54:54.801212 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:54.801178    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1a00af5e-6526-4eb0-951d-1ac554676d9e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-kc5t6\" (UID: \"1a00af5e-6526-4eb0-951d-1ac554676d9e\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-kc5t6"
Jul 28 12:54:54.801547 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:54.801365    2554 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 12:54:54.801547 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:54.801451    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a00af5e-6526-4eb0-951d-1ac554676d9e-cluster-monitoring-operator-tls podName:1a00af5e-6526-4eb0-951d-1ac554676d9e nodeName:}" failed. No retries permitted until 2026-07-28 12:55:10.80142895 +0000 UTC m=+107.587304045 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1a00af5e-6526-4eb0-951d-1ac554676d9e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-kc5t6" (UID: "1a00af5e-6526-4eb0-951d-1ac554676d9e") : secret "cluster-monitoring-operator-tls" not found
Jul 28 12:54:54.873131 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:54.873064    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-767866489f-t5k98"]
Jul 28 12:54:54.876290 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:54:54.876248    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod128b78d0_c26a_4fb6_b914_1cafd9565ff7.slice/crio-c025c2577844e98d5875053e153048a701452e61bf2a2d7a91b94c46a2dd1cdc WatchSource:0}: Error finding container c025c2577844e98d5875053e153048a701452e61bf2a2d7a91b94c46a2dd1cdc: Status 404 returned error can't find the container with id c025c2577844e98d5875053e153048a701452e61bf2a2d7a91b94c46a2dd1cdc
Jul 28 12:54:55.209006 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:55.208970    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-767866489f-t5k98" event={"ID":"128b78d0-c26a-4fb6-b914-1cafd9565ff7","Type":"ContainerStarted","Data":"ee6110d9697bc3fe7928df5bf823b37013abb4046d1bd45fd2d5ffa59113f7e8"}
Jul 28 12:54:55.209195 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:55.209018    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-767866489f-t5k98" event={"ID":"128b78d0-c26a-4fb6-b914-1cafd9565ff7","Type":"ContainerStarted","Data":"c025c2577844e98d5875053e153048a701452e61bf2a2d7a91b94c46a2dd1cdc"}
Jul 28 12:54:55.234037 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:55.233984    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-767866489f-t5k98" podStartSLOduration=17.233965076 podStartE2EDuration="17.233965076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:54:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:54:55.23317726 +0000 UTC m=+92.019052387" watchObservedRunningTime="2026-07-28 12:54:55.233965076 +0000 UTC m=+92.019840188"
Jul 28 12:54:55.747463 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:55.747429    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-767866489f-t5k98"
Jul 28 12:54:55.750163 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:55.750143    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-767866489f-t5k98"
Jul 28 12:54:56.211524 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:56.211489    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-767866489f-t5k98"
Jul 28 12:54:56.212713 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:56.212692    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-767866489f-t5k98"
Jul 28 12:54:58.231796 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:58.231769    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fbd7f2cb-b409-4e32-854e-043f12d72d21-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rgh7t\" (UID: \"fbd7f2cb-b409-4e32-854e-043f12d72d21\") " pod="openshift-insights/insights-runtime-extractor-rgh7t"
Jul 28 12:54:58.232193 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:58.231833    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fbd7f2cb-b409-4e32-854e-043f12d72d21-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rgh7t\" (UID: \"fbd7f2cb-b409-4e32-854e-043f12d72d21\") " pod="openshift-insights/insights-runtime-extractor-rgh7t"
Jul 28 12:54:58.232193 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:58.231918    2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 12:54:58.232193 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:54:58.231977    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbd7f2cb-b409-4e32-854e-043f12d72d21-insights-runtime-extractor-tls podName:fbd7f2cb-b409-4e32-854e-043f12d72d21 nodeName:}" failed. No retries permitted until 2026-07-28 12:55:14.231959967 +0000 UTC m=+111.017835057 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fbd7f2cb-b409-4e32-854e-043f12d72d21-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rgh7t" (UID: "fbd7f2cb-b409-4e32-854e-043f12d72d21") : secret "insights-runtime-extractor-tls" not found
Jul 28 12:54:58.232351 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:54:58.232253    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fbd7f2cb-b409-4e32-854e-043f12d72d21-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rgh7t\" (UID: \"fbd7f2cb-b409-4e32-854e-043f12d72d21\") " pod="openshift-insights/insights-runtime-extractor-rgh7t"
Jul 28 12:55:02.967764 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:02.967723    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7-metrics-tls\") pod \"dns-default-t4cwf\" (UID: \"0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7\") " pod="openshift-dns/dns-default-t4cwf"
Jul 28 12:55:02.970083 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:02.970049    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7-metrics-tls\") pod \"dns-default-t4cwf\" (UID: \"0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7\") " pod="openshift-dns/dns-default-t4cwf"
Jul 28 12:55:03.264675 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:03.264593    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4l68m\""
Jul 28 12:55:03.272206 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:03.272180    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-t4cwf"
Jul 28 12:55:03.402720 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:03.402691    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-t4cwf"]
Jul 28 12:55:03.405150 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:55:03.405129    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cb79c0b_58f8_414b_bc96_8f3c83f2f7a7.slice/crio-07ed83971a8b25c02c69574c51a76db75c8f044e342867de362edb5e9bb96503 WatchSource:0}: Error finding container 07ed83971a8b25c02c69574c51a76db75c8f044e342867de362edb5e9bb96503: Status 404 returned error can't find the container with id 07ed83971a8b25c02c69574c51a76db75c8f044e342867de362edb5e9bb96503
Jul 28 12:55:04.233743 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:04.233704    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-t4cwf" event={"ID":"0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7","Type":"ContainerStarted","Data":"07ed83971a8b25c02c69574c51a76db75c8f044e342867de362edb5e9bb96503"}
Jul 28 12:55:05.141408 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:05.141382    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-4l2q4"
Jul 28 12:55:05.238948 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:05.238916    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-t4cwf" event={"ID":"0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7","Type":"ContainerStarted","Data":"d36fec11f9245e694cc48ab9634f9d3c42fd98db3d68d1f59b8335b54c7e1a1d"}
Jul 28 12:55:06.243474 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:06.243443    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-t4cwf" event={"ID":"0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7","Type":"ContainerStarted","Data":"c7b182c256a069e2072ee358ef492150e06dcb433e3c7469fa6549476fc97ded"}
Jul 28 12:55:06.243842 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:06.243553    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-t4cwf"
Jul 28 12:55:06.262665 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:06.262620    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-t4cwf" podStartSLOduration=65.568583333 podStartE2EDuration="1m7.262605308s" totalImagesPullingTime="1.694021975s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:53:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:55:06.26136572 +0000 UTC m=+103.047241028" watchObservedRunningTime="2026-07-28 12:55:06.262605308 +0000 UTC m=+103.048480418"
Jul 28 12:55:07.438111 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:07.438083    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6dc5c5f66d-zjkn5"]
Jul 28 12:55:07.458963 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:07.458922    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6dc5c5f66d-zjkn5"]
Jul 28 12:55:07.459137 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:07.459051    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6dc5c5f66d-zjkn5"
Jul 28 12:55:07.463585 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:07.463559    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 28 12:55:07.463800 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:07.463571    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 28 12:55:07.463800 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:07.463559    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 28 12:55:07.463980 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:07.463633    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 28 12:55:07.464117 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:07.464102    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-sc944\""
Jul 28 12:55:07.512489 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:07.512464    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-dd79996f-p9sgt"]
Jul 28 12:55:07.529063 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:07.529034    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-dd79996f-p9sgt"
Jul 28 12:55:07.530609 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:07.530587    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-dd79996f-p9sgt"]
Jul 28 12:55:07.531598 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:07.531583    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 28 12:55:07.531675 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:07.531594    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 28 12:55:07.532356 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:07.532333    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6hlfc\""
Jul 28 12:55:07.533085 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:07.533062    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 28 12:55:07.540387 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:07.540367    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 28 12:55:07.603473 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:07.603444    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d85329d3-3a73-4ffa-9b7a-4aa122921a9a-trusted-ca\") pod \"image-registry-dd79996f-p9sgt\" (UID: \"d85329d3-3a73-4ffa-9b7a-4aa122921a9a\") " pod="openshift-image-registry/image-registry-dd79996f-p9sgt"
Jul 28 12:55:07.603630 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:07.603480    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d85329d3-3a73-4ffa-9b7a-4aa122921a9a-image-registry-private-configuration\") pod \"image-registry-dd79996f-p9sgt\" (UID: \"d85329d3-3a73-4ffa-9b7a-4aa122921a9a\") " pod="openshift-image-registry/image-registry-dd79996f-p9sgt"
Jul 28 12:55:07.603630 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:07.603499    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d85329d3-3a73-4ffa-9b7a-4aa122921a9a-installation-pull-secrets\") pod \"image-registry-dd79996f-p9sgt\" (UID: \"d85329d3-3a73-4ffa-9b7a-4aa122921a9a\") " pod="openshift-image-registry/image-registry-dd79996f-p9sgt"
Jul 28 12:55:07.603630 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:07.603560    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk5kg\" (UniqueName: \"kubernetes.io/projected/b4d058cc-758e-45db-9522-7bd5c9fc57c8-kube-api-access-qk5kg\") pod \"managed-serviceaccount-addon-agent-6dc5c5f66d-zjkn5\" (UID: \"b4d058cc-758e-45db-9522-7bd5c9fc57c8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6dc5c5f66d-zjkn5"
Jul 28 12:55:07.603630 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:07.603624    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b4d058cc-758e-45db-9522-7bd5c9fc57c8-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6dc5c5f66d-zjkn5\" (UID: \"b4d058cc-758e-45db-9522-7bd5c9fc57c8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6dc5c5f66d-zjkn5"
Jul 28 12:55:07.603823 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:07.603642    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d85329d3-3a73-4ffa-9b7a-4aa122921a9a-registry-certificates\") pod \"image-registry-dd79996f-p9sgt\" (UID: \"d85329d3-3a73-4ffa-9b7a-4aa122921a9a\") " pod="openshift-image-registry/image-registry-dd79996f-p9sgt"
Jul 28 12:55:07.603823 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:07.603662    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d85329d3-3a73-4ffa-9b7a-4aa122921a9a-ca-trust-extracted\") pod \"image-registry-dd79996f-p9sgt\" (UID: \"d85329d3-3a73-4ffa-9b7a-4aa122921a9a\") " pod="openshift-image-registry/image-registry-dd79996f-p9sgt"
Jul 28 12:55:07.603823 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:07.603678    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfpnc\" (UniqueName: \"kubernetes.io/projected/d85329d3-3a73-4ffa-9b7a-4aa122921a9a-kube-api-access-sfpnc\") pod \"image-registry-dd79996f-p9sgt\" (UID: \"d85329d3-3a73-4ffa-9b7a-4aa122921a9a\") " pod="openshift-image-registry/image-registry-dd79996f-p9sgt"
Jul 28 12:55:07.603823 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:07.603732    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d85329d3-3a73-4ffa-9b7a-4aa122921a9a-bound-sa-token\") pod \"image-registry-dd79996f-p9sgt\" (UID: \"d85329d3-3a73-4ffa-9b7a-4aa122921a9a\") " pod="openshift-image-registry/image-registry-dd79996f-p9sgt"
Jul 28 12:55:07.603823 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:07.603814    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d85329d3-3a73-4ffa-9b7a-4aa122921a9a-registry-tls\") pod \"image-registry-dd79996f-p9sgt\" (UID: \"d85329d3-3a73-4ffa-9b7a-4aa122921a9a\") " pod="openshift-image-registry/image-registry-dd79996f-p9sgt"
Jul 28 12:55:07.704574 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:07.704544    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk5kg\" (UniqueName: \"kubernetes.io/projected/b4d058cc-758e-45db-9522-7bd5c9fc57c8-kube-api-access-qk5kg\") pod \"managed-serviceaccount-addon-agent-6dc5c5f66d-zjkn5\" (UID: \"b4d058cc-758e-45db-9522-7bd5c9fc57c8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6dc5c5f66d-zjkn5"
Jul 28 12:55:07.704732 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:07.704578    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b4d058cc-758e-45db-9522-7bd5c9fc57c8-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6dc5c5f66d-zjkn5\" (UID: \"b4d058cc-758e-45db-9522-7bd5c9fc57c8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6dc5c5f66d-zjkn5"
Jul 28 12:55:07.704732 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:07.704597    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d85329d3-3a73-4ffa-9b7a-4aa122921a9a-registry-certificates\") pod \"image-registry-dd79996f-p9sgt\" (UID: \"d85329d3-3a73-4ffa-9b7a-4aa122921a9a\") " pod="openshift-image-registry/image-registry-dd79996f-p9sgt"
Jul 28 12:55:07.704732 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:07.704617    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d85329d3-3a73-4ffa-9b7a-4aa122921a9a-ca-trust-extracted\") pod \"image-registry-dd79996f-p9sgt\" (UID: \"d85329d3-3a73-4ffa-9b7a-4aa122921a9a\") " pod="openshift-image-registry/image-registry-dd79996f-p9sgt"
Jul 28 12:55:07.704929 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:07.704751    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfpnc\" (UniqueName: \"kubernetes.io/projected/d85329d3-3a73-4ffa-9b7a-4aa122921a9a-kube-api-access-sfpnc\") pod \"image-registry-dd79996f-p9sgt\" (UID: \"d85329d3-3a73-4ffa-9b7a-4aa122921a9a\") " pod="openshift-image-registry/image-registry-dd79996f-p9sgt"
Jul 28 12:55:07.704929 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:07.704791    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d85329d3-3a73-4ffa-9b7a-4aa122921a9a-bound-sa-token\") pod \"image-registry-dd79996f-p9sgt\" (UID: \"d85329d3-3a73-4ffa-9b7a-4aa122921a9a\") " pod="openshift-image-registry/image-registry-dd79996f-p9sgt"
Jul 28 12:55:07.705026 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:07.704934    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d85329d3-3a73-4ffa-9b7a-4aa122921a9a-registry-tls\") pod \"image-registry-dd79996f-p9sgt\" (UID: \"d85329d3-3a73-4ffa-9b7a-4aa122921a9a\") " pod="openshift-image-registry/image-registry-dd79996f-p9sgt"
Jul 28 12:55:07.705026 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:07.704975    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d85329d3-3a73-4ffa-9b7a-4aa122921a9a-ca-trust-extracted\") pod \"image-registry-dd79996f-p9sgt\" (UID: \"d85329d3-3a73-4ffa-9b7a-4aa122921a9a\") " pod="openshift-image-registry/image-registry-dd79996f-p9sgt"
Jul 28 12:55:07.705127 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:07.705026    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d85329d3-3a73-4ffa-9b7a-4aa122921a9a-trusted-ca\") pod \"image-registry-dd79996f-p9sgt\" (UID: \"d85329d3-3a73-4ffa-9b7a-4aa122921a9a\") " pod="openshift-image-registry/image-registry-dd79996f-p9sgt"
Jul 28 12:55:07.705127 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:07.705060    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d85329d3-3a73-4ffa-9b7a-4aa122921a9a-image-registry-private-configuration\") pod \"image-registry-dd79996f-p9sgt\" (UID: \"d85329d3-3a73-4ffa-9b7a-4aa122921a9a\") " pod="openshift-image-registry/image-registry-dd79996f-p9sgt"
Jul 28 12:55:07.705127 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:07.705085    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d85329d3-3a73-4ffa-9b7a-4aa122921a9a-installation-pull-secrets\") pod \"image-registry-dd79996f-p9sgt\" (UID: \"d85329d3-3a73-4ffa-9b7a-4aa122921a9a\") " pod="openshift-image-registry/image-registry-dd79996f-p9sgt"
Jul 28 12:55:07.705568 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:07.705539    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d85329d3-3a73-4ffa-9b7a-4aa122921a9a-registry-certificates\") pod \"image-registry-dd79996f-p9sgt\" (UID: \"d85329d3-3a73-4ffa-9b7a-4aa122921a9a\") " pod="openshift-image-registry/image-registry-dd79996f-p9sgt"
Jul 28 12:55:07.705828 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:07.705804    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d85329d3-3a73-4ffa-9b7a-4aa122921a9a-trusted-ca\") pod \"image-registry-dd79996f-p9sgt\" (UID: \"d85329d3-3a73-4ffa-9b7a-4aa122921a9a\") " pod="openshift-image-registry/image-registry-dd79996f-p9sgt"
Jul 28 12:55:07.707493 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:07.707472    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d85329d3-3a73-4ffa-9b7a-4aa122921a9a-image-registry-private-configuration\") pod \"image-registry-dd79996f-p9sgt\" (UID: \"d85329d3-3a73-4ffa-9b7a-4aa122921a9a\") " pod="openshift-image-registry/image-registry-dd79996f-p9sgt"
Jul 28 12:55:07.707673 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:07.707619    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d85329d3-3a73-4ffa-9b7a-4aa122921a9a-installation-pull-secrets\") pod \"image-registry-dd79996f-p9sgt\" (UID: \"d85329d3-3a73-4ffa-9b7a-4aa122921a9a\") " pod="openshift-image-registry/image-registry-dd79996f-p9sgt"
Jul 28 12:55:07.707910 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:07.707890    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b4d058cc-758e-45db-9522-7bd5c9fc57c8-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6dc5c5f66d-zjkn5\" (UID: \"b4d058cc-758e-45db-9522-7bd5c9fc57c8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6dc5c5f66d-zjkn5"
Jul 28 12:55:07.708067 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:07.708048    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d85329d3-3a73-4ffa-9b7a-4aa122921a9a-registry-tls\") pod \"image-registry-dd79996f-p9sgt\" (UID: \"d85329d3-3a73-4ffa-9b7a-4aa122921a9a\") " pod="openshift-image-registry/image-registry-dd79996f-p9sgt"
Jul 28 12:55:07.720756 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:07.720728    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d85329d3-3a73-4ffa-9b7a-4aa122921a9a-bound-sa-token\") pod \"image-registry-dd79996f-p9sgt\" (UID: \"d85329d3-3a73-4ffa-9b7a-4aa122921a9a\") " pod="openshift-image-registry/image-registry-dd79996f-p9sgt"
Jul 28 12:55:07.720849 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:07.720799    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfpnc\" (UniqueName: \"kubernetes.io/projected/d85329d3-3a73-4ffa-9b7a-4aa122921a9a-kube-api-access-sfpnc\") pod \"image-registry-dd79996f-p9sgt\" (UID: \"d85329d3-3a73-4ffa-9b7a-4aa122921a9a\") " pod="openshift-image-registry/image-registry-dd79996f-p9sgt"
Jul 28 12:55:07.721008 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:07.720989    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk5kg\" (UniqueName: \"kubernetes.io/projected/b4d058cc-758e-45db-9522-7bd5c9fc57c8-kube-api-access-qk5kg\") pod \"managed-serviceaccount-addon-agent-6dc5c5f66d-zjkn5\" (UID: \"b4d058cc-758e-45db-9522-7bd5c9fc57c8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6dc5c5f66d-zjkn5"
Jul 28 12:55:07.793261 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:07.793233    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6dc5c5f66d-zjkn5"
Jul 28 12:55:07.838992 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:07.838964    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-dd79996f-p9sgt"
Jul 28 12:55:07.960756 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:07.960684    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6dc5c5f66d-zjkn5"]
Jul 28 12:55:07.966560 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:55:07.966533    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb4d058cc_758e_45db_9522_7bd5c9fc57c8.slice/crio-5592c7d311207acd54866c87ba7757f16bfe7c8dc38a684de5f932c6ff619f31 WatchSource:0}: Error finding container 5592c7d311207acd54866c87ba7757f16bfe7c8dc38a684de5f932c6ff619f31: Status 404 returned error can't find the container with id 5592c7d311207acd54866c87ba7757f16bfe7c8dc38a684de5f932c6ff619f31
Jul 28 12:55:07.994256 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:07.994206    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-dd79996f-p9sgt"]
Jul 28 12:55:07.996578 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:55:07.996553    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd85329d3_3a73_4ffa_9b7a_4aa122921a9a.slice/crio-7f0a065ac2e400503703ba34c9793eafd45c004d988b884f733d3030fe5a3c22 WatchSource:0}: Error finding container 7f0a065ac2e400503703ba34c9793eafd45c004d988b884f733d3030fe5a3c22: Status 404 returned error can't find the container with id 7f0a065ac2e400503703ba34c9793eafd45c004d988b884f733d3030fe5a3c22
Jul 28 12:55:08.250573 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:08.250490    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6dc5c5f66d-zjkn5" event={"ID":"b4d058cc-758e-45db-9522-7bd5c9fc57c8","Type":"ContainerStarted","Data":"5592c7d311207acd54866c87ba7757f16bfe7c8dc38a684de5f932c6ff619f31"}
Jul 28 12:55:08.251979 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:08.251951    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-dd79996f-p9sgt" event={"ID":"d85329d3-3a73-4ffa-9b7a-4aa122921a9a","Type":"ContainerStarted","Data":"bf1ce58ddb95b50f69cb1d60f9eff692ebfa486e49e85b4d1f8f3e674e478db9"}
Jul 28 12:55:08.252106 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:08.251999    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-dd79996f-p9sgt" event={"ID":"d85329d3-3a73-4ffa-9b7a-4aa122921a9a","Type":"ContainerStarted","Data":"7f0a065ac2e400503703ba34c9793eafd45c004d988b884f733d3030fe5a3c22"}
Jul 28 12:55:08.252106 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:08.252069    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-dd79996f-p9sgt"
Jul 28 12:55:08.272245 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:08.272195    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-dd79996f-p9sgt" podStartSLOduration=1.272180728 podStartE2EDuration="1.272180728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:55:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:55:08.270957278 +0000 UTC m=+105.056832401" watchObservedRunningTime="2026-07-28 12:55:08.272180728 +0000 UTC m=+105.058055817"
Jul 28 12:55:10.833034 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:10.832995    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1a00af5e-6526-4eb0-951d-1ac554676d9e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-kc5t6\" (UID: \"1a00af5e-6526-4eb0-951d-1ac554676d9e\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-kc5t6"
Jul 28 12:55:10.835534 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:10.835512    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1a00af5e-6526-4eb0-951d-1ac554676d9e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-kc5t6\" (UID: \"1a00af5e-6526-4eb0-951d-1ac554676d9e\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-kc5t6"
Jul 28 12:55:11.133238 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:11.133191    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-kc5t6"
Jul 28 12:55:11.260730 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:11.260700    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-kc5t6"]
Jul 28 12:55:11.263289 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:55:11.263250    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a00af5e_6526_4eb0_951d_1ac554676d9e.slice/crio-932b952722f4d71ed63e49d34ae547090f32ef520cb6a52252e20e1f8c0fc8dc WatchSource:0}: Error finding container 932b952722f4d71ed63e49d34ae547090f32ef520cb6a52252e20e1f8c0fc8dc: Status 404 returned error can't find the container with id 932b952722f4d71ed63e49d34ae547090f32ef520cb6a52252e20e1f8c0fc8dc
Jul 28 12:55:11.268029 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:11.267952    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-kc5t6" event={"ID":"1a00af5e-6526-4eb0-951d-1ac554676d9e","Type":"ContainerStarted","Data":"932b952722f4d71ed63e49d34ae547090f32ef520cb6a52252e20e1f8c0fc8dc"}
Jul 28 12:55:11.269577 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:11.269553    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6dc5c5f66d-zjkn5" event={"ID":"b4d058cc-758e-45db-9522-7bd5c9fc57c8","Type":"ContainerStarted","Data":"11fc476c49f8447dd5250539dedfc213619600f2fee59851cb05f86880209cf5"}
Jul 28 12:55:11.286507 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:11.286464    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6dc5c5f66d-zjkn5" podStartSLOduration=1.632892067 podStartE2EDuration="4.286447041s" totalImagesPullingTime="2.653554974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:55:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:55:11.285177727 +0000 UTC m=+108.071052838" watchObservedRunningTime="2026-07-28 12:55:11.286447041 +0000 UTC m=+108.072322152"
Jul 28 12:55:13.277133 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:13.277097    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-kc5t6" event={"ID":"1a00af5e-6526-4eb0-951d-1ac554676d9e","Type":"ContainerStarted","Data":"e37c7f858c8e0ef208871e208d82d47f2110ef789e8fd77444439c2d36e8563a"}
Jul 28 12:55:13.295628 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:13.295581    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-kc5t6" podStartSLOduration=32.367637144 podStartE2EDuration="34.295565275s" totalImagesPullingTime="1.927928131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:54:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:55:13.294377885 +0000 UTC m=+110.080253020" watchObservedRunningTime="2026-07-28 12:55:13.295565275 +0000 UTC m=+110.081440385"
Jul 28 12:55:13.686240 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:13.686207    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-b5g48"]
Jul 28 12:55:13.690348 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:13.690333    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-b5g48"
Jul 28 12:55:13.692988 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:13.692965    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-f2lvx\""
Jul 28 12:55:13.693178 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:13.693156    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 28 12:55:13.698409 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:13.698106    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-b5g48"]
Jul 28 12:55:13.861169 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:13.861134    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/90416dc6-1ef1-4e61-91fa-f5aaaddc09e5-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-b5g48\" (UID: \"90416dc6-1ef1-4e61-91fa-f5aaaddc09e5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-b5g48"
Jul 28 12:55:13.961835 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:13.961811    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/90416dc6-1ef1-4e61-91fa-f5aaaddc09e5-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-b5g48\" (UID: \"90416dc6-1ef1-4e61-91fa-f5aaaddc09e5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-b5g48"
Jul 28 12:55:13.964384 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:13.964360    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/90416dc6-1ef1-4e61-91fa-f5aaaddc09e5-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-b5g48\" (UID: \"90416dc6-1ef1-4e61-91fa-f5aaaddc09e5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-b5g48"
Jul 28 12:55:14.001356 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:14.001330    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-b5g48"
Jul 28 12:55:14.133664 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:14.133635    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-b5g48"]
Jul 28 12:55:14.136444 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:55:14.136419    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90416dc6_1ef1_4e61_91fa_f5aaaddc09e5.slice/crio-0b28bdec4f224285e8e063af905cda4ae731d4a9cce16a9691a6b80705b91c4a WatchSource:0}: Error finding container 0b28bdec4f224285e8e063af905cda4ae731d4a9cce16a9691a6b80705b91c4a: Status 404 returned error can't find the container with id 0b28bdec4f224285e8e063af905cda4ae731d4a9cce16a9691a6b80705b91c4a
Jul 28 12:55:14.264172 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:14.264100    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fbd7f2cb-b409-4e32-854e-043f12d72d21-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rgh7t\" (UID: \"fbd7f2cb-b409-4e32-854e-043f12d72d21\") " pod="openshift-insights/insights-runtime-extractor-rgh7t"
Jul 28 12:55:14.266476 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:14.266458    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fbd7f2cb-b409-4e32-854e-043f12d72d21-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rgh7t\" (UID: \"fbd7f2cb-b409-4e32-854e-043f12d72d21\") " pod="openshift-insights/insights-runtime-extractor-rgh7t"
Jul 28 12:55:14.281843 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:14.281806    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-b5g48" event={"ID":"90416dc6-1ef1-4e61-91fa-f5aaaddc09e5","Type":"ContainerStarted","Data":"0b28bdec4f224285e8e063af905cda4ae731d4a9cce16a9691a6b80705b91c4a"}
Jul 28 12:55:14.509288 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:14.509251    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rgh7t"
Jul 28 12:55:14.650314 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:14.650289    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rgh7t"]
Jul 28 12:55:14.652832 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:55:14.652809    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbd7f2cb_b409_4e32_854e_043f12d72d21.slice/crio-46cb6de6f87ddfa9c7c16750ba3b9384056369f29aa43d3959f3678e864ee2d4 WatchSource:0}: Error finding container 46cb6de6f87ddfa9c7c16750ba3b9384056369f29aa43d3959f3678e864ee2d4: Status 404 returned error can't find the container with id 46cb6de6f87ddfa9c7c16750ba3b9384056369f29aa43d3959f3678e864ee2d4
Jul 28 12:55:15.285963 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:15.285933    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-b5g48" event={"ID":"90416dc6-1ef1-4e61-91fa-f5aaaddc09e5","Type":"ContainerStarted","Data":"935cc12b104aad50a3cf4961150df4974152a3b2745721927b1a1a76cc77eb13"}
Jul 28 12:55:15.286417 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:15.286118    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-b5g48"
Jul 28 12:55:15.287304 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:15.287265    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rgh7t" event={"ID":"fbd7f2cb-b409-4e32-854e-043f12d72d21","Type":"ContainerStarted","Data":"46cb6de6f87ddfa9c7c16750ba3b9384056369f29aa43d3959f3678e864ee2d4"}
Jul 28 12:55:15.290878 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:15.290860    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-b5g48"
Jul 28 12:55:15.302244 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:15.302200    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-b5g48" podStartSLOduration=1.240463007 podStartE2EDuration="2.302183366s" totalImagesPullingTime="1.061720359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:55:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:55:15.302051183 +0000 UTC m=+112.087926295" watchObservedRunningTime="2026-07-28 12:55:15.302183366 +0000 UTC m=+112.088058479"
Jul 28 12:55:16.248908 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:16.248876    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-t4cwf"
Jul 28 12:55:17.064741 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:17.064708    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-849d74bcb5-dh55v"]
Jul 28 12:55:17.067782 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:17.067761    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-849d74bcb5-dh55v"
Jul 28 12:55:17.073218 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:17.073197    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 28 12:55:17.073359 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:17.073223    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 28 12:55:17.073359 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:17.073310    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 28 12:55:17.074261 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:17.074243    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 28 12:55:17.074393 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:17.074242    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 28 12:55:17.075464 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:17.075433    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 28 12:55:17.076937 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:17.076918    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 28 12:55:17.082633 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:17.082609    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-8pbqd\""
Jul 28 12:55:17.085892 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:17.085874    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 28 12:55:17.089619 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:17.089588    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-849d74bcb5-dh55v"]
Jul 28 12:55:17.191359 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:17.191319    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a24d572-fb68-40bd-a1f7-df3c8dcccc03-trusted-ca-bundle\") pod \"console-849d74bcb5-dh55v\" (UID: \"4a24d572-fb68-40bd-a1f7-df3c8dcccc03\") " pod="openshift-console/console-849d74bcb5-dh55v"
Jul 28 12:55:17.191543 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:17.191380    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a24d572-fb68-40bd-a1f7-df3c8dcccc03-console-oauth-config\") pod \"console-849d74bcb5-dh55v\" (UID: \"4a24d572-fb68-40bd-a1f7-df3c8dcccc03\") " pod="openshift-console/console-849d74bcb5-dh55v"
Jul 28 12:55:17.191543 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:17.191413    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a24d572-fb68-40bd-a1f7-df3c8dcccc03-console-config\") pod \"console-849d74bcb5-dh55v\" (UID: \"4a24d572-fb68-40bd-a1f7-df3c8dcccc03\") " pod="openshift-console/console-849d74bcb5-dh55v"
Jul 28 12:55:17.191543 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:17.191475    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a24d572-fb68-40bd-a1f7-df3c8dcccc03-service-ca\") pod \"console-849d74bcb5-dh55v\" (UID: \"4a24d572-fb68-40bd-a1f7-df3c8dcccc03\") " pod="openshift-console/console-849d74bcb5-dh55v"
Jul 28 12:55:17.191543 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:17.191505    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a24d572-fb68-40bd-a1f7-df3c8dcccc03-oauth-serving-cert\") pod \"console-849d74bcb5-dh55v\" (UID: \"4a24d572-fb68-40bd-a1f7-df3c8dcccc03\") " pod="openshift-console/console-849d74bcb5-dh55v"
Jul 28 12:55:17.191699 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:17.191626    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a24d572-fb68-40bd-a1f7-df3c8dcccc03-console-serving-cert\") pod \"console-849d74bcb5-dh55v\" (UID: \"4a24d572-fb68-40bd-a1f7-df3c8dcccc03\") " pod="openshift-console/console-849d74bcb5-dh55v"
Jul 28 12:55:17.191699 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:17.191661    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75lfb\" (UniqueName: \"kubernetes.io/projected/4a24d572-fb68-40bd-a1f7-df3c8dcccc03-kube-api-access-75lfb\") pod \"console-849d74bcb5-dh55v\" (UID: \"4a24d572-fb68-40bd-a1f7-df3c8dcccc03\") " pod="openshift-console/console-849d74bcb5-dh55v"
Jul 28 12:55:17.292801 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:17.292775    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a24d572-fb68-40bd-a1f7-df3c8dcccc03-console-oauth-config\") pod \"console-849d74bcb5-dh55v\" (UID: \"4a24d572-fb68-40bd-a1f7-df3c8dcccc03\") " pod="openshift-console/console-849d74bcb5-dh55v"
Jul 28 12:55:17.292997 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:17.292809    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a24d572-fb68-40bd-a1f7-df3c8dcccc03-console-config\") pod \"console-849d74bcb5-dh55v\" (UID: \"4a24d572-fb68-40bd-a1f7-df3c8dcccc03\") " pod="openshift-console/console-849d74bcb5-dh55v"
Jul 28 12:55:17.293067 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:17.293001    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a24d572-fb68-40bd-a1f7-df3c8dcccc03-service-ca\") pod \"console-849d74bcb5-dh55v\" (UID: \"4a24d572-fb68-40bd-a1f7-df3c8dcccc03\") " pod="openshift-console/console-849d74bcb5-dh55v"
Jul 28 12:55:17.293067 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:17.293048    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a24d572-fb68-40bd-a1f7-df3c8dcccc03-oauth-serving-cert\") pod \"console-849d74bcb5-dh55v\" (UID: \"4a24d572-fb68-40bd-a1f7-df3c8dcccc03\") " pod="openshift-console/console-849d74bcb5-dh55v"
Jul 28 12:55:17.293173 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:17.293128    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a24d572-fb68-40bd-a1f7-df3c8dcccc03-console-serving-cert\") pod \"console-849d74bcb5-dh55v\" (UID: \"4a24d572-fb68-40bd-a1f7-df3c8dcccc03\") " pod="openshift-console/console-849d74bcb5-dh55v"
Jul 28 12:55:17.293173 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:17.293156    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75lfb\" (UniqueName: \"kubernetes.io/projected/4a24d572-fb68-40bd-a1f7-df3c8dcccc03-kube-api-access-75lfb\") pod \"console-849d74bcb5-dh55v\" (UID: \"4a24d572-fb68-40bd-a1f7-df3c8dcccc03\") " pod="openshift-console/console-849d74bcb5-dh55v"
Jul 28 12:55:17.293302 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:17.293187    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a24d572-fb68-40bd-a1f7-df3c8dcccc03-trusted-ca-bundle\") pod \"console-849d74bcb5-dh55v\" (UID: \"4a24d572-fb68-40bd-a1f7-df3c8dcccc03\") " pod="openshift-console/console-849d74bcb5-dh55v"
Jul 28 12:55:17.293618 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:17.293597    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a24d572-fb68-40bd-a1f7-df3c8dcccc03-console-config\") pod \"console-849d74bcb5-dh55v\" (UID: \"4a24d572-fb68-40bd-a1f7-df3c8dcccc03\") " pod="openshift-console/console-849d74bcb5-dh55v"
Jul 28 12:55:17.293924 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:17.293902    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a24d572-fb68-40bd-a1f7-df3c8dcccc03-oauth-serving-cert\") pod \"console-849d74bcb5-dh55v\" (UID: \"4a24d572-fb68-40bd-a1f7-df3c8dcccc03\") " pod="openshift-console/console-849d74bcb5-dh55v"
Jul 28 12:55:17.294581 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:17.294555    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a24d572-fb68-40bd-a1f7-df3c8dcccc03-service-ca\") pod \"console-849d74bcb5-dh55v\" (UID: \"4a24d572-fb68-40bd-a1f7-df3c8dcccc03\") " pod="openshift-console/console-849d74bcb5-dh55v"
Jul 28 12:55:17.294914 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:17.294896    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a24d572-fb68-40bd-a1f7-df3c8dcccc03-trusted-ca-bundle\") pod \"console-849d74bcb5-dh55v\" (UID: \"4a24d572-fb68-40bd-a1f7-df3c8dcccc03\") " pod="openshift-console/console-849d74bcb5-dh55v"
Jul 28 12:55:17.295984 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:17.295960    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a24d572-fb68-40bd-a1f7-df3c8dcccc03-console-serving-cert\") pod \"console-849d74bcb5-dh55v\" (UID: \"4a24d572-fb68-40bd-a1f7-df3c8dcccc03\") " pod="openshift-console/console-849d74bcb5-dh55v"
Jul 28 12:55:17.296073 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:17.296004    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a24d572-fb68-40bd-a1f7-df3c8dcccc03-console-oauth-config\") pod \"console-849d74bcb5-dh55v\" (UID: \"4a24d572-fb68-40bd-a1f7-df3c8dcccc03\") " pod="openshift-console/console-849d74bcb5-dh55v"
Jul 28 12:55:17.304824 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:17.304799    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75lfb\" (UniqueName: \"kubernetes.io/projected/4a24d572-fb68-40bd-a1f7-df3c8dcccc03-kube-api-access-75lfb\") pod \"console-849d74bcb5-dh55v\" (UID: \"4a24d572-fb68-40bd-a1f7-df3c8dcccc03\") " pod="openshift-console/console-849d74bcb5-dh55v"
Jul 28 12:55:17.378771 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:17.378690    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-849d74bcb5-dh55v"
Jul 28 12:55:17.541991 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:17.541963    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-849d74bcb5-dh55v"]
Jul 28 12:55:19.515712 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:55:19.515672    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a24d572_fb68_40bd_a1f7_df3c8dcccc03.slice/crio-fc0b633b169f581e1732ca12b0eb17d6788f766a5cf729e2c46ef72084cc4f9d WatchSource:0}: Error finding container fc0b633b169f581e1732ca12b0eb17d6788f766a5cf729e2c46ef72084cc4f9d: Status 404 returned error can't find the container with id fc0b633b169f581e1732ca12b0eb17d6788f766a5cf729e2c46ef72084cc4f9d
Jul 28 12:55:20.312512 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:20.312477    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-849d74bcb5-dh55v" event={"ID":"4a24d572-fb68-40bd-a1f7-df3c8dcccc03","Type":"ContainerStarted","Data":"fc0b633b169f581e1732ca12b0eb17d6788f766a5cf729e2c46ef72084cc4f9d"}
Jul 28 12:55:20.314726 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:20.314698    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rgh7t" event={"ID":"fbd7f2cb-b409-4e32-854e-043f12d72d21","Type":"ContainerStarted","Data":"76fd6fa75b23ee12474378ed30a794f65c37c4d9ab60c3bbf6e6c74dd71ad8df"}
Jul 28 12:55:25.333981 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:25.333883    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-849d74bcb5-dh55v" event={"ID":"4a24d572-fb68-40bd-a1f7-df3c8dcccc03","Type":"ContainerStarted","Data":"aca873a14e154e2a4b4b48b47a3c6c35889e584436c5371cd3bcc507f1b9db1d"}
Jul 28 12:55:25.335756 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:25.335720    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rgh7t" event={"ID":"fbd7f2cb-b409-4e32-854e-043f12d72d21","Type":"ContainerStarted","Data":"5d0051dc069753a6411fcb4697f7ca383543dbfcdf9c56fe5060ee66df7ce386"}
Jul 28 12:55:25.353234 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:25.353190    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-849d74bcb5-dh55v" podStartSLOduration=2.907853729 podStartE2EDuration="8.353176031s" totalImagesPullingTime="5.445322302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:55:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:55:25.352185494 +0000 UTC m=+122.138060607" watchObservedRunningTime="2026-07-28 12:55:25.353176031 +0000 UTC m=+122.139051119"
Jul 28 12:55:27.379120 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:27.379076    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-849d74bcb5-dh55v"
Jul 28 12:55:27.379593 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:27.379194    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-849d74bcb5-dh55v"
Jul 28 12:55:27.385297 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:27.385255    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-849d74bcb5-dh55v"
Jul 28 12:55:28.351580 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:28.351550    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-849d74bcb5-dh55v"
Jul 28 12:55:29.260077 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:29.260051    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-dd79996f-p9sgt"
Jul 28 12:55:29.354355 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:29.354319    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rgh7t" event={"ID":"fbd7f2cb-b409-4e32-854e-043f12d72d21","Type":"ContainerStarted","Data":"9fd35c85901e96e1e32e7eda2a4d6277ddc43e293738058eb978e67baefd268a"}
Jul 28 12:55:31.282950 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:31.282890    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-rgh7t" podStartSLOduration=35.529821746 podStartE2EDuration="49.282867595s" totalImagesPullingTime="13.753045849s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 12:54:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:55:29.373434546 +0000 UTC m=+126.159309657" watchObservedRunningTime="2026-07-28 12:55:31.282867595 +0000 UTC m=+128.068742707"
Jul 28 12:55:31.283463 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:31.283447    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-lwp6v"]
Jul 28 12:55:31.287170 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:31.287149    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lwp6v"
Jul 28 12:55:31.290340 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:31.290319    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 12:55:31.290503 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:31.290483    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-trl9r\""
Jul 28 12:55:31.290583 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:31.290483    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 12:55:31.290732 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:31.290705    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 12:55:31.295369 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:31.295352    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 12:55:31.312339 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:31.312315    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6-sys\") pod \"node-exporter-lwp6v\" (UID: \"b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6\") " pod="openshift-monitoring/node-exporter-lwp6v"
Jul 28 12:55:31.312436 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:31.312352    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6-node-exporter-tls\") pod \"node-exporter-lwp6v\" (UID: \"b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6\") " pod="openshift-monitoring/node-exporter-lwp6v"
Jul 28 12:55:31.312436 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:31.312380    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6-root\") pod \"node-exporter-lwp6v\" (UID: \"b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6\") " pod="openshift-monitoring/node-exporter-lwp6v"
Jul 28 12:55:31.312546 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:31.312437    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6-node-exporter-wtmp\") pod \"node-exporter-lwp6v\" (UID: \"b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6\") " pod="openshift-monitoring/node-exporter-lwp6v"
Jul 28 12:55:31.312546 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:31.312510    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6-node-exporter-accelerators-collector-config\") pod \"node-exporter-lwp6v\" (UID: \"b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6\") " pod="openshift-monitoring/node-exporter-lwp6v"
Jul 28 12:55:31.312636 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:31.312545    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rw5q\" (UniqueName: \"kubernetes.io/projected/b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6-kube-api-access-9rw5q\") pod \"node-exporter-lwp6v\" (UID: \"b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6\") " pod="openshift-monitoring/node-exporter-lwp6v"
Jul 28 12:55:31.312636 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:31.312588    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6-node-exporter-textfile\") pod \"node-exporter-lwp6v\" (UID: \"b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6\") " pod="openshift-monitoring/node-exporter-lwp6v"
Jul 28 12:55:31.312636 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:31.312614    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6-metrics-client-ca\") pod \"node-exporter-lwp6v\" (UID: \"b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6\") " pod="openshift-monitoring/node-exporter-lwp6v"
Jul 28 12:55:31.312770 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:31.312677    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lwp6v\" (UID: \"b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6\") " pod="openshift-monitoring/node-exporter-lwp6v"
Jul 28 12:55:31.413954 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:31.413921    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lwp6v\" (UID: \"b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6\") " pod="openshift-monitoring/node-exporter-lwp6v"
Jul 28 12:55:31.414134 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:31.413971    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6-sys\") pod \"node-exporter-lwp6v\" (UID: \"b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6\") " pod="openshift-monitoring/node-exporter-lwp6v"
Jul 28 12:55:31.414134 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:31.413999    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6-node-exporter-tls\") pod \"node-exporter-lwp6v\" (UID: \"b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6\") " pod="openshift-monitoring/node-exporter-lwp6v"
Jul 28 12:55:31.414134 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:31.414062    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6-sys\") pod \"node-exporter-lwp6v\" (UID: \"b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6\") " pod="openshift-monitoring/node-exporter-lwp6v"
Jul 28 12:55:31.414134 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:55:31.414101    2554 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 28 12:55:31.414134 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:31.414124    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6-root\") pod \"node-exporter-lwp6v\" (UID: \"b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6\") " pod="openshift-monitoring/node-exporter-lwp6v"
Jul 28 12:55:31.414401 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:31.414102    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6-root\") pod \"node-exporter-lwp6v\" (UID: \"b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6\") " pod="openshift-monitoring/node-exporter-lwp6v"
Jul 28 12:55:31.414401 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:55:31.414165    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6-node-exporter-tls podName:b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6 nodeName:}" failed. No retries permitted until 2026-07-28 12:55:31.914145571 +0000 UTC m=+128.700020660 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6-node-exporter-tls") pod "node-exporter-lwp6v" (UID: "b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6") : secret "node-exporter-tls" not found
Jul 28 12:55:31.414401 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:31.414206    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6-node-exporter-wtmp\") pod \"node-exporter-lwp6v\" (UID: \"b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6\") " pod="openshift-monitoring/node-exporter-lwp6v"
Jul 28 12:55:31.414401 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:31.414245    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6-node-exporter-accelerators-collector-config\") pod \"node-exporter-lwp6v\" (UID: \"b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6\") " pod="openshift-monitoring/node-exporter-lwp6v"
Jul 28 12:55:31.414401 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:31.414295    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rw5q\" (UniqueName: \"kubernetes.io/projected/b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6-kube-api-access-9rw5q\") pod \"node-exporter-lwp6v\" (UID: \"b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6\") " pod="openshift-monitoring/node-exporter-lwp6v"
Jul 28 12:55:31.414401 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:31.414328    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6-node-exporter-textfile\") pod \"node-exporter-lwp6v\" (UID: \"b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6\") " pod="openshift-monitoring/node-exporter-lwp6v"
Jul 28 12:55:31.414401 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:31.414355    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6-metrics-client-ca\") pod \"node-exporter-lwp6v\" (UID: \"b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6\") " pod="openshift-monitoring/node-exporter-lwp6v"
Jul 28 12:55:31.414756 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:31.414736    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6-node-exporter-wtmp\") pod \"node-exporter-lwp6v\" (UID: \"b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6\") " pod="openshift-monitoring/node-exporter-lwp6v"
Jul 28 12:55:31.415017 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:31.414995    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6-metrics-client-ca\") pod \"node-exporter-lwp6v\" (UID: \"b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6\") " pod="openshift-monitoring/node-exporter-lwp6v"
Jul 28 12:55:31.415101 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:31.415005    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6-node-exporter-accelerators-collector-config\") pod \"node-exporter-lwp6v\" (UID: \"b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6\") " pod="openshift-monitoring/node-exporter-lwp6v"
Jul 28 12:55:31.415146 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:31.415112    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6-node-exporter-textfile\") pod \"node-exporter-lwp6v\" (UID: \"b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6\") " pod="openshift-monitoring/node-exporter-lwp6v"
Jul 28 12:55:31.417012 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:31.416991    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lwp6v\" (UID: \"b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6\") " pod="openshift-monitoring/node-exporter-lwp6v"
Jul 28 12:55:31.459269 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:31.459233    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rw5q\" (UniqueName: \"kubernetes.io/projected/b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6-kube-api-access-9rw5q\") pod \"node-exporter-lwp6v\" (UID: \"b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6\") " pod="openshift-monitoring/node-exporter-lwp6v"
Jul 28 12:55:31.918890 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:31.918853    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6-node-exporter-tls\") pod \"node-exporter-lwp6v\" (UID: \"b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6\") " pod="openshift-monitoring/node-exporter-lwp6v"
Jul 28 12:55:31.921322 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:31.921243    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6-node-exporter-tls\") pod \"node-exporter-lwp6v\" (UID: \"b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6\") " pod="openshift-monitoring/node-exporter-lwp6v"
Jul 28 12:55:32.197762 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:32.197732    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lwp6v"
Jul 28 12:55:32.211741 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:55:32.211699    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0640ec3_f2c9_407d_80b2_c9e7b4b30aa6.slice/crio-14420c5808b3b83b14b418b4b4fdcf6715f282340038a31580d3ac39d81041ba WatchSource:0}: Error finding container 14420c5808b3b83b14b418b4b4fdcf6715f282340038a31580d3ac39d81041ba: Status 404 returned error can't find the container with id 14420c5808b3b83b14b418b4b4fdcf6715f282340038a31580d3ac39d81041ba
Jul 28 12:55:32.365412 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:32.365378    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lwp6v" event={"ID":"b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6","Type":"ContainerStarted","Data":"14420c5808b3b83b14b418b4b4fdcf6715f282340038a31580d3ac39d81041ba"}
Jul 28 12:55:32.524299 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:32.524214    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d04af0e5-3e6d-4027-8509-0af92acc4035-metrics-certs\") pod \"network-metrics-daemon-vvbch\" (UID: \"d04af0e5-3e6d-4027-8509-0af92acc4035\") " pod="openshift-multus/network-metrics-daemon-vvbch"
Jul 28 12:55:32.526736 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:32.526711    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d04af0e5-3e6d-4027-8509-0af92acc4035-metrics-certs\") pod \"network-metrics-daemon-vvbch\" (UID: \"d04af0e5-3e6d-4027-8509-0af92acc4035\") " pod="openshift-multus/network-metrics-daemon-vvbch"
Jul 28 12:55:32.826776 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:32.826699    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wvgpn\""
Jul 28 12:55:32.834004 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:32.833983    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vvbch"
Jul 28 12:55:32.965865 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:32.965833    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vvbch"]
Jul 28 12:55:32.969234 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:55:32.969207    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd04af0e5_3e6d_4027_8509_0af92acc4035.slice/crio-b8b021b18d0d2ae1eca28548ac8d31ff0911a3f89b54c773378549532deb1a5a WatchSource:0}: Error finding container b8b021b18d0d2ae1eca28548ac8d31ff0911a3f89b54c773378549532deb1a5a: Status 404 returned error can't find the container with id b8b021b18d0d2ae1eca28548ac8d31ff0911a3f89b54c773378549532deb1a5a
Jul 28 12:55:33.371675 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:33.371641    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vvbch" event={"ID":"d04af0e5-3e6d-4027-8509-0af92acc4035","Type":"ContainerStarted","Data":"b8b021b18d0d2ae1eca28548ac8d31ff0911a3f89b54c773378549532deb1a5a"}
Jul 28 12:55:34.294201 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:34.294150    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-665d6c9587-cjqhj"]
Jul 28 12:55:34.298410 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:34.298393    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-665d6c9587-cjqhj"
Jul 28 12:55:34.301423 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:34.301258    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-5d0l698og80b4\""
Jul 28 12:55:34.301423 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:34.301259    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 28 12:55:34.301573 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:34.301516    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 28 12:55:34.301573 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:34.301516    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 28 12:55:34.301573 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:34.301564    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-xzjg8\""
Jul 28 12:55:34.301829 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:34.301813    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 28 12:55:34.301829 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:34.301823    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 28 12:55:34.316137 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:34.316115    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-665d6c9587-cjqhj"]
Jul 28 12:55:34.338343 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:34.338322    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/66a5dfa8-fc4f-4fae-810f-3074802421a2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-665d6c9587-cjqhj\" (UID: \"66a5dfa8-fc4f-4fae-810f-3074802421a2\") " pod="openshift-monitoring/thanos-querier-665d6c9587-cjqhj"
Jul 28 12:55:34.338459 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:34.338365    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/66a5dfa8-fc4f-4fae-810f-3074802421a2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-665d6c9587-cjqhj\" (UID: \"66a5dfa8-fc4f-4fae-810f-3074802421a2\") " pod="openshift-monitoring/thanos-querier-665d6c9587-cjqhj"
Jul 28 12:55:34.338459 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:34.338391    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/66a5dfa8-fc4f-4fae-810f-3074802421a2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-665d6c9587-cjqhj\" (UID: \"66a5dfa8-fc4f-4fae-810f-3074802421a2\") " pod="openshift-monitoring/thanos-querier-665d6c9587-cjqhj"
Jul 28 12:55:34.338564 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:34.338481    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66a5dfa8-fc4f-4fae-810f-3074802421a2-metrics-client-ca\") pod \"thanos-querier-665d6c9587-cjqhj\" (UID: \"66a5dfa8-fc4f-4fae-810f-3074802421a2\") " pod="openshift-monitoring/thanos-querier-665d6c9587-cjqhj"
Jul 28 12:55:34.338564 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:34.338518    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/66a5dfa8-fc4f-4fae-810f-3074802421a2-secret-thanos-querier-tls\") pod \"thanos-querier-665d6c9587-cjqhj\" (UID: \"66a5dfa8-fc4f-4fae-810f-3074802421a2\") " pod="openshift-monitoring/thanos-querier-665d6c9587-cjqhj"
Jul 28 12:55:34.338564 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:34.338548    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/66a5dfa8-fc4f-4fae-810f-3074802421a2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-665d6c9587-cjqhj\" (UID: \"66a5dfa8-fc4f-4fae-810f-3074802421a2\") " pod="openshift-monitoring/thanos-querier-665d6c9587-cjqhj"
Jul 28 12:55:34.338708 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:34.338622    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/66a5dfa8-fc4f-4fae-810f-3074802421a2-secret-grpc-tls\") pod \"thanos-querier-665d6c9587-cjqhj\" (UID: \"66a5dfa8-fc4f-4fae-810f-3074802421a2\") " pod="openshift-monitoring/thanos-querier-665d6c9587-cjqhj"
Jul 28 12:55:34.338708 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:34.338656    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrcml\" (UniqueName: \"kubernetes.io/projected/66a5dfa8-fc4f-4fae-810f-3074802421a2-kube-api-access-rrcml\") pod \"thanos-querier-665d6c9587-cjqhj\" (UID: \"66a5dfa8-fc4f-4fae-810f-3074802421a2\") " pod="openshift-monitoring/thanos-querier-665d6c9587-cjqhj"
Jul 28 12:55:34.376853 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:34.376825    2554 generic.go:356] "Generic (PLEG): container finished" podID="b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6" containerID="febc04d35c85fa1e584706200df8784b0b2445dcb0285546f666a997bbc97f1f" exitCode=0
Jul 28 12:55:34.377211 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:34.376910    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lwp6v" event={"ID":"b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6","Type":"ContainerDied","Data":"febc04d35c85fa1e584706200df8784b0b2445dcb0285546f666a997bbc97f1f"}
Jul 28 12:55:34.378973 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:34.378952    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vvbch" event={"ID":"d04af0e5-3e6d-4027-8509-0af92acc4035","Type":"ContainerStarted","Data":"997dd53ae16f1abf7763bbd72e2028bb1d708ad575f1f87e7f1df7026d1c91ca"}
Jul 28 12:55:34.379072 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:34.378995    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vvbch" event={"ID":"d04af0e5-3e6d-4027-8509-0af92acc4035","Type":"ContainerStarted","Data":"66648c098e3e48d20120c15ffd65446a247e7735f2202320624b3014bdc22149"}
Jul 28 12:55:34.413147 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:34.413097    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vvbch" podStartSLOduration=129.565785747 podStartE2EDuration="2m10.41307909s" totalImagesPullingTime="847.293343ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:53:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:55:34.412818954 +0000 UTC m=+131.198694066" watchObservedRunningTime="2026-07-28 12:55:34.41307909 +0000 UTC m=+131.198954201"
Jul 28 12:55:34.439164 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:34.439065    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66a5dfa8-fc4f-4fae-810f-3074802421a2-metrics-client-ca\") pod \"thanos-querier-665d6c9587-cjqhj\" (UID: \"66a5dfa8-fc4f-4fae-810f-3074802421a2\") " pod="openshift-monitoring/thanos-querier-665d6c9587-cjqhj"
Jul 28 12:55:34.439164 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:34.439102    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/66a5dfa8-fc4f-4fae-810f-3074802421a2-secret-thanos-querier-tls\") pod \"thanos-querier-665d6c9587-cjqhj\" (UID: \"66a5dfa8-fc4f-4fae-810f-3074802421a2\") " pod="openshift-monitoring/thanos-querier-665d6c9587-cjqhj"
Jul 28 12:55:34.439164 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:34.439130    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/66a5dfa8-fc4f-4fae-810f-3074802421a2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-665d6c9587-cjqhj\" (UID: \"66a5dfa8-fc4f-4fae-810f-3074802421a2\") " pod="openshift-monitoring/thanos-querier-665d6c9587-cjqhj"
Jul 28 12:55:34.439164 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:34.439165    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/66a5dfa8-fc4f-4fae-810f-3074802421a2-secret-grpc-tls\") pod \"thanos-querier-665d6c9587-cjqhj\" (UID: \"66a5dfa8-fc4f-4fae-810f-3074802421a2\") " pod="openshift-monitoring/thanos-querier-665d6c9587-cjqhj"
Jul 28 12:55:34.439400 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:34.439187    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrcml\" (UniqueName: \"kubernetes.io/projected/66a5dfa8-fc4f-4fae-810f-3074802421a2-kube-api-access-rrcml\") pod \"thanos-querier-665d6c9587-cjqhj\" (UID: \"66a5dfa8-fc4f-4fae-810f-3074802421a2\") " pod="openshift-monitoring/thanos-querier-665d6c9587-cjqhj"
Jul 28 12:55:34.439400 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:34.439250    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/66a5dfa8-fc4f-4fae-810f-3074802421a2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-665d6c9587-cjqhj\" (UID: \"66a5dfa8-fc4f-4fae-810f-3074802421a2\") " pod="openshift-monitoring/thanos-querier-665d6c9587-cjqhj"
Jul 28 12:55:34.439400 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:34.439327    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/66a5dfa8-fc4f-4fae-810f-3074802421a2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-665d6c9587-cjqhj\" (UID: \"66a5dfa8-fc4f-4fae-810f-3074802421a2\") " pod="openshift-monitoring/thanos-querier-665d6c9587-cjqhj"
Jul 28 12:55:34.439400 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:34.439354    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/66a5dfa8-fc4f-4fae-810f-3074802421a2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-665d6c9587-cjqhj\" (UID: \"66a5dfa8-fc4f-4fae-810f-3074802421a2\") " pod="openshift-monitoring/thanos-querier-665d6c9587-cjqhj"
Jul 28 12:55:34.440600 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:34.440010    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66a5dfa8-fc4f-4fae-810f-3074802421a2-metrics-client-ca\") pod \"thanos-querier-665d6c9587-cjqhj\" (UID: \"66a5dfa8-fc4f-4fae-810f-3074802421a2\") " pod="openshift-monitoring/thanos-querier-665d6c9587-cjqhj"
Jul 28 12:55:34.442343 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:34.442317    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/66a5dfa8-fc4f-4fae-810f-3074802421a2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-665d6c9587-cjqhj\" (UID: \"66a5dfa8-fc4f-4fae-810f-3074802421a2\") " pod="openshift-monitoring/thanos-querier-665d6c9587-cjqhj"
Jul 28 12:55:34.442894 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:34.442867    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/66a5dfa8-fc4f-4fae-810f-3074802421a2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-665d6c9587-cjqhj\" (UID: \"66a5dfa8-fc4f-4fae-810f-3074802421a2\") " pod="openshift-monitoring/thanos-querier-665d6c9587-cjqhj"
Jul 28 12:55:34.443013 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:34.442985    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/66a5dfa8-fc4f-4fae-810f-3074802421a2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-665d6c9587-cjqhj\" (UID: \"66a5dfa8-fc4f-4fae-810f-3074802421a2\") " pod="openshift-monitoring/thanos-querier-665d6c9587-cjqhj"
Jul 28 12:55:34.443311 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:34.443239    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/66a5dfa8-fc4f-4fae-810f-3074802421a2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-665d6c9587-cjqhj\" (UID: \"66a5dfa8-fc4f-4fae-810f-3074802421a2\") " pod="openshift-monitoring/thanos-querier-665d6c9587-cjqhj"
Jul 28 12:55:34.443311 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:34.443291    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/66a5dfa8-fc4f-4fae-810f-3074802421a2-secret-grpc-tls\") pod \"thanos-querier-665d6c9587-cjqhj\" (UID: \"66a5dfa8-fc4f-4fae-810f-3074802421a2\") " pod="openshift-monitoring/thanos-querier-665d6c9587-cjqhj"
Jul 28 12:55:34.443631 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:34.443613    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/66a5dfa8-fc4f-4fae-810f-3074802421a2-secret-thanos-querier-tls\") pod \"thanos-querier-665d6c9587-cjqhj\" (UID: \"66a5dfa8-fc4f-4fae-810f-3074802421a2\") " pod="openshift-monitoring/thanos-querier-665d6c9587-cjqhj"
Jul 28 12:55:34.452952 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:34.452927    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrcml\" (UniqueName: \"kubernetes.io/projected/66a5dfa8-fc4f-4fae-810f-3074802421a2-kube-api-access-rrcml\") pod \"thanos-querier-665d6c9587-cjqhj\" (UID: \"66a5dfa8-fc4f-4fae-810f-3074802421a2\") " pod="openshift-monitoring/thanos-querier-665d6c9587-cjqhj"
Jul 28 12:55:34.609619 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:34.609536    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-665d6c9587-cjqhj"
Jul 28 12:55:34.745870 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:34.745686    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-665d6c9587-cjqhj"]
Jul 28 12:55:34.748437 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:55:34.748411    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66a5dfa8_fc4f_4fae_810f_3074802421a2.slice/crio-b3c4454b35bfe36e77ff48ac34e8018ad6eb0d21cd88603e2126ee57a59bca88 WatchSource:0}: Error finding container b3c4454b35bfe36e77ff48ac34e8018ad6eb0d21cd88603e2126ee57a59bca88: Status 404 returned error can't find the container with id b3c4454b35bfe36e77ff48ac34e8018ad6eb0d21cd88603e2126ee57a59bca88
Jul 28 12:55:35.383065 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:35.383028    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-665d6c9587-cjqhj" event={"ID":"66a5dfa8-fc4f-4fae-810f-3074802421a2","Type":"ContainerStarted","Data":"b3c4454b35bfe36e77ff48ac34e8018ad6eb0d21cd88603e2126ee57a59bca88"}
Jul 28 12:55:35.385004 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:35.384965    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lwp6v" event={"ID":"b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6","Type":"ContainerStarted","Data":"6bc1133ded6b5677c293a476569c47d7d36b7c9eaa637cd74a4bb4fe06d101e4"}
Jul 28 12:55:35.385118 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:35.385029    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lwp6v" event={"ID":"b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6","Type":"ContainerStarted","Data":"883ad8bb437ae1737cb489567ba73cf6eace98a796daadc87d5ef9fe04f4ae00"}
Jul 28 12:55:35.405418 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:35.405374    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-lwp6v" podStartSLOduration=2.7203995130000003 podStartE2EDuration="4.405360841s" totalImagesPullingTime="1.601997776s" totalInitContainerRuntime="82.963552ms" isStatefulPod=true podCreationTimestamp="2026-07-28 12:55:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:55:35.404263142 +0000 UTC m=+132.190138276" watchObservedRunningTime="2026-07-28 12:55:35.405360841 +0000 UTC m=+132.191235951"
Jul 28 12:55:36.653066 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:36.653033    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7574496c8c-kpnd9"]
Jul 28 12:55:36.657711 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:36.657675    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7574496c8c-kpnd9"
Jul 28 12:55:36.666731 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:36.666485    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7574496c8c-kpnd9"]
Jul 28 12:55:36.758854 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:36.758820    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sq5f\" (UniqueName: \"kubernetes.io/projected/2d6443ca-792d-4463-b68f-2f9dfa14b76e-kube-api-access-7sq5f\") pod \"console-7574496c8c-kpnd9\" (UID: \"2d6443ca-792d-4463-b68f-2f9dfa14b76e\") " pod="openshift-console/console-7574496c8c-kpnd9"
Jul 28 12:55:36.759003 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:36.758882    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2d6443ca-792d-4463-b68f-2f9dfa14b76e-service-ca\") pod \"console-7574496c8c-kpnd9\" (UID: \"2d6443ca-792d-4463-b68f-2f9dfa14b76e\") " pod="openshift-console/console-7574496c8c-kpnd9"
Jul 28 12:55:36.759003 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:36.758939    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2d6443ca-792d-4463-b68f-2f9dfa14b76e-console-oauth-config\") pod \"console-7574496c8c-kpnd9\" (UID: \"2d6443ca-792d-4463-b68f-2f9dfa14b76e\") " pod="openshift-console/console-7574496c8c-kpnd9"
Jul 28 12:55:36.759003 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:36.758955    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2d6443ca-792d-4463-b68f-2f9dfa14b76e-oauth-serving-cert\") pod \"console-7574496c8c-kpnd9\" (UID: \"2d6443ca-792d-4463-b68f-2f9dfa14b76e\") " pod="openshift-console/console-7574496c8c-kpnd9"
Jul 28 12:55:36.759003 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:36.758972    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d6443ca-792d-4463-b68f-2f9dfa14b76e-trusted-ca-bundle\") pod \"console-7574496c8c-kpnd9\" (UID: \"2d6443ca-792d-4463-b68f-2f9dfa14b76e\") " pod="openshift-console/console-7574496c8c-kpnd9"
Jul 28 12:55:36.759003 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:36.758993    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2d6443ca-792d-4463-b68f-2f9dfa14b76e-console-config\") pod \"console-7574496c8c-kpnd9\" (UID: \"2d6443ca-792d-4463-b68f-2f9dfa14b76e\") " pod="openshift-console/console-7574496c8c-kpnd9"
Jul 28 12:55:36.759151 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:36.759017    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2d6443ca-792d-4463-b68f-2f9dfa14b76e-console-serving-cert\") pod \"console-7574496c8c-kpnd9\" (UID: \"2d6443ca-792d-4463-b68f-2f9dfa14b76e\") " pod="openshift-console/console-7574496c8c-kpnd9"
Jul 28 12:55:36.860370 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:36.860334    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2d6443ca-792d-4463-b68f-2f9dfa14b76e-service-ca\") pod \"console-7574496c8c-kpnd9\" (UID: \"2d6443ca-792d-4463-b68f-2f9dfa14b76e\") " pod="openshift-console/console-7574496c8c-kpnd9"
Jul 28 12:55:36.860496 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:36.860421    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2d6443ca-792d-4463-b68f-2f9dfa14b76e-console-oauth-config\") pod \"console-7574496c8c-kpnd9\" (UID: \"2d6443ca-792d-4463-b68f-2f9dfa14b76e\") " pod="openshift-console/console-7574496c8c-kpnd9"
Jul 28 12:55:36.860496 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:36.860451    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2d6443ca-792d-4463-b68f-2f9dfa14b76e-oauth-serving-cert\") pod \"console-7574496c8c-kpnd9\" (UID: \"2d6443ca-792d-4463-b68f-2f9dfa14b76e\") " pod="openshift-console/console-7574496c8c-kpnd9"
Jul 28 12:55:36.860496 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:36.860480    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d6443ca-792d-4463-b68f-2f9dfa14b76e-trusted-ca-bundle\") pod \"console-7574496c8c-kpnd9\" (UID: \"2d6443ca-792d-4463-b68f-2f9dfa14b76e\") " pod="openshift-console/console-7574496c8c-kpnd9"
Jul 28 12:55:36.860664 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:36.860514    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2d6443ca-792d-4463-b68f-2f9dfa14b76e-console-config\") pod \"console-7574496c8c-kpnd9\" (UID: \"2d6443ca-792d-4463-b68f-2f9dfa14b76e\") " pod="openshift-console/console-7574496c8c-kpnd9"
Jul 28 12:55:36.860664 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:36.860549    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2d6443ca-792d-4463-b68f-2f9dfa14b76e-console-serving-cert\") pod \"console-7574496c8c-kpnd9\" (UID: \"2d6443ca-792d-4463-b68f-2f9dfa14b76e\") " pod="openshift-console/console-7574496c8c-kpnd9"
Jul 28 12:55:36.860664 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:36.860591    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sq5f\" (UniqueName: \"kubernetes.io/projected/2d6443ca-792d-4463-b68f-2f9dfa14b76e-kube-api-access-7sq5f\") pod \"console-7574496c8c-kpnd9\" (UID: \"2d6443ca-792d-4463-b68f-2f9dfa14b76e\") " pod="openshift-console/console-7574496c8c-kpnd9"
Jul 28 12:55:36.861246 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:36.861197    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2d6443ca-792d-4463-b68f-2f9dfa14b76e-service-ca\") pod \"console-7574496c8c-kpnd9\" (UID: \"2d6443ca-792d-4463-b68f-2f9dfa14b76e\") " pod="openshift-console/console-7574496c8c-kpnd9"
Jul 28 12:55:36.861368 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:36.861262    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2d6443ca-792d-4463-b68f-2f9dfa14b76e-oauth-serving-cert\") pod \"console-7574496c8c-kpnd9\" (UID: \"2d6443ca-792d-4463-b68f-2f9dfa14b76e\") " pod="openshift-console/console-7574496c8c-kpnd9"
Jul 28 12:55:36.861818 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:36.861788    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d6443ca-792d-4463-b68f-2f9dfa14b76e-trusted-ca-bundle\") pod \"console-7574496c8c-kpnd9\" (UID: \"2d6443ca-792d-4463-b68f-2f9dfa14b76e\") " pod="openshift-console/console-7574496c8c-kpnd9"
Jul 28 12:55:36.861905 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:36.861858    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2d6443ca-792d-4463-b68f-2f9dfa14b76e-console-config\") pod \"console-7574496c8c-kpnd9\" (UID: \"2d6443ca-792d-4463-b68f-2f9dfa14b76e\") " pod="openshift-console/console-7574496c8c-kpnd9"
Jul 28 12:55:36.863112 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:36.863091    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2d6443ca-792d-4463-b68f-2f9dfa14b76e-console-oauth-config\") pod \"console-7574496c8c-kpnd9\" (UID: \"2d6443ca-792d-4463-b68f-2f9dfa14b76e\") " pod="openshift-console/console-7574496c8c-kpnd9"
Jul 28 12:55:36.863611 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:36.863584    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2d6443ca-792d-4463-b68f-2f9dfa14b76e-console-serving-cert\") pod \"console-7574496c8c-kpnd9\" (UID: \"2d6443ca-792d-4463-b68f-2f9dfa14b76e\") " pod="openshift-console/console-7574496c8c-kpnd9"
Jul 28 12:55:36.868561 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:36.868539    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sq5f\" (UniqueName: \"kubernetes.io/projected/2d6443ca-792d-4463-b68f-2f9dfa14b76e-kube-api-access-7sq5f\") pod \"console-7574496c8c-kpnd9\" (UID: \"2d6443ca-792d-4463-b68f-2f9dfa14b76e\") " pod="openshift-console/console-7574496c8c-kpnd9"
Jul 28 12:55:36.971805 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:36.971777    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7574496c8c-kpnd9"
Jul 28 12:55:37.096990 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:37.096962    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7574496c8c-kpnd9"]
Jul 28 12:55:37.099724 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:55:37.099698    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d6443ca_792d_4463_b68f_2f9dfa14b76e.slice/crio-05c050fb4b8c36cb517b2e11db0846ef169dac952f45e8826da65cae268facc1 WatchSource:0}: Error finding container 05c050fb4b8c36cb517b2e11db0846ef169dac952f45e8826da65cae268facc1: Status 404 returned error can't find the container with id 05c050fb4b8c36cb517b2e11db0846ef169dac952f45e8826da65cae268facc1
Jul 28 12:55:37.395031 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:37.394990    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-665d6c9587-cjqhj" event={"ID":"66a5dfa8-fc4f-4fae-810f-3074802421a2","Type":"ContainerStarted","Data":"78c3c7d9f8c16c39e9ddb411ffd0aec050b29dbcd7115254cf5092593678be86"}
Jul 28 12:55:37.395172 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:37.395085    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-665d6c9587-cjqhj" event={"ID":"66a5dfa8-fc4f-4fae-810f-3074802421a2","Type":"ContainerStarted","Data":"b7cef5adbb67fd78d00c3b3b87bfec045bea843976fb74916fd299e63d5d85f8"}
Jul 28 12:55:37.395172 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:37.395137    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-665d6c9587-cjqhj" event={"ID":"66a5dfa8-fc4f-4fae-810f-3074802421a2","Type":"ContainerStarted","Data":"3607a23607928b510830d690012f5435872f64df0ee24cdac6f20a1de5c58721"}
Jul 28 12:55:37.396514 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:37.396483    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7574496c8c-kpnd9" event={"ID":"2d6443ca-792d-4463-b68f-2f9dfa14b76e","Type":"ContainerStarted","Data":"32cf7c98e3b0cabd4cf986edee5a7d9cb445bd48e8ce9552908e251d5e622272"}
Jul 28 12:55:37.396613 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:37.396523    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7574496c8c-kpnd9" event={"ID":"2d6443ca-792d-4463-b68f-2f9dfa14b76e","Type":"ContainerStarted","Data":"05c050fb4b8c36cb517b2e11db0846ef169dac952f45e8826da65cae268facc1"}
Jul 28 12:55:37.414983 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:37.414924    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7574496c8c-kpnd9" podStartSLOduration=1.41490867 podStartE2EDuration="1.41490867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:55:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:55:37.414033044 +0000 UTC m=+134.199908165" watchObservedRunningTime="2026-07-28 12:55:37.41490867 +0000 UTC m=+134.200783781"
Jul 28 12:55:38.403348 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:38.403312    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-665d6c9587-cjqhj" event={"ID":"66a5dfa8-fc4f-4fae-810f-3074802421a2","Type":"ContainerStarted","Data":"f5767b52449a0676dac66e0396ca8d36b73a54a2022d64137db783e20f680fec"}
Jul 28 12:55:38.403704 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:38.403376    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-665d6c9587-cjqhj" event={"ID":"66a5dfa8-fc4f-4fae-810f-3074802421a2","Type":"ContainerStarted","Data":"53ffa8b0f5f82106d10e660ef6bc2612c578030b57bb7188b4aea7940996b965"}
Jul 28 12:55:38.403704 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:38.403402    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-665d6c9587-cjqhj" event={"ID":"66a5dfa8-fc4f-4fae-810f-3074802421a2","Type":"ContainerStarted","Data":"8c17a5ab2aebe7a613db5588983b9aa90b943f40e90457937c29ba009f6a9540"}
Jul 28 12:55:38.436708 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:38.436654    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-665d6c9587-cjqhj" podStartSLOduration=1.566532214 podStartE2EDuration="4.436637094s" totalImagesPullingTime="2.87010488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:55:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:55:38.435313989 +0000 UTC m=+135.221189101" watchObservedRunningTime="2026-07-28 12:55:38.436637094 +0000 UTC m=+135.222512204"
Jul 28 12:55:39.407989 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:39.407952    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-665d6c9587-cjqhj"
Jul 28 12:55:40.430043 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:40.430000    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7574496c8c-kpnd9"]
Jul 28 12:55:40.462975 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:40.462948    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5b86d6c456-6wvlj"]
Jul 28 12:55:40.466141 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:40.466128    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b86d6c456-6wvlj"
Jul 28 12:55:40.480163 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:40.480136    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b86d6c456-6wvlj"]
Jul 28 12:55:40.594866 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:40.594831    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ad00d630-0ca5-455d-a6a3-91aeaf89694f-console-serving-cert\") pod \"console-5b86d6c456-6wvlj\" (UID: \"ad00d630-0ca5-455d-a6a3-91aeaf89694f\") " pod="openshift-console/console-5b86d6c456-6wvlj"
Jul 28 12:55:40.594866 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:40.594865    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ad00d630-0ca5-455d-a6a3-91aeaf89694f-console-config\") pod \"console-5b86d6c456-6wvlj\" (UID: \"ad00d630-0ca5-455d-a6a3-91aeaf89694f\") " pod="openshift-console/console-5b86d6c456-6wvlj"
Jul 28 12:55:40.595073 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:40.594890    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ad00d630-0ca5-455d-a6a3-91aeaf89694f-console-oauth-config\") pod \"console-5b86d6c456-6wvlj\" (UID: \"ad00d630-0ca5-455d-a6a3-91aeaf89694f\") " pod="openshift-console/console-5b86d6c456-6wvlj"
Jul 28 12:55:40.595073 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:40.594911    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ad00d630-0ca5-455d-a6a3-91aeaf89694f-oauth-serving-cert\") pod \"console-5b86d6c456-6wvlj\" (UID: \"ad00d630-0ca5-455d-a6a3-91aeaf89694f\") " pod="openshift-console/console-5b86d6c456-6wvlj"
Jul 28 12:55:40.595073 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:40.594960    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ad00d630-0ca5-455d-a6a3-91aeaf89694f-service-ca\") pod \"console-5b86d6c456-6wvlj\" (UID: \"ad00d630-0ca5-455d-a6a3-91aeaf89694f\") " pod="openshift-console/console-5b86d6c456-6wvlj"
Jul 28 12:55:40.595073 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:40.594976    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad00d630-0ca5-455d-a6a3-91aeaf89694f-trusted-ca-bundle\") pod \"console-5b86d6c456-6wvlj\" (UID: \"ad00d630-0ca5-455d-a6a3-91aeaf89694f\") " pod="openshift-console/console-5b86d6c456-6wvlj"
Jul 28 12:55:40.595073 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:40.594991    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gqmq\" (UniqueName: \"kubernetes.io/projected/ad00d630-0ca5-455d-a6a3-91aeaf89694f-kube-api-access-8gqmq\") pod \"console-5b86d6c456-6wvlj\" (UID: \"ad00d630-0ca5-455d-a6a3-91aeaf89694f\") " pod="openshift-console/console-5b86d6c456-6wvlj"
Jul 28 12:55:40.696197 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:40.696173    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ad00d630-0ca5-455d-a6a3-91aeaf89694f-console-serving-cert\") pod \"console-5b86d6c456-6wvlj\" (UID: \"ad00d630-0ca5-455d-a6a3-91aeaf89694f\") " pod="openshift-console/console-5b86d6c456-6wvlj"
Jul 28 12:55:40.696390 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:40.696207    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ad00d630-0ca5-455d-a6a3-91aeaf89694f-console-config\") pod \"console-5b86d6c456-6wvlj\" (UID: \"ad00d630-0ca5-455d-a6a3-91aeaf89694f\") " pod="openshift-console/console-5b86d6c456-6wvlj"
Jul 28 12:55:40.696390 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:40.696233    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ad00d630-0ca5-455d-a6a3-91aeaf89694f-console-oauth-config\") pod \"console-5b86d6c456-6wvlj\" (UID: \"ad00d630-0ca5-455d-a6a3-91aeaf89694f\") " pod="openshift-console/console-5b86d6c456-6wvlj"
Jul 28 12:55:40.696390 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:40.696254    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ad00d630-0ca5-455d-a6a3-91aeaf89694f-oauth-serving-cert\") pod \"console-5b86d6c456-6wvlj\" (UID: \"ad00d630-0ca5-455d-a6a3-91aeaf89694f\") " pod="openshift-console/console-5b86d6c456-6wvlj"
Jul 28 12:55:40.696523 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:40.696458    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ad00d630-0ca5-455d-a6a3-91aeaf89694f-service-ca\") pod \"console-5b86d6c456-6wvlj\" (UID: \"ad00d630-0ca5-455d-a6a3-91aeaf89694f\") " pod="openshift-console/console-5b86d6c456-6wvlj"
Jul 28 12:55:40.696523 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:40.696505    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad00d630-0ca5-455d-a6a3-91aeaf89694f-trusted-ca-bundle\") pod \"console-5b86d6c456-6wvlj\" (UID: \"ad00d630-0ca5-455d-a6a3-91aeaf89694f\") " pod="openshift-console/console-5b86d6c456-6wvlj"
Jul 28 12:55:40.696628 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:40.696534    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gqmq\" (UniqueName: \"kubernetes.io/projected/ad00d630-0ca5-455d-a6a3-91aeaf89694f-kube-api-access-8gqmq\") pod \"console-5b86d6c456-6wvlj\" (UID: \"ad00d630-0ca5-455d-a6a3-91aeaf89694f\") " pod="openshift-console/console-5b86d6c456-6wvlj"
Jul 28 12:55:40.697005 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:40.696980    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ad00d630-0ca5-455d-a6a3-91aeaf89694f-oauth-serving-cert\") pod \"console-5b86d6c456-6wvlj\" (UID: \"ad00d630-0ca5-455d-a6a3-91aeaf89694f\") " pod="openshift-console/console-5b86d6c456-6wvlj"
Jul 28 12:55:40.697351 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:40.697178    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ad00d630-0ca5-455d-a6a3-91aeaf89694f-console-config\") pod \"console-5b86d6c456-6wvlj\" (UID: \"ad00d630-0ca5-455d-a6a3-91aeaf89694f\") " pod="openshift-console/console-5b86d6c456-6wvlj"
Jul 28 12:55:40.697470 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:40.697443    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ad00d630-0ca5-455d-a6a3-91aeaf89694f-service-ca\") pod \"console-5b86d6c456-6wvlj\" (UID: \"ad00d630-0ca5-455d-a6a3-91aeaf89694f\") " pod="openshift-console/console-5b86d6c456-6wvlj"
Jul 28 12:55:40.697536 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:40.697495    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad00d630-0ca5-455d-a6a3-91aeaf89694f-trusted-ca-bundle\") pod \"console-5b86d6c456-6wvlj\" (UID: \"ad00d630-0ca5-455d-a6a3-91aeaf89694f\") " pod="openshift-console/console-5b86d6c456-6wvlj"
Jul 28 12:55:40.698931 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:40.698909    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ad00d630-0ca5-455d-a6a3-91aeaf89694f-console-serving-cert\") pod \"console-5b86d6c456-6wvlj\" (UID: \"ad00d630-0ca5-455d-a6a3-91aeaf89694f\") " pod="openshift-console/console-5b86d6c456-6wvlj"
Jul 28 12:55:40.699042 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:40.699023    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ad00d630-0ca5-455d-a6a3-91aeaf89694f-console-oauth-config\") pod \"console-5b86d6c456-6wvlj\" (UID: \"ad00d630-0ca5-455d-a6a3-91aeaf89694f\") " pod="openshift-console/console-5b86d6c456-6wvlj"
Jul 28 12:55:40.705747 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:40.705725    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gqmq\" (UniqueName: \"kubernetes.io/projected/ad00d630-0ca5-455d-a6a3-91aeaf89694f-kube-api-access-8gqmq\") pod \"console-5b86d6c456-6wvlj\" (UID: \"ad00d630-0ca5-455d-a6a3-91aeaf89694f\") " pod="openshift-console/console-5b86d6c456-6wvlj"
Jul 28 12:55:40.775068 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:40.775035    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b86d6c456-6wvlj"
Jul 28 12:55:40.907556 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:40.907530    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b86d6c456-6wvlj"]
Jul 28 12:55:40.909529 ip-10-0-129-177 kubenswrapper[2554]: W0728 12:55:40.909505    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad00d630_0ca5_455d_a6a3_91aeaf89694f.slice/crio-3fb59cd8d6044974dd227fd24d973c674a8f516e37d8650912521d3d1aaae09e WatchSource:0}: Error finding container 3fb59cd8d6044974dd227fd24d973c674a8f516e37d8650912521d3d1aaae09e: Status 404 returned error can't find the container with id 3fb59cd8d6044974dd227fd24d973c674a8f516e37d8650912521d3d1aaae09e
Jul 28 12:55:41.419236 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:41.419148    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b86d6c456-6wvlj" event={"ID":"ad00d630-0ca5-455d-a6a3-91aeaf89694f","Type":"ContainerStarted","Data":"45112d33580b7aef58f36753890e057921d6b4fcb04ddf6475c187fecd41799d"}
Jul 28 12:55:41.419236 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:41.419194    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b86d6c456-6wvlj" event={"ID":"ad00d630-0ca5-455d-a6a3-91aeaf89694f","Type":"ContainerStarted","Data":"3fb59cd8d6044974dd227fd24d973c674a8f516e37d8650912521d3d1aaae09e"}
Jul 28 12:55:41.442006 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:41.441956    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5b86d6c456-6wvlj" podStartSLOduration=1.441941242 podStartE2EDuration="1.441941242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 12:55:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 12:55:41.440914149 +0000 UTC m=+138.226789273" watchObservedRunningTime="2026-07-28 12:55:41.441941242 +0000 UTC m=+138.227816404"
Jul 28 12:55:45.420781 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:45.420755    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-665d6c9587-cjqhj"
Jul 28 12:55:46.972539 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:46.972503    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7574496c8c-kpnd9"
Jul 28 12:55:47.445798 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:47.445766    2554 generic.go:356] "Generic (PLEG): container finished" podID="324a14fe-0e26-4cea-b17f-5efda6b502de" containerID="c8640666ec9d2afa1ca79022a648415bd1e1be3efe3b4a10405f419cae972516" exitCode=0
Jul 28 12:55:47.445954 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:47.445843    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-mpgfw" event={"ID":"324a14fe-0e26-4cea-b17f-5efda6b502de","Type":"ContainerDied","Data":"c8640666ec9d2afa1ca79022a648415bd1e1be3efe3b4a10405f419cae972516"}
Jul 28 12:55:47.446246 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:47.446227    2554 scope.go:117] "RemoveContainer" containerID="c8640666ec9d2afa1ca79022a648415bd1e1be3efe3b4a10405f419cae972516"
Jul 28 12:55:48.316311 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:48.316261    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-767866489f-t5k98_128b78d0-c26a-4fb6-b914-1cafd9565ff7/router/0.log"
Jul 28 12:55:48.322428 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:48.322407    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-29778_41f6b60b-caf3-4c4c-92b0-386a82bcfc4a/serve-healthcheck-canary/0.log"
Jul 28 12:55:48.451290 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:48.451239    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-mpgfw" event={"ID":"324a14fe-0e26-4cea-b17f-5efda6b502de","Type":"ContainerStarted","Data":"fa2a6589f0117c0c09b19cab8c9f3feaf040a98c287b4ee7cb2b165d44ed7b2f"}
Jul 28 12:55:50.775320 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:50.775261    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5b86d6c456-6wvlj"
Jul 28 12:55:50.775747 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:50.775359    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5b86d6c456-6wvlj"
Jul 28 12:55:50.780104 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:50.780085    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5b86d6c456-6wvlj"
Jul 28 12:55:51.465848 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:51.465824    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5b86d6c456-6wvlj"
Jul 28 12:55:51.514946 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:55:51.514911    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-849d74bcb5-dh55v"]
Jul 28 12:56:05.449178 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:05.449121    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7574496c8c-kpnd9" podUID="2d6443ca-792d-4463-b68f-2f9dfa14b76e" containerName="console" containerID="cri-o://32cf7c98e3b0cabd4cf986edee5a7d9cb445bd48e8ce9552908e251d5e622272" gracePeriod=15
Jul 28 12:56:05.715315 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:05.715293    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7574496c8c-kpnd9"
Jul 28 12:56:05.810880 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:05.810838    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2d6443ca-792d-4463-b68f-2f9dfa14b76e-console-serving-cert\") pod \"2d6443ca-792d-4463-b68f-2f9dfa14b76e\" (UID: \"2d6443ca-792d-4463-b68f-2f9dfa14b76e\") "
Jul 28 12:56:05.811054 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:05.810891    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d6443ca-792d-4463-b68f-2f9dfa14b76e-trusted-ca-bundle\") pod \"2d6443ca-792d-4463-b68f-2f9dfa14b76e\" (UID: \"2d6443ca-792d-4463-b68f-2f9dfa14b76e\") "
Jul 28 12:56:05.811054 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:05.810925    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2d6443ca-792d-4463-b68f-2f9dfa14b76e-oauth-serving-cert\") pod \"2d6443ca-792d-4463-b68f-2f9dfa14b76e\" (UID: \"2d6443ca-792d-4463-b68f-2f9dfa14b76e\") "
Jul 28 12:56:05.811054 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:05.810957    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2d6443ca-792d-4463-b68f-2f9dfa14b76e-console-oauth-config\") pod \"2d6443ca-792d-4463-b68f-2f9dfa14b76e\" (UID: \"2d6443ca-792d-4463-b68f-2f9dfa14b76e\") "
Jul 28 12:56:05.811054 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:05.811002    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7sq5f\" (UniqueName: \"kubernetes.io/projected/2d6443ca-792d-4463-b68f-2f9dfa14b76e-kube-api-access-7sq5f\") pod \"2d6443ca-792d-4463-b68f-2f9dfa14b76e\" (UID: \"2d6443ca-792d-4463-b68f-2f9dfa14b76e\") "
Jul 28 12:56:05.811054 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:05.811039    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2d6443ca-792d-4463-b68f-2f9dfa14b76e-service-ca\") pod \"2d6443ca-792d-4463-b68f-2f9dfa14b76e\" (UID: \"2d6443ca-792d-4463-b68f-2f9dfa14b76e\") "
Jul 28 12:56:05.811317 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:05.811082    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2d6443ca-792d-4463-b68f-2f9dfa14b76e-console-config\") pod \"2d6443ca-792d-4463-b68f-2f9dfa14b76e\" (UID: \"2d6443ca-792d-4463-b68f-2f9dfa14b76e\") "
Jul 28 12:56:05.811439 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:05.811414    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d6443ca-792d-4463-b68f-2f9dfa14b76e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2d6443ca-792d-4463-b68f-2f9dfa14b76e" (UID: "2d6443ca-792d-4463-b68f-2f9dfa14b76e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:56:05.811498 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:05.811421    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d6443ca-792d-4463-b68f-2f9dfa14b76e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2d6443ca-792d-4463-b68f-2f9dfa14b76e" (UID: "2d6443ca-792d-4463-b68f-2f9dfa14b76e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:56:05.811498 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:05.811479    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d6443ca-792d-4463-b68f-2f9dfa14b76e-console-config" (OuterVolumeSpecName: "console-config") pod "2d6443ca-792d-4463-b68f-2f9dfa14b76e" (UID: "2d6443ca-792d-4463-b68f-2f9dfa14b76e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:56:05.811597 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:05.811497    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d6443ca-792d-4463-b68f-2f9dfa14b76e-service-ca" (OuterVolumeSpecName: "service-ca") pod "2d6443ca-792d-4463-b68f-2f9dfa14b76e" (UID: "2d6443ca-792d-4463-b68f-2f9dfa14b76e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:56:05.813322 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:05.813299    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d6443ca-792d-4463-b68f-2f9dfa14b76e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2d6443ca-792d-4463-b68f-2f9dfa14b76e" (UID: "2d6443ca-792d-4463-b68f-2f9dfa14b76e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:56:05.813857 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:05.813840    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d6443ca-792d-4463-b68f-2f9dfa14b76e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2d6443ca-792d-4463-b68f-2f9dfa14b76e" (UID: "2d6443ca-792d-4463-b68f-2f9dfa14b76e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:56:05.813939 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:05.813887    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d6443ca-792d-4463-b68f-2f9dfa14b76e-kube-api-access-7sq5f" (OuterVolumeSpecName: "kube-api-access-7sq5f") pod "2d6443ca-792d-4463-b68f-2f9dfa14b76e" (UID: "2d6443ca-792d-4463-b68f-2f9dfa14b76e"). InnerVolumeSpecName "kube-api-access-7sq5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:56:05.912517 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:05.912472    2554 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2d6443ca-792d-4463-b68f-2f9dfa14b76e-console-config\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 12:56:05.912517 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:05.912513    2554 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2d6443ca-792d-4463-b68f-2f9dfa14b76e-console-serving-cert\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 12:56:05.912517 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:05.912523    2554 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d6443ca-792d-4463-b68f-2f9dfa14b76e-trusted-ca-bundle\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 12:56:05.912517 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:05.912533    2554 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2d6443ca-792d-4463-b68f-2f9dfa14b76e-oauth-serving-cert\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 12:56:05.912782 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:05.912541    2554 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2d6443ca-792d-4463-b68f-2f9dfa14b76e-console-oauth-config\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 12:56:05.912782 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:05.912550    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sq5f\" (UniqueName: \"kubernetes.io/projected/2d6443ca-792d-4463-b68f-2f9dfa14b76e-kube-api-access-7sq5f\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 12:56:05.912782 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:05.912558    2554 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2d6443ca-792d-4463-b68f-2f9dfa14b76e-service-ca\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 12:56:06.519707 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:06.519673    2554 generic.go:356] "Generic (PLEG): container finished" podID="2d6443ca-792d-4463-b68f-2f9dfa14b76e" containerID="32cf7c98e3b0cabd4cf986edee5a7d9cb445bd48e8ce9552908e251d5e622272" exitCode=0
Jul 28 12:56:06.520094 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:06.519721    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7574496c8c-kpnd9"
Jul 28 12:56:06.520094 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:06.519730    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7574496c8c-kpnd9" event={"ID":"2d6443ca-792d-4463-b68f-2f9dfa14b76e","Type":"ContainerDied","Data":"32cf7c98e3b0cabd4cf986edee5a7d9cb445bd48e8ce9552908e251d5e622272"}
Jul 28 12:56:06.520094 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:06.519770    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7574496c8c-kpnd9" event={"ID":"2d6443ca-792d-4463-b68f-2f9dfa14b76e","Type":"ContainerDied","Data":"05c050fb4b8c36cb517b2e11db0846ef169dac952f45e8826da65cae268facc1"}
Jul 28 12:56:06.520094 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:06.519823    2554 scope.go:117] "RemoveContainer" containerID="32cf7c98e3b0cabd4cf986edee5a7d9cb445bd48e8ce9552908e251d5e622272"
Jul 28 12:56:06.528452 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:06.528432    2554 scope.go:117] "RemoveContainer" containerID="32cf7c98e3b0cabd4cf986edee5a7d9cb445bd48e8ce9552908e251d5e622272"
Jul 28 12:56:06.528700 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:56:06.528682    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32cf7c98e3b0cabd4cf986edee5a7d9cb445bd48e8ce9552908e251d5e622272\": container with ID starting with 32cf7c98e3b0cabd4cf986edee5a7d9cb445bd48e8ce9552908e251d5e622272 not found: ID does not exist" containerID="32cf7c98e3b0cabd4cf986edee5a7d9cb445bd48e8ce9552908e251d5e622272"
Jul 28 12:56:06.528768 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:06.528707    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32cf7c98e3b0cabd4cf986edee5a7d9cb445bd48e8ce9552908e251d5e622272"} err="failed to get container status \"32cf7c98e3b0cabd4cf986edee5a7d9cb445bd48e8ce9552908e251d5e622272\": rpc error: code = NotFound desc = could not find container \"32cf7c98e3b0cabd4cf986edee5a7d9cb445bd48e8ce9552908e251d5e622272\": container with ID starting with 32cf7c98e3b0cabd4cf986edee5a7d9cb445bd48e8ce9552908e251d5e622272 not found: ID does not exist"
Jul 28 12:56:06.540929 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:06.540903    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7574496c8c-kpnd9"]
Jul 28 12:56:06.544720 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:06.544701    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7574496c8c-kpnd9"]
Jul 28 12:56:07.801637 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:07.801603    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d6443ca-792d-4463-b68f-2f9dfa14b76e" path="/var/lib/kubelet/pods/2d6443ca-792d-4463-b68f-2f9dfa14b76e/volumes"
Jul 28 12:56:16.534398 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:16.534289    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-849d74bcb5-dh55v" podUID="4a24d572-fb68-40bd-a1f7-df3c8dcccc03" containerName="console" containerID="cri-o://aca873a14e154e2a4b4b48b47a3c6c35889e584436c5371cd3bcc507f1b9db1d" gracePeriod=15
Jul 28 12:56:16.780155 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:16.780132    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-849d74bcb5-dh55v"
Jul 28 12:56:16.896244 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:16.896166    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a24d572-fb68-40bd-a1f7-df3c8dcccc03-trusted-ca-bundle\") pod \"4a24d572-fb68-40bd-a1f7-df3c8dcccc03\" (UID: \"4a24d572-fb68-40bd-a1f7-df3c8dcccc03\") "
Jul 28 12:56:16.896244 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:16.896217    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a24d572-fb68-40bd-a1f7-df3c8dcccc03-oauth-serving-cert\") pod \"4a24d572-fb68-40bd-a1f7-df3c8dcccc03\" (UID: \"4a24d572-fb68-40bd-a1f7-df3c8dcccc03\") "
Jul 28 12:56:16.896497 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:16.896288    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a24d572-fb68-40bd-a1f7-df3c8dcccc03-service-ca\") pod \"4a24d572-fb68-40bd-a1f7-df3c8dcccc03\" (UID: \"4a24d572-fb68-40bd-a1f7-df3c8dcccc03\") "
Jul 28 12:56:16.896497 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:16.896319    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a24d572-fb68-40bd-a1f7-df3c8dcccc03-console-config\") pod \"4a24d572-fb68-40bd-a1f7-df3c8dcccc03\" (UID: \"4a24d572-fb68-40bd-a1f7-df3c8dcccc03\") "
Jul 28 12:56:16.896497 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:16.896345    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75lfb\" (UniqueName: \"kubernetes.io/projected/4a24d572-fb68-40bd-a1f7-df3c8dcccc03-kube-api-access-75lfb\") pod \"4a24d572-fb68-40bd-a1f7-df3c8dcccc03\" (UID: \"4a24d572-fb68-40bd-a1f7-df3c8dcccc03\") "
Jul 28 12:56:16.896497 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:16.896372    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a24d572-fb68-40bd-a1f7-df3c8dcccc03-console-serving-cert\") pod \"4a24d572-fb68-40bd-a1f7-df3c8dcccc03\" (UID: \"4a24d572-fb68-40bd-a1f7-df3c8dcccc03\") "
Jul 28 12:56:16.896497 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:16.896413    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a24d572-fb68-40bd-a1f7-df3c8dcccc03-console-oauth-config\") pod \"4a24d572-fb68-40bd-a1f7-df3c8dcccc03\" (UID: \"4a24d572-fb68-40bd-a1f7-df3c8dcccc03\") "
Jul 28 12:56:16.896746 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:16.896706    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a24d572-fb68-40bd-a1f7-df3c8dcccc03-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4a24d572-fb68-40bd-a1f7-df3c8dcccc03" (UID: "4a24d572-fb68-40bd-a1f7-df3c8dcccc03"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:56:16.896746 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:16.896716    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a24d572-fb68-40bd-a1f7-df3c8dcccc03-service-ca" (OuterVolumeSpecName: "service-ca") pod "4a24d572-fb68-40bd-a1f7-df3c8dcccc03" (UID: "4a24d572-fb68-40bd-a1f7-df3c8dcccc03"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:56:16.896746 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:16.896723    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a24d572-fb68-40bd-a1f7-df3c8dcccc03-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4a24d572-fb68-40bd-a1f7-df3c8dcccc03" (UID: "4a24d572-fb68-40bd-a1f7-df3c8dcccc03"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:56:16.897021 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:16.896998    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a24d572-fb68-40bd-a1f7-df3c8dcccc03-console-config" (OuterVolumeSpecName: "console-config") pod "4a24d572-fb68-40bd-a1f7-df3c8dcccc03" (UID: "4a24d572-fb68-40bd-a1f7-df3c8dcccc03"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:56:16.898904 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:16.898879    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a24d572-fb68-40bd-a1f7-df3c8dcccc03-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4a24d572-fb68-40bd-a1f7-df3c8dcccc03" (UID: "4a24d572-fb68-40bd-a1f7-df3c8dcccc03"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:56:16.898997 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:16.898905    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a24d572-fb68-40bd-a1f7-df3c8dcccc03-kube-api-access-75lfb" (OuterVolumeSpecName: "kube-api-access-75lfb") pod "4a24d572-fb68-40bd-a1f7-df3c8dcccc03" (UID: "4a24d572-fb68-40bd-a1f7-df3c8dcccc03"). InnerVolumeSpecName "kube-api-access-75lfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:56:16.898997 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:16.898905    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a24d572-fb68-40bd-a1f7-df3c8dcccc03-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4a24d572-fb68-40bd-a1f7-df3c8dcccc03" (UID: "4a24d572-fb68-40bd-a1f7-df3c8dcccc03"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:56:16.997214 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:16.997184    2554 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a24d572-fb68-40bd-a1f7-df3c8dcccc03-service-ca\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 12:56:16.997214 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:16.997209    2554 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a24d572-fb68-40bd-a1f7-df3c8dcccc03-console-config\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 12:56:16.997214 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:16.997218    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75lfb\" (UniqueName: \"kubernetes.io/projected/4a24d572-fb68-40bd-a1f7-df3c8dcccc03-kube-api-access-75lfb\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 12:56:16.997430 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:16.997227    2554 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a24d572-fb68-40bd-a1f7-df3c8dcccc03-console-serving-cert\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 12:56:16.997430 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:16.997236    2554 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a24d572-fb68-40bd-a1f7-df3c8dcccc03-console-oauth-config\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 12:56:16.997430 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:16.997244    2554 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a24d572-fb68-40bd-a1f7-df3c8dcccc03-trusted-ca-bundle\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 12:56:16.997430 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:16.997251    2554 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a24d572-fb68-40bd-a1f7-df3c8dcccc03-oauth-serving-cert\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 12:56:17.561362 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:17.561329    2554 generic.go:356] "Generic (PLEG): container finished" podID="4a24d572-fb68-40bd-a1f7-df3c8dcccc03" containerID="aca873a14e154e2a4b4b48b47a3c6c35889e584436c5371cd3bcc507f1b9db1d" exitCode=0
Jul 28 12:56:17.561754 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:17.561375    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-849d74bcb5-dh55v"
Jul 28 12:56:17.561754 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:17.561405    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-849d74bcb5-dh55v" event={"ID":"4a24d572-fb68-40bd-a1f7-df3c8dcccc03","Type":"ContainerDied","Data":"aca873a14e154e2a4b4b48b47a3c6c35889e584436c5371cd3bcc507f1b9db1d"}
Jul 28 12:56:17.561754 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:17.561447    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-849d74bcb5-dh55v" event={"ID":"4a24d572-fb68-40bd-a1f7-df3c8dcccc03","Type":"ContainerDied","Data":"fc0b633b169f581e1732ca12b0eb17d6788f766a5cf729e2c46ef72084cc4f9d"}
Jul 28 12:56:17.561754 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:17.561465    2554 scope.go:117] "RemoveContainer" containerID="aca873a14e154e2a4b4b48b47a3c6c35889e584436c5371cd3bcc507f1b9db1d"
Jul 28 12:56:17.569694 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:17.569491    2554 scope.go:117] "RemoveContainer" containerID="aca873a14e154e2a4b4b48b47a3c6c35889e584436c5371cd3bcc507f1b9db1d"
Jul 28 12:56:17.569814 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:56:17.569792    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aca873a14e154e2a4b4b48b47a3c6c35889e584436c5371cd3bcc507f1b9db1d\": container with ID starting with aca873a14e154e2a4b4b48b47a3c6c35889e584436c5371cd3bcc507f1b9db1d not found: ID does not exist" containerID="aca873a14e154e2a4b4b48b47a3c6c35889e584436c5371cd3bcc507f1b9db1d"
Jul 28 12:56:17.569872 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:17.569822    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aca873a14e154e2a4b4b48b47a3c6c35889e584436c5371cd3bcc507f1b9db1d"} err="failed to get container status \"aca873a14e154e2a4b4b48b47a3c6c35889e584436c5371cd3bcc507f1b9db1d\": rpc error: code = NotFound desc = could not find container \"aca873a14e154e2a4b4b48b47a3c6c35889e584436c5371cd3bcc507f1b9db1d\": container with ID starting with aca873a14e154e2a4b4b48b47a3c6c35889e584436c5371cd3bcc507f1b9db1d not found: ID does not exist"
Jul 28 12:56:17.586768 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:17.586744    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-849d74bcb5-dh55v"]
Jul 28 12:56:17.592207 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:17.592188    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-849d74bcb5-dh55v"]
Jul 28 12:56:17.800658 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:56:17.800627    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a24d572-fb68-40bd-a1f7-df3c8dcccc03" path="/var/lib/kubelet/pods/4a24d572-fb68-40bd-a1f7-df3c8dcccc03/volumes"
Jul 28 12:57:13.348746 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:57:13.348707    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b86d6c456-6wvlj"]
Jul 28 12:57:38.369256 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:57:38.369216    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5b86d6c456-6wvlj" podUID="ad00d630-0ca5-455d-a6a3-91aeaf89694f" containerName="console" containerID="cri-o://45112d33580b7aef58f36753890e057921d6b4fcb04ddf6475c187fecd41799d" gracePeriod=15
Jul 28 12:57:38.611826 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:57:38.611806    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b86d6c456-6wvlj"
Jul 28 12:57:38.703607 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:57:38.703581    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ad00d630-0ca5-455d-a6a3-91aeaf89694f-oauth-serving-cert\") pod \"ad00d630-0ca5-455d-a6a3-91aeaf89694f\" (UID: \"ad00d630-0ca5-455d-a6a3-91aeaf89694f\") "
Jul 28 12:57:38.703808 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:57:38.703683    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ad00d630-0ca5-455d-a6a3-91aeaf89694f-console-serving-cert\") pod \"ad00d630-0ca5-455d-a6a3-91aeaf89694f\" (UID: \"ad00d630-0ca5-455d-a6a3-91aeaf89694f\") "
Jul 28 12:57:38.703808 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:57:38.703734    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ad00d630-0ca5-455d-a6a3-91aeaf89694f-console-config\") pod \"ad00d630-0ca5-455d-a6a3-91aeaf89694f\" (UID: \"ad00d630-0ca5-455d-a6a3-91aeaf89694f\") "
Jul 28 12:57:38.703808 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:57:38.703754    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ad00d630-0ca5-455d-a6a3-91aeaf89694f-service-ca\") pod \"ad00d630-0ca5-455d-a6a3-91aeaf89694f\" (UID: \"ad00d630-0ca5-455d-a6a3-91aeaf89694f\") "
Jul 28 12:57:38.703808 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:57:38.703788    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8gqmq\" (UniqueName: \"kubernetes.io/projected/ad00d630-0ca5-455d-a6a3-91aeaf89694f-kube-api-access-8gqmq\") pod \"ad00d630-0ca5-455d-a6a3-91aeaf89694f\" (UID: \"ad00d630-0ca5-455d-a6a3-91aeaf89694f\") "
Jul 28 12:57:38.703808 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:57:38.703811    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ad00d630-0ca5-455d-a6a3-91aeaf89694f-console-oauth-config\") pod \"ad00d630-0ca5-455d-a6a3-91aeaf89694f\" (UID: \"ad00d630-0ca5-455d-a6a3-91aeaf89694f\") "
Jul 28 12:57:38.704063 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:57:38.703968    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad00d630-0ca5-455d-a6a3-91aeaf89694f-trusted-ca-bundle\") pod \"ad00d630-0ca5-455d-a6a3-91aeaf89694f\" (UID: \"ad00d630-0ca5-455d-a6a3-91aeaf89694f\") "
Jul 28 12:57:38.704157 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:57:38.704134    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad00d630-0ca5-455d-a6a3-91aeaf89694f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ad00d630-0ca5-455d-a6a3-91aeaf89694f" (UID: "ad00d630-0ca5-455d-a6a3-91aeaf89694f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:57:38.704157 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:57:38.704145    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad00d630-0ca5-455d-a6a3-91aeaf89694f-console-config" (OuterVolumeSpecName: "console-config") pod "ad00d630-0ca5-455d-a6a3-91aeaf89694f" (UID: "ad00d630-0ca5-455d-a6a3-91aeaf89694f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:57:38.704311 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:57:38.704238    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad00d630-0ca5-455d-a6a3-91aeaf89694f-service-ca" (OuterVolumeSpecName: "service-ca") pod "ad00d630-0ca5-455d-a6a3-91aeaf89694f" (UID: "ad00d630-0ca5-455d-a6a3-91aeaf89694f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:57:38.704374 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:57:38.704338    2554 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ad00d630-0ca5-455d-a6a3-91aeaf89694f-oauth-serving-cert\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 12:57:38.704374 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:57:38.704359    2554 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ad00d630-0ca5-455d-a6a3-91aeaf89694f-console-config\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 12:57:38.704728 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:57:38.704701    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad00d630-0ca5-455d-a6a3-91aeaf89694f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ad00d630-0ca5-455d-a6a3-91aeaf89694f" (UID: "ad00d630-0ca5-455d-a6a3-91aeaf89694f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 12:57:38.705934 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:57:38.705910    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad00d630-0ca5-455d-a6a3-91aeaf89694f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ad00d630-0ca5-455d-a6a3-91aeaf89694f" (UID: "ad00d630-0ca5-455d-a6a3-91aeaf89694f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:57:38.706019 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:57:38.705980    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad00d630-0ca5-455d-a6a3-91aeaf89694f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ad00d630-0ca5-455d-a6a3-91aeaf89694f" (UID: "ad00d630-0ca5-455d-a6a3-91aeaf89694f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 12:57:38.706117 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:57:38.706098    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad00d630-0ca5-455d-a6a3-91aeaf89694f-kube-api-access-8gqmq" (OuterVolumeSpecName: "kube-api-access-8gqmq") pod "ad00d630-0ca5-455d-a6a3-91aeaf89694f" (UID: "ad00d630-0ca5-455d-a6a3-91aeaf89694f"). InnerVolumeSpecName "kube-api-access-8gqmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 12:57:38.805377 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:57:38.805345    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gqmq\" (UniqueName: \"kubernetes.io/projected/ad00d630-0ca5-455d-a6a3-91aeaf89694f-kube-api-access-8gqmq\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 12:57:38.805377 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:57:38.805371    2554 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ad00d630-0ca5-455d-a6a3-91aeaf89694f-console-oauth-config\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 12:57:38.805547 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:57:38.805383    2554 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad00d630-0ca5-455d-a6a3-91aeaf89694f-trusted-ca-bundle\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 12:57:38.805547 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:57:38.805396    2554 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ad00d630-0ca5-455d-a6a3-91aeaf89694f-console-serving-cert\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 12:57:38.805547 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:57:38.805408    2554 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ad00d630-0ca5-455d-a6a3-91aeaf89694f-service-ca\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 12:57:38.842710 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:57:38.842679    2554 generic.go:356] "Generic (PLEG): container finished" podID="ad00d630-0ca5-455d-a6a3-91aeaf89694f" containerID="45112d33580b7aef58f36753890e057921d6b4fcb04ddf6475c187fecd41799d" exitCode=0
Jul 28 12:57:38.842840 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:57:38.842715    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b86d6c456-6wvlj"
Jul 28 12:57:38.842840 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:57:38.842714    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b86d6c456-6wvlj" event={"ID":"ad00d630-0ca5-455d-a6a3-91aeaf89694f","Type":"ContainerDied","Data":"45112d33580b7aef58f36753890e057921d6b4fcb04ddf6475c187fecd41799d"}
Jul 28 12:57:38.842948 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:57:38.842871    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b86d6c456-6wvlj" event={"ID":"ad00d630-0ca5-455d-a6a3-91aeaf89694f","Type":"ContainerDied","Data":"3fb59cd8d6044974dd227fd24d973c674a8f516e37d8650912521d3d1aaae09e"}
Jul 28 12:57:38.842948 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:57:38.842912    2554 scope.go:117] "RemoveContainer" containerID="45112d33580b7aef58f36753890e057921d6b4fcb04ddf6475c187fecd41799d"
Jul 28 12:57:38.852157 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:57:38.852143    2554 scope.go:117] "RemoveContainer" containerID="45112d33580b7aef58f36753890e057921d6b4fcb04ddf6475c187fecd41799d"
Jul 28 12:57:38.852420 ip-10-0-129-177 kubenswrapper[2554]: E0728 12:57:38.852402    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45112d33580b7aef58f36753890e057921d6b4fcb04ddf6475c187fecd41799d\": container with ID starting with 45112d33580b7aef58f36753890e057921d6b4fcb04ddf6475c187fecd41799d not found: ID does not exist" containerID="45112d33580b7aef58f36753890e057921d6b4fcb04ddf6475c187fecd41799d"
Jul 28 12:57:38.852469 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:57:38.852428    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45112d33580b7aef58f36753890e057921d6b4fcb04ddf6475c187fecd41799d"} err="failed to get container status \"45112d33580b7aef58f36753890e057921d6b4fcb04ddf6475c187fecd41799d\": rpc error: code = NotFound desc = could not find container \"45112d33580b7aef58f36753890e057921d6b4fcb04ddf6475c187fecd41799d\": container with ID starting with 45112d33580b7aef58f36753890e057921d6b4fcb04ddf6475c187fecd41799d not found: ID does not exist"
Jul 28 12:57:38.862906 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:57:38.862886    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b86d6c456-6wvlj"]
Jul 28 12:57:38.866468 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:57:38.866450    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5b86d6c456-6wvlj"]
Jul 28 12:57:39.801400 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:57:39.801365    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad00d630-0ca5-455d-a6a3-91aeaf89694f" path="/var/lib/kubelet/pods/ad00d630-0ca5-455d-a6a3-91aeaf89694f/volumes"
Jul 28 12:58:23.633019 ip-10-0-129-177 kubenswrapper[2554]: I0728 12:58:23.632978    2554 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 13:00:28.228429 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:28.228393    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-bw875"]
Jul 28 13:00:28.228823 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:28.228684    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a24d572-fb68-40bd-a1f7-df3c8dcccc03" containerName="console"
Jul 28 13:00:28.228823 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:28.228696    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a24d572-fb68-40bd-a1f7-df3c8dcccc03" containerName="console"
Jul 28 13:00:28.228823 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:28.228706    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad00d630-0ca5-455d-a6a3-91aeaf89694f" containerName="console"
Jul 28 13:00:28.228823 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:28.228711    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad00d630-0ca5-455d-a6a3-91aeaf89694f" containerName="console"
Jul 28 13:00:28.228823 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:28.228727    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d6443ca-792d-4463-b68f-2f9dfa14b76e" containerName="console"
Jul 28 13:00:28.228823 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:28.228733    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d6443ca-792d-4463-b68f-2f9dfa14b76e" containerName="console"
Jul 28 13:00:28.228823 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:28.228781    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a24d572-fb68-40bd-a1f7-df3c8dcccc03" containerName="console"
Jul 28 13:00:28.228823 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:28.228788    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d6443ca-792d-4463-b68f-2f9dfa14b76e" containerName="console"
Jul 28 13:00:28.228823 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:28.228795    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad00d630-0ca5-455d-a6a3-91aeaf89694f" containerName="console"
Jul 28 13:00:28.231506 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:28.231491    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-bw875"
Jul 28 13:00:28.233982 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:28.233958    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 28 13:00:28.233982 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:28.233974    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-8c2d6\""
Jul 28 13:00:28.234179 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:28.234041    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 28 13:00:28.234808 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:28.234788    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 28 13:00:28.234914 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:28.234790    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 28 13:00:28.240865 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:28.240846    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-bw875"]
Jul 28 13:00:28.355908 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:28.355868    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e3df9219-1d2a-42ef-9b96-f80fceefd95e-certificates\") pod \"keda-admission-56f9798bb4-bw875\" (UID: \"e3df9219-1d2a-42ef-9b96-f80fceefd95e\") " pod="openshift-keda/keda-admission-56f9798bb4-bw875"
Jul 28 13:00:28.355908 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:28.355909    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmnqb\" (UniqueName: \"kubernetes.io/projected/e3df9219-1d2a-42ef-9b96-f80fceefd95e-kube-api-access-gmnqb\") pod \"keda-admission-56f9798bb4-bw875\" (UID: \"e3df9219-1d2a-42ef-9b96-f80fceefd95e\") " pod="openshift-keda/keda-admission-56f9798bb4-bw875"
Jul 28 13:00:28.456726 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:28.456698    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e3df9219-1d2a-42ef-9b96-f80fceefd95e-certificates\") pod \"keda-admission-56f9798bb4-bw875\" (UID: \"e3df9219-1d2a-42ef-9b96-f80fceefd95e\") " pod="openshift-keda/keda-admission-56f9798bb4-bw875"
Jul 28 13:00:28.456892 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:28.456737    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmnqb\" (UniqueName: \"kubernetes.io/projected/e3df9219-1d2a-42ef-9b96-f80fceefd95e-kube-api-access-gmnqb\") pod \"keda-admission-56f9798bb4-bw875\" (UID: \"e3df9219-1d2a-42ef-9b96-f80fceefd95e\") " pod="openshift-keda/keda-admission-56f9798bb4-bw875"
Jul 28 13:00:28.459462 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:28.459432    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e3df9219-1d2a-42ef-9b96-f80fceefd95e-certificates\") pod \"keda-admission-56f9798bb4-bw875\" (UID: \"e3df9219-1d2a-42ef-9b96-f80fceefd95e\") " pod="openshift-keda/keda-admission-56f9798bb4-bw875"
Jul 28 13:00:28.464852 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:28.464821    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmnqb\" (UniqueName: \"kubernetes.io/projected/e3df9219-1d2a-42ef-9b96-f80fceefd95e-kube-api-access-gmnqb\") pod \"keda-admission-56f9798bb4-bw875\" (UID: \"e3df9219-1d2a-42ef-9b96-f80fceefd95e\") " pod="openshift-keda/keda-admission-56f9798bb4-bw875"
Jul 28 13:00:28.542594 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:28.542512    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-bw875"
Jul 28 13:00:28.671104 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:28.671067    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-bw875"]
Jul 28 13:00:28.674354 ip-10-0-129-177 kubenswrapper[2554]: W0728 13:00:28.674331    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3df9219_1d2a_42ef_9b96_f80fceefd95e.slice/crio-8c1b428010d13c4a44dadddb8b04b23ede238b45b0dd987d3be22f94c5b67d2b WatchSource:0}: Error finding container 8c1b428010d13c4a44dadddb8b04b23ede238b45b0dd987d3be22f94c5b67d2b: Status 404 returned error can't find the container with id 8c1b428010d13c4a44dadddb8b04b23ede238b45b0dd987d3be22f94c5b67d2b
Jul 28 13:00:28.675619 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:28.675602    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 13:00:29.449266 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:29.449221    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-bw875" event={"ID":"e3df9219-1d2a-42ef-9b96-f80fceefd95e","Type":"ContainerStarted","Data":"8c1b428010d13c4a44dadddb8b04b23ede238b45b0dd987d3be22f94c5b67d2b"}
Jul 28 13:00:33.466084 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:33.466049    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-bw875" event={"ID":"e3df9219-1d2a-42ef-9b96-f80fceefd95e","Type":"ContainerStarted","Data":"68e528845f087ef1ebdeee26b3a2a3a1f74da134d85c29813654ebc8ed7be2a2"}
Jul 28 13:00:33.466549 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:33.466188    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-56f9798bb4-bw875"
Jul 28 13:00:33.486264 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:33.486213    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-56f9798bb4-bw875" podStartSLOduration=1.486200615 podStartE2EDuration="5.486199743s" totalImagesPullingTime="3.999999128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:00:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:00:33.485577099 +0000 UTC m=+430.271452229" watchObservedRunningTime="2026-07-28 13:00:33.486199743 +0000 UTC m=+430.272075000"
Jul 28 13:00:48.869431 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:48.869353    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-f599fd97c-ndwrb"]
Jul 28 13:00:48.874828 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:48.874792    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f599fd97c-ndwrb"
Jul 28 13:00:48.880323 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:48.880302    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 28 13:00:48.880451 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:48.880308    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 28 13:00:48.881303 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:48.881262    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 28 13:00:48.881419 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:48.881315    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 28 13:00:48.881477 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:48.881429    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 28 13:00:48.881477 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:48.881445    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 28 13:00:48.881585 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:48.881490    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 28 13:00:48.881585 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:48.881555    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-8pbqd\""
Jul 28 13:00:48.886608 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:48.886590    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 28 13:00:48.891776 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:48.891758    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f599fd97c-ndwrb"]
Jul 28 13:00:49.011447 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:49.011421    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b504eb3b-e209-4ed9-b348-fa7cbbea7681-console-oauth-config\") pod \"console-f599fd97c-ndwrb\" (UID: \"b504eb3b-e209-4ed9-b348-fa7cbbea7681\") " pod="openshift-console/console-f599fd97c-ndwrb"
Jul 28 13:00:49.011447 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:49.011450    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb57p\" (UniqueName: \"kubernetes.io/projected/b504eb3b-e209-4ed9-b348-fa7cbbea7681-kube-api-access-kb57p\") pod \"console-f599fd97c-ndwrb\" (UID: \"b504eb3b-e209-4ed9-b348-fa7cbbea7681\") " pod="openshift-console/console-f599fd97c-ndwrb"
Jul 28 13:00:49.011643 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:49.011473    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b504eb3b-e209-4ed9-b348-fa7cbbea7681-oauth-serving-cert\") pod \"console-f599fd97c-ndwrb\" (UID: \"b504eb3b-e209-4ed9-b348-fa7cbbea7681\") " pod="openshift-console/console-f599fd97c-ndwrb"
Jul 28 13:00:49.011643 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:49.011521    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b504eb3b-e209-4ed9-b348-fa7cbbea7681-console-serving-cert\") pod \"console-f599fd97c-ndwrb\" (UID: \"b504eb3b-e209-4ed9-b348-fa7cbbea7681\") " pod="openshift-console/console-f599fd97c-ndwrb"
Jul 28 13:00:49.011643 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:49.011549    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b504eb3b-e209-4ed9-b348-fa7cbbea7681-service-ca\") pod \"console-f599fd97c-ndwrb\" (UID: \"b504eb3b-e209-4ed9-b348-fa7cbbea7681\") " pod="openshift-console/console-f599fd97c-ndwrb"
Jul 28 13:00:49.011643 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:49.011565    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b504eb3b-e209-4ed9-b348-fa7cbbea7681-trusted-ca-bundle\") pod \"console-f599fd97c-ndwrb\" (UID: \"b504eb3b-e209-4ed9-b348-fa7cbbea7681\") " pod="openshift-console/console-f599fd97c-ndwrb"
Jul 28 13:00:49.011643 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:49.011614    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b504eb3b-e209-4ed9-b348-fa7cbbea7681-console-config\") pod \"console-f599fd97c-ndwrb\" (UID: \"b504eb3b-e209-4ed9-b348-fa7cbbea7681\") " pod="openshift-console/console-f599fd97c-ndwrb"
Jul 28 13:00:49.112819 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:49.112777    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b504eb3b-e209-4ed9-b348-fa7cbbea7681-console-config\") pod \"console-f599fd97c-ndwrb\" (UID: \"b504eb3b-e209-4ed9-b348-fa7cbbea7681\") " pod="openshift-console/console-f599fd97c-ndwrb"
Jul 28 13:00:49.112819 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:49.112832    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b504eb3b-e209-4ed9-b348-fa7cbbea7681-console-oauth-config\") pod \"console-f599fd97c-ndwrb\" (UID: \"b504eb3b-e209-4ed9-b348-fa7cbbea7681\") " pod="openshift-console/console-f599fd97c-ndwrb"
Jul 28 13:00:49.112985 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:49.112849    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb57p\" (UniqueName: \"kubernetes.io/projected/b504eb3b-e209-4ed9-b348-fa7cbbea7681-kube-api-access-kb57p\") pod \"console-f599fd97c-ndwrb\" (UID: \"b504eb3b-e209-4ed9-b348-fa7cbbea7681\") " pod="openshift-console/console-f599fd97c-ndwrb"
Jul 28 13:00:49.112985 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:49.112869    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b504eb3b-e209-4ed9-b348-fa7cbbea7681-oauth-serving-cert\") pod \"console-f599fd97c-ndwrb\" (UID: \"b504eb3b-e209-4ed9-b348-fa7cbbea7681\") " pod="openshift-console/console-f599fd97c-ndwrb"
Jul 28 13:00:49.112985 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:49.112890    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b504eb3b-e209-4ed9-b348-fa7cbbea7681-console-serving-cert\") pod \"console-f599fd97c-ndwrb\" (UID: \"b504eb3b-e209-4ed9-b348-fa7cbbea7681\") " pod="openshift-console/console-f599fd97c-ndwrb"
Jul 28 13:00:49.112985 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:49.112903    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b504eb3b-e209-4ed9-b348-fa7cbbea7681-service-ca\") pod \"console-f599fd97c-ndwrb\" (UID: \"b504eb3b-e209-4ed9-b348-fa7cbbea7681\") " pod="openshift-console/console-f599fd97c-ndwrb"
Jul 28 13:00:49.112985 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:49.112920    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b504eb3b-e209-4ed9-b348-fa7cbbea7681-trusted-ca-bundle\") pod \"console-f599fd97c-ndwrb\" (UID: \"b504eb3b-e209-4ed9-b348-fa7cbbea7681\") " pod="openshift-console/console-f599fd97c-ndwrb"
Jul 28 13:00:49.113661 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:49.113625    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b504eb3b-e209-4ed9-b348-fa7cbbea7681-service-ca\") pod \"console-f599fd97c-ndwrb\" (UID: \"b504eb3b-e209-4ed9-b348-fa7cbbea7681\") " pod="openshift-console/console-f599fd97c-ndwrb"
Jul 28 13:00:49.113822 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:49.113799    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b504eb3b-e209-4ed9-b348-fa7cbbea7681-console-config\") pod \"console-f599fd97c-ndwrb\" (UID: \"b504eb3b-e209-4ed9-b348-fa7cbbea7681\") " pod="openshift-console/console-f599fd97c-ndwrb"
Jul 28 13:00:49.113919 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:49.113901    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b504eb3b-e209-4ed9-b348-fa7cbbea7681-trusted-ca-bundle\") pod \"console-f599fd97c-ndwrb\" (UID: \"b504eb3b-e209-4ed9-b348-fa7cbbea7681\") " pod="openshift-console/console-f599fd97c-ndwrb"
Jul 28 13:00:49.113972 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:49.113923    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b504eb3b-e209-4ed9-b348-fa7cbbea7681-oauth-serving-cert\") pod \"console-f599fd97c-ndwrb\" (UID: \"b504eb3b-e209-4ed9-b348-fa7cbbea7681\") " pod="openshift-console/console-f599fd97c-ndwrb"
Jul 28 13:00:49.115403 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:49.115377    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b504eb3b-e209-4ed9-b348-fa7cbbea7681-console-serving-cert\") pod \"console-f599fd97c-ndwrb\" (UID: \"b504eb3b-e209-4ed9-b348-fa7cbbea7681\") " pod="openshift-console/console-f599fd97c-ndwrb"
Jul 28 13:00:49.115513 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:49.115479    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b504eb3b-e209-4ed9-b348-fa7cbbea7681-console-oauth-config\") pod \"console-f599fd97c-ndwrb\" (UID: \"b504eb3b-e209-4ed9-b348-fa7cbbea7681\") " pod="openshift-console/console-f599fd97c-ndwrb"
Jul 28 13:00:49.122038 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:49.121981    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb57p\" (UniqueName: \"kubernetes.io/projected/b504eb3b-e209-4ed9-b348-fa7cbbea7681-kube-api-access-kb57p\") pod \"console-f599fd97c-ndwrb\" (UID: \"b504eb3b-e209-4ed9-b348-fa7cbbea7681\") " pod="openshift-console/console-f599fd97c-ndwrb"
Jul 28 13:00:49.184648 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:49.184617    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f599fd97c-ndwrb"
Jul 28 13:00:49.315470 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:49.315442    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f599fd97c-ndwrb"]
Jul 28 13:00:49.317831 ip-10-0-129-177 kubenswrapper[2554]: W0728 13:00:49.317804    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb504eb3b_e209_4ed9_b348_fa7cbbea7681.slice/crio-f525fa6fc77702952d87211a6802f9a131c414ad4d8083b766939aed6970a1c7 WatchSource:0}: Error finding container f525fa6fc77702952d87211a6802f9a131c414ad4d8083b766939aed6970a1c7: Status 404 returned error can't find the container with id f525fa6fc77702952d87211a6802f9a131c414ad4d8083b766939aed6970a1c7
Jul 28 13:00:49.521768 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:49.521732    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f599fd97c-ndwrb" event={"ID":"b504eb3b-e209-4ed9-b348-fa7cbbea7681","Type":"ContainerStarted","Data":"44f08914a147c2df7fd2b3a13374ac5cd915538f19e55b1d9801db88cc8e056e"}
Jul 28 13:00:49.521922 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:49.521791    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f599fd97c-ndwrb" event={"ID":"b504eb3b-e209-4ed9-b348-fa7cbbea7681","Type":"ContainerStarted","Data":"f525fa6fc77702952d87211a6802f9a131c414ad4d8083b766939aed6970a1c7"}
Jul 28 13:00:49.541830 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:49.541778    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-f599fd97c-ndwrb" podStartSLOduration=1.541763903 podStartE2EDuration="1.541763903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:00:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:00:49.541228702 +0000 UTC m=+446.327103807" watchObservedRunningTime="2026-07-28 13:00:49.541763903 +0000 UTC m=+446.327639014"
Jul 28 13:00:54.471986 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:54.471960    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-56f9798bb4-bw875"
Jul 28 13:00:59.185114 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:59.185076    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-f599fd97c-ndwrb"
Jul 28 13:00:59.185528 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:59.185138    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-f599fd97c-ndwrb"
Jul 28 13:00:59.189931 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:59.189912    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-f599fd97c-ndwrb"
Jul 28 13:00:59.562889 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:00:59.562858    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-f599fd97c-ndwrb"
Jul 28 13:01:50.186475 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:01:50.186444    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-kr4f5"]
Jul 28 13:01:50.189561 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:01:50.189544    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-ff96d5959-kr4f5"
Jul 28 13:01:50.193149 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:01:50.193127    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 28 13:01:50.193902 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:01:50.193877    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 28 13:01:50.193976 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:01:50.193959    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 28 13:01:50.194032 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:01:50.193959    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-92bxs\""
Jul 28 13:01:50.211713 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:01:50.211683    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-kr4f5"]
Jul 28 13:01:50.257815 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:01:50.257783    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-zdmzp"]
Jul 28 13:01:50.262362 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:01:50.262333    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-zdmzp"
Jul 28 13:01:50.268016 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:01:50.267990    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 28 13:01:50.268139 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:01:50.268046    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-q9h84\""
Jul 28 13:01:50.275033 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:01:50.275010    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e8182398-5493-4e60-b1ce-e8993d390029-cert\") pod \"kserve-controller-manager-ff96d5959-kr4f5\" (UID: \"e8182398-5493-4e60-b1ce-e8993d390029\") " pod="kserve/kserve-controller-manager-ff96d5959-kr4f5"
Jul 28 13:01:50.275136 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:01:50.275119    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rv54\" (UniqueName: \"kubernetes.io/projected/e8182398-5493-4e60-b1ce-e8993d390029-kube-api-access-6rv54\") pod \"kserve-controller-manager-ff96d5959-kr4f5\" (UID: \"e8182398-5493-4e60-b1ce-e8993d390029\") " pod="kserve/kserve-controller-manager-ff96d5959-kr4f5"
Jul 28 13:01:50.283065 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:01:50.283045    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-zdmzp"]
Jul 28 13:01:50.375864 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:01:50.375831    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rv54\" (UniqueName: \"kubernetes.io/projected/e8182398-5493-4e60-b1ce-e8993d390029-kube-api-access-6rv54\") pod \"kserve-controller-manager-ff96d5959-kr4f5\" (UID: \"e8182398-5493-4e60-b1ce-e8993d390029\") " pod="kserve/kserve-controller-manager-ff96d5959-kr4f5"
Jul 28 13:01:50.376049 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:01:50.375889    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e8182398-5493-4e60-b1ce-e8993d390029-cert\") pod \"kserve-controller-manager-ff96d5959-kr4f5\" (UID: \"e8182398-5493-4e60-b1ce-e8993d390029\") " pod="kserve/kserve-controller-manager-ff96d5959-kr4f5"
Jul 28 13:01:50.376116 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:01:50.376051    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2bvq\" (UniqueName: \"kubernetes.io/projected/8b41140e-33c2-496d-8330-fd37f1a99165-kube-api-access-f2bvq\") pod \"seaweedfs-64568bcd49-zdmzp\" (UID: \"8b41140e-33c2-496d-8330-fd37f1a99165\") " pod="kserve/seaweedfs-64568bcd49-zdmzp"
Jul 28 13:01:50.376116 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:01:50.376084    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8b41140e-33c2-496d-8330-fd37f1a99165-data\") pod \"seaweedfs-64568bcd49-zdmzp\" (UID: \"8b41140e-33c2-496d-8330-fd37f1a99165\") " pod="kserve/seaweedfs-64568bcd49-zdmzp"
Jul 28 13:01:50.378417 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:01:50.378395    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e8182398-5493-4e60-b1ce-e8993d390029-cert\") pod \"kserve-controller-manager-ff96d5959-kr4f5\" (UID: \"e8182398-5493-4e60-b1ce-e8993d390029\") " pod="kserve/kserve-controller-manager-ff96d5959-kr4f5"
Jul 28 13:01:50.401023 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:01:50.400997    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rv54\" (UniqueName: \"kubernetes.io/projected/e8182398-5493-4e60-b1ce-e8993d390029-kube-api-access-6rv54\") pod \"kserve-controller-manager-ff96d5959-kr4f5\" (UID: \"e8182398-5493-4e60-b1ce-e8993d390029\") " pod="kserve/kserve-controller-manager-ff96d5959-kr4f5"
Jul 28 13:01:50.477429 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:01:50.477404    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2bvq\" (UniqueName: \"kubernetes.io/projected/8b41140e-33c2-496d-8330-fd37f1a99165-kube-api-access-f2bvq\") pod \"seaweedfs-64568bcd49-zdmzp\" (UID: \"8b41140e-33c2-496d-8330-fd37f1a99165\") " pod="kserve/seaweedfs-64568bcd49-zdmzp"
Jul 28 13:01:50.477561 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:01:50.477433    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8b41140e-33c2-496d-8330-fd37f1a99165-data\") pod \"seaweedfs-64568bcd49-zdmzp\" (UID: \"8b41140e-33c2-496d-8330-fd37f1a99165\") " pod="kserve/seaweedfs-64568bcd49-zdmzp"
Jul 28 13:01:50.477738 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:01:50.477722    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8b41140e-33c2-496d-8330-fd37f1a99165-data\") pod \"seaweedfs-64568bcd49-zdmzp\" (UID: \"8b41140e-33c2-496d-8330-fd37f1a99165\") " pod="kserve/seaweedfs-64568bcd49-zdmzp"
Jul 28 13:01:50.499314 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:01:50.499286    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2bvq\" (UniqueName: \"kubernetes.io/projected/8b41140e-33c2-496d-8330-fd37f1a99165-kube-api-access-f2bvq\") pod \"seaweedfs-64568bcd49-zdmzp\" (UID: \"8b41140e-33c2-496d-8330-fd37f1a99165\") " pod="kserve/seaweedfs-64568bcd49-zdmzp"
Jul 28 13:01:50.500077 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:01:50.500062    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-ff96d5959-kr4f5"
Jul 28 13:01:50.574436 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:01:50.574406    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-zdmzp"
Jul 28 13:01:50.643052 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:01:50.643025    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-kr4f5"]
Jul 28 13:01:50.645750 ip-10-0-129-177 kubenswrapper[2554]: W0728 13:01:50.645718    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8182398_5493_4e60_b1ce_e8993d390029.slice/crio-c15f645d9bfc8b2d711b2fe7ee56261cfedbad5059301ef7cfda583e9147e8ae WatchSource:0}: Error finding container c15f645d9bfc8b2d711b2fe7ee56261cfedbad5059301ef7cfda583e9147e8ae: Status 404 returned error can't find the container with id c15f645d9bfc8b2d711b2fe7ee56261cfedbad5059301ef7cfda583e9147e8ae
Jul 28 13:01:50.724376 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:01:50.724344    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-zdmzp"]
Jul 28 13:01:50.727799 ip-10-0-129-177 kubenswrapper[2554]: W0728 13:01:50.727751    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b41140e_33c2_496d_8330_fd37f1a99165.slice/crio-16b233defa0bb281b47e4daff22fef61d0026fc21d443660406f1739cc628ef5 WatchSource:0}: Error finding container 16b233defa0bb281b47e4daff22fef61d0026fc21d443660406f1739cc628ef5: Status 404 returned error can't find the container with id 16b233defa0bb281b47e4daff22fef61d0026fc21d443660406f1739cc628ef5
Jul 28 13:01:50.752107 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:01:50.752074    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-zdmzp" event={"ID":"8b41140e-33c2-496d-8330-fd37f1a99165","Type":"ContainerStarted","Data":"16b233defa0bb281b47e4daff22fef61d0026fc21d443660406f1739cc628ef5"}
Jul 28 13:01:50.753087 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:01:50.753061    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-ff96d5959-kr4f5" event={"ID":"e8182398-5493-4e60-b1ce-e8993d390029","Type":"ContainerStarted","Data":"c15f645d9bfc8b2d711b2fe7ee56261cfedbad5059301ef7cfda583e9147e8ae"}
Jul 28 13:01:53.769631 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:01:53.769599    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-ff96d5959-kr4f5" event={"ID":"e8182398-5493-4e60-b1ce-e8993d390029","Type":"ContainerStarted","Data":"97cfc9b29b24fd06d990c7fde90232b5601da2bb864fb44ecd42c190ea4fbdc8"}
Jul 28 13:01:53.770077 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:01:53.769736    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-ff96d5959-kr4f5"
Jul 28 13:01:53.789112 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:01:53.789046    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-ff96d5959-kr4f5" podStartSLOduration=1.192153223 podStartE2EDuration="3.789024733s" totalImagesPullingTime="2.59687151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:01:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:01:53.787025063 +0000 UTC m=+510.572900186" watchObservedRunningTime="2026-07-28 13:01:53.789024733 +0000 UTC m=+510.574899847"
Jul 28 13:01:55.778691 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:01:55.778663    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-zdmzp" event={"ID":"8b41140e-33c2-496d-8330-fd37f1a99165","Type":"ContainerStarted","Data":"5ed060c2f73f8bc95606cf31bfe537f9e228939fb7f7d5402da0740b41b3f9a5"}
Jul 28 13:01:55.779039 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:01:55.778780    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-zdmzp"
Jul 28 13:01:55.796945 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:01:55.796896    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-zdmzp" podStartSLOduration=0.815905745 podStartE2EDuration="5.796879895s" totalImagesPullingTime="4.98097415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:01:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:01:55.795229396 +0000 UTC m=+512.581104510" watchObservedRunningTime="2026-07-28 13:01:55.796879895 +0000 UTC m=+512.582755006"
Jul 28 13:02:01.785503 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:02:01.785473    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-zdmzp"
Jul 28 13:02:24.779068 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:02:24.778993    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-ff96d5959-kr4f5"
Jul 28 13:02:34.971664 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:02:34.971633    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-kr4f5"]
Jul 28 13:02:34.972094 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:02:34.971835    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-ff96d5959-kr4f5" podUID="e8182398-5493-4e60-b1ce-e8993d390029" containerName="manager" containerID="cri-o://97cfc9b29b24fd06d990c7fde90232b5601da2bb864fb44ecd42c190ea4fbdc8" gracePeriod=10
Jul 28 13:02:34.995753 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:02:34.995732    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-dssz7"]
Jul 28 13:02:34.998701 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:02:34.998688    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-ff96d5959-dssz7"
Jul 28 13:02:35.009080 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:02:35.009057    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-dssz7"]
Jul 28 13:02:35.039091 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:02:35.039069    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dgq5\" (UniqueName: \"kubernetes.io/projected/f3565914-e2f8-4684-a46d-f3bc4659b7de-kube-api-access-5dgq5\") pod \"kserve-controller-manager-ff96d5959-dssz7\" (UID: \"f3565914-e2f8-4684-a46d-f3bc4659b7de\") " pod="kserve/kserve-controller-manager-ff96d5959-dssz7"
Jul 28 13:02:35.039187 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:02:35.039125    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f3565914-e2f8-4684-a46d-f3bc4659b7de-cert\") pod \"kserve-controller-manager-ff96d5959-dssz7\" (UID: \"f3565914-e2f8-4684-a46d-f3bc4659b7de\") " pod="kserve/kserve-controller-manager-ff96d5959-dssz7"
Jul 28 13:02:35.139686 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:02:35.139650    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f3565914-e2f8-4684-a46d-f3bc4659b7de-cert\") pod \"kserve-controller-manager-ff96d5959-dssz7\" (UID: \"f3565914-e2f8-4684-a46d-f3bc4659b7de\") " pod="kserve/kserve-controller-manager-ff96d5959-dssz7"
Jul 28 13:02:35.139826 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:02:35.139705    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dgq5\" (UniqueName: \"kubernetes.io/projected/f3565914-e2f8-4684-a46d-f3bc4659b7de-kube-api-access-5dgq5\") pod \"kserve-controller-manager-ff96d5959-dssz7\" (UID: \"f3565914-e2f8-4684-a46d-f3bc4659b7de\") " pod="kserve/kserve-controller-manager-ff96d5959-dssz7"
Jul 28 13:02:35.142329 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:02:35.142303    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f3565914-e2f8-4684-a46d-f3bc4659b7de-cert\") pod \"kserve-controller-manager-ff96d5959-dssz7\" (UID: \"f3565914-e2f8-4684-a46d-f3bc4659b7de\") " pod="kserve/kserve-controller-manager-ff96d5959-dssz7"
Jul 28 13:02:35.147874 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:02:35.147850    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dgq5\" (UniqueName: \"kubernetes.io/projected/f3565914-e2f8-4684-a46d-f3bc4659b7de-kube-api-access-5dgq5\") pod \"kserve-controller-manager-ff96d5959-dssz7\" (UID: \"f3565914-e2f8-4684-a46d-f3bc4659b7de\") " pod="kserve/kserve-controller-manager-ff96d5959-dssz7"
Jul 28 13:02:35.205775 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:02:35.205752    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-ff96d5959-kr4f5"
Jul 28 13:02:35.240633 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:02:35.240557    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rv54\" (UniqueName: \"kubernetes.io/projected/e8182398-5493-4e60-b1ce-e8993d390029-kube-api-access-6rv54\") pod \"e8182398-5493-4e60-b1ce-e8993d390029\" (UID: \"e8182398-5493-4e60-b1ce-e8993d390029\") "
Jul 28 13:02:35.240633 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:02:35.240602    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e8182398-5493-4e60-b1ce-e8993d390029-cert\") pod \"e8182398-5493-4e60-b1ce-e8993d390029\" (UID: \"e8182398-5493-4e60-b1ce-e8993d390029\") "
Jul 28 13:02:35.242961 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:02:35.242929    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8182398-5493-4e60-b1ce-e8993d390029-cert" (OuterVolumeSpecName: "cert") pod "e8182398-5493-4e60-b1ce-e8993d390029" (UID: "e8182398-5493-4e60-b1ce-e8993d390029"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:02:35.242961 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:02:35.242938    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8182398-5493-4e60-b1ce-e8993d390029-kube-api-access-6rv54" (OuterVolumeSpecName: "kube-api-access-6rv54") pod "e8182398-5493-4e60-b1ce-e8993d390029" (UID: "e8182398-5493-4e60-b1ce-e8993d390029"). InnerVolumeSpecName "kube-api-access-6rv54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:02:35.341695 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:02:35.341661    2554 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e8182398-5493-4e60-b1ce-e8993d390029-cert\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:02:35.341695 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:02:35.341689    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rv54\" (UniqueName: \"kubernetes.io/projected/e8182398-5493-4e60-b1ce-e8993d390029-kube-api-access-6rv54\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:02:35.344370 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:02:35.344345    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-ff96d5959-dssz7"
Jul 28 13:02:35.470406 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:02:35.470376    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-dssz7"]
Jul 28 13:02:35.473044 ip-10-0-129-177 kubenswrapper[2554]: W0728 13:02:35.473019    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3565914_e2f8_4684_a46d_f3bc4659b7de.slice/crio-df954508a932f9131ae6b2e6d9c62e32f1ea39a369291645ae75e07ef60b27a1 WatchSource:0}: Error finding container df954508a932f9131ae6b2e6d9c62e32f1ea39a369291645ae75e07ef60b27a1: Status 404 returned error can't find the container with id df954508a932f9131ae6b2e6d9c62e32f1ea39a369291645ae75e07ef60b27a1
Jul 28 13:02:35.933922 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:02:35.933886    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-ff96d5959-dssz7" event={"ID":"f3565914-e2f8-4684-a46d-f3bc4659b7de","Type":"ContainerStarted","Data":"5a9a8a24645c832fff1be807927c5ab11196ba57cc656bc2fd61e94006ef34e5"}
Jul 28 13:02:35.934184 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:02:35.934164    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-ff96d5959-dssz7"
Jul 28 13:02:35.934357 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:02:35.934334    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-ff96d5959-dssz7" event={"ID":"f3565914-e2f8-4684-a46d-f3bc4659b7de","Type":"ContainerStarted","Data":"df954508a932f9131ae6b2e6d9c62e32f1ea39a369291645ae75e07ef60b27a1"}
Jul 28 13:02:35.935316 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:02:35.935267    2554 generic.go:356] "Generic (PLEG): container finished" podID="e8182398-5493-4e60-b1ce-e8993d390029" containerID="97cfc9b29b24fd06d990c7fde90232b5601da2bb864fb44ecd42c190ea4fbdc8" exitCode=0
Jul 28 13:02:35.935412 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:02:35.935313    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-ff96d5959-kr4f5" event={"ID":"e8182398-5493-4e60-b1ce-e8993d390029","Type":"ContainerDied","Data":"97cfc9b29b24fd06d990c7fde90232b5601da2bb864fb44ecd42c190ea4fbdc8"}
Jul 28 13:02:35.935412 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:02:35.935327    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-ff96d5959-kr4f5"
Jul 28 13:02:35.935412 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:02:35.935358    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-ff96d5959-kr4f5" event={"ID":"e8182398-5493-4e60-b1ce-e8993d390029","Type":"ContainerDied","Data":"c15f645d9bfc8b2d711b2fe7ee56261cfedbad5059301ef7cfda583e9147e8ae"}
Jul 28 13:02:35.935412 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:02:35.935382    2554 scope.go:117] "RemoveContainer" containerID="97cfc9b29b24fd06d990c7fde90232b5601da2bb864fb44ecd42c190ea4fbdc8"
Jul 28 13:02:35.943294 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:02:35.943254    2554 scope.go:117] "RemoveContainer" containerID="97cfc9b29b24fd06d990c7fde90232b5601da2bb864fb44ecd42c190ea4fbdc8"
Jul 28 13:02:35.943541 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:02:35.943518    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97cfc9b29b24fd06d990c7fde90232b5601da2bb864fb44ecd42c190ea4fbdc8\": container with ID starting with 97cfc9b29b24fd06d990c7fde90232b5601da2bb864fb44ecd42c190ea4fbdc8 not found: ID does not exist" containerID="97cfc9b29b24fd06d990c7fde90232b5601da2bb864fb44ecd42c190ea4fbdc8"
Jul 28 13:02:35.943605 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:02:35.943554    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97cfc9b29b24fd06d990c7fde90232b5601da2bb864fb44ecd42c190ea4fbdc8"} err="failed to get container status \"97cfc9b29b24fd06d990c7fde90232b5601da2bb864fb44ecd42c190ea4fbdc8\": rpc error: code = NotFound desc = could not find container \"97cfc9b29b24fd06d990c7fde90232b5601da2bb864fb44ecd42c190ea4fbdc8\": container with ID starting with 97cfc9b29b24fd06d990c7fde90232b5601da2bb864fb44ecd42c190ea4fbdc8 not found: ID does not exist"
Jul 28 13:02:35.952886 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:02:35.952845    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-ff96d5959-dssz7" podStartSLOduration=1.578769729 podStartE2EDuration="1.952833115s" totalImagesPullingTime="374.063386ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:02:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:02:35.952369873 +0000 UTC m=+552.738245017" watchObservedRunningTime="2026-07-28 13:02:35.952833115 +0000 UTC m=+552.738708225"
Jul 28 13:02:35.969714 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:02:35.969687    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-kr4f5"]
Jul 28 13:02:35.972231 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:02:35.972212    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-kr4f5"]
Jul 28 13:02:37.801057 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:02:37.801023    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8182398-5493-4e60-b1ce-e8993d390029" path="/var/lib/kubelet/pods/e8182398-5493-4e60-b1ce-e8993d390029/volumes"
Jul 28 13:03:06.944519 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:03:06.944484    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-ff96d5959-dssz7"
Jul 28 13:03:08.135078 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:03:08.135023    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-jrxz2"]
Jul 28 13:03:08.135541 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:03:08.135342    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8182398-5493-4e60-b1ce-e8993d390029" containerName="manager"
Jul 28 13:03:08.135541 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:03:08.135354    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8182398-5493-4e60-b1ce-e8993d390029" containerName="manager"
Jul 28 13:03:08.135541 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:03:08.135411    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8182398-5493-4e60-b1ce-e8993d390029" containerName="manager"
Jul 28 13:03:08.138526 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:03:08.138505    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-jrxz2"
Jul 28 13:03:08.141255 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:03:08.141239    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 28 13:03:08.141409 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:03:08.141387    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-j2j8z\""
Jul 28 13:03:08.149118 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:03:08.149095    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-7lz7w"]
Jul 28 13:03:08.152492 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:03:08.152473    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-jrxz2"]
Jul 28 13:03:08.152597 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:03:08.152583    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-7lz7w"
Jul 28 13:03:08.154824 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:03:08.154804    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 28 13:03:08.154931 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:03:08.154886    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-hf2z4\""
Jul 28 13:03:08.161996 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:03:08.161975    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-7lz7w"]
Jul 28 13:03:08.194883 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:03:08.194851    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2ksn\" (UniqueName: \"kubernetes.io/projected/de86784c-96ee-431a-a222-054aef330287-kube-api-access-k2ksn\") pod \"model-serving-api-7b954ffc95-jrxz2\" (UID: \"de86784c-96ee-431a-a222-054aef330287\") " pod="kserve/model-serving-api-7b954ffc95-jrxz2"
Jul 28 13:03:08.195039 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:03:08.194904    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de86784c-96ee-431a-a222-054aef330287-tls-certs\") pod \"model-serving-api-7b954ffc95-jrxz2\" (UID: \"de86784c-96ee-431a-a222-054aef330287\") " pod="kserve/model-serving-api-7b954ffc95-jrxz2"
Jul 28 13:03:08.295727 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:03:08.295681    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de86784c-96ee-431a-a222-054aef330287-tls-certs\") pod \"model-serving-api-7b954ffc95-jrxz2\" (UID: \"de86784c-96ee-431a-a222-054aef330287\") " pod="kserve/model-serving-api-7b954ffc95-jrxz2"
Jul 28 13:03:08.295918 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:03:08.295739    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/19f8fbde-a596-4350-a520-e6dabe88a1c0-cert\") pod \"odh-model-controller-75d9cdbddf-7lz7w\" (UID: \"19f8fbde-a596-4350-a520-e6dabe88a1c0\") " pod="kserve/odh-model-controller-75d9cdbddf-7lz7w"
Jul 28 13:03:08.295918 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:03:08.295778    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52nf2\" (UniqueName: \"kubernetes.io/projected/19f8fbde-a596-4350-a520-e6dabe88a1c0-kube-api-access-52nf2\") pod \"odh-model-controller-75d9cdbddf-7lz7w\" (UID: \"19f8fbde-a596-4350-a520-e6dabe88a1c0\") " pod="kserve/odh-model-controller-75d9cdbddf-7lz7w"
Jul 28 13:03:08.295918 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:03:08.295843    2554 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 28 13:03:08.295918 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:03:08.295907    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de86784c-96ee-431a-a222-054aef330287-tls-certs podName:de86784c-96ee-431a-a222-054aef330287 nodeName:}" failed. No retries permitted until 2026-07-28 13:03:08.795886464 +0000 UTC m=+585.581761564 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/de86784c-96ee-431a-a222-054aef330287-tls-certs") pod "model-serving-api-7b954ffc95-jrxz2" (UID: "de86784c-96ee-431a-a222-054aef330287") : secret "model-serving-api-tls" not found
Jul 28 13:03:08.295918 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:03:08.295841    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2ksn\" (UniqueName: \"kubernetes.io/projected/de86784c-96ee-431a-a222-054aef330287-kube-api-access-k2ksn\") pod \"model-serving-api-7b954ffc95-jrxz2\" (UID: \"de86784c-96ee-431a-a222-054aef330287\") " pod="kserve/model-serving-api-7b954ffc95-jrxz2"
Jul 28 13:03:08.304661 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:03:08.304630    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2ksn\" (UniqueName: \"kubernetes.io/projected/de86784c-96ee-431a-a222-054aef330287-kube-api-access-k2ksn\") pod \"model-serving-api-7b954ffc95-jrxz2\" (UID: \"de86784c-96ee-431a-a222-054aef330287\") " pod="kserve/model-serving-api-7b954ffc95-jrxz2"
Jul 28 13:03:08.397316 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:03:08.397204    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/19f8fbde-a596-4350-a520-e6dabe88a1c0-cert\") pod \"odh-model-controller-75d9cdbddf-7lz7w\" (UID: \"19f8fbde-a596-4350-a520-e6dabe88a1c0\") " pod="kserve/odh-model-controller-75d9cdbddf-7lz7w"
Jul 28 13:03:08.397316 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:03:08.397253    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52nf2\" (UniqueName: \"kubernetes.io/projected/19f8fbde-a596-4350-a520-e6dabe88a1c0-kube-api-access-52nf2\") pod \"odh-model-controller-75d9cdbddf-7lz7w\" (UID: \"19f8fbde-a596-4350-a520-e6dabe88a1c0\") " pod="kserve/odh-model-controller-75d9cdbddf-7lz7w"
Jul 28 13:03:08.399825 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:03:08.399804    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/19f8fbde-a596-4350-a520-e6dabe88a1c0-cert\") pod \"odh-model-controller-75d9cdbddf-7lz7w\" (UID: \"19f8fbde-a596-4350-a520-e6dabe88a1c0\") " pod="kserve/odh-model-controller-75d9cdbddf-7lz7w"
Jul 28 13:03:08.406846 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:03:08.406816    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52nf2\" (UniqueName: \"kubernetes.io/projected/19f8fbde-a596-4350-a520-e6dabe88a1c0-kube-api-access-52nf2\") pod \"odh-model-controller-75d9cdbddf-7lz7w\" (UID: \"19f8fbde-a596-4350-a520-e6dabe88a1c0\") " pod="kserve/odh-model-controller-75d9cdbddf-7lz7w"
Jul 28 13:03:08.464431 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:03:08.464404    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-7lz7w"
Jul 28 13:03:08.594760 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:03:08.594720    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-7lz7w"]
Jul 28 13:03:08.597409 ip-10-0-129-177 kubenswrapper[2554]: W0728 13:03:08.597381    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19f8fbde_a596_4350_a520_e6dabe88a1c0.slice/crio-19beaeb334b741aa1553ad81876bd9c894b5b554b013ea0b032dd400498790ab WatchSource:0}: Error finding container 19beaeb334b741aa1553ad81876bd9c894b5b554b013ea0b032dd400498790ab: Status 404 returned error can't find the container with id 19beaeb334b741aa1553ad81876bd9c894b5b554b013ea0b032dd400498790ab
Jul 28 13:03:08.801018 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:03:08.800984    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de86784c-96ee-431a-a222-054aef330287-tls-certs\") pod \"model-serving-api-7b954ffc95-jrxz2\" (UID: \"de86784c-96ee-431a-a222-054aef330287\") " pod="kserve/model-serving-api-7b954ffc95-jrxz2"
Jul 28 13:03:08.803607 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:03:08.803583    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de86784c-96ee-431a-a222-054aef330287-tls-certs\") pod \"model-serving-api-7b954ffc95-jrxz2\" (UID: \"de86784c-96ee-431a-a222-054aef330287\") " pod="kserve/model-serving-api-7b954ffc95-jrxz2"
Jul 28 13:03:09.049837 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:03:09.049801    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-jrxz2"
Jul 28 13:03:09.062240 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:03:09.062202    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-7lz7w" event={"ID":"19f8fbde-a596-4350-a520-e6dabe88a1c0","Type":"ContainerStarted","Data":"19beaeb334b741aa1553ad81876bd9c894b5b554b013ea0b032dd400498790ab"}
Jul 28 13:03:09.180030 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:03:09.180002    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-jrxz2"]
Jul 28 13:03:09.182539 ip-10-0-129-177 kubenswrapper[2554]: W0728 13:03:09.182508    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde86784c_96ee_431a_a222_054aef330287.slice/crio-ebb25865288cefdbd7e1c519ada061355c24c22e77d5ce9451ca506cce9045d1 WatchSource:0}: Error finding container ebb25865288cefdbd7e1c519ada061355c24c22e77d5ce9451ca506cce9045d1: Status 404 returned error can't find the container with id ebb25865288cefdbd7e1c519ada061355c24c22e77d5ce9451ca506cce9045d1
Jul 28 13:03:10.070401 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:03:10.070319    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-jrxz2" event={"ID":"de86784c-96ee-431a-a222-054aef330287","Type":"ContainerStarted","Data":"ebb25865288cefdbd7e1c519ada061355c24c22e77d5ce9451ca506cce9045d1"}
Jul 28 13:03:12.081243 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:03:12.081136    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-jrxz2" event={"ID":"de86784c-96ee-431a-a222-054aef330287","Type":"ContainerStarted","Data":"29337c01df8fc79ee98681d133398333b9ed6f760393a83dd0a6de2f2aa8a999"}
Jul 28 13:03:12.081693 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:03:12.081246    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-jrxz2"
Jul 28 13:03:12.082545 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:03:12.082522    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-7lz7w" event={"ID":"19f8fbde-a596-4350-a520-e6dabe88a1c0","Type":"ContainerStarted","Data":"827ea02ef5063f72930e7a9ef25c6bfae9c343ee4eff22f50f2efabffad6d19d"}
Jul 28 13:03:12.082695 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:03:12.082652    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-75d9cdbddf-7lz7w"
Jul 28 13:03:12.099103 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:03:12.099056    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-jrxz2" podStartSLOduration=1.562099315 podStartE2EDuration="4.099041677s" totalImagesPullingTime="2.536942362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:03:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:03:12.096960738 +0000 UTC m=+588.882835850" watchObservedRunningTime="2026-07-28 13:03:12.099041677 +0000 UTC m=+588.884916788"
Jul 28 13:03:12.113471 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:03:12.113424    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-75d9cdbddf-7lz7w" podStartSLOduration=0.991087195 podStartE2EDuration="4.113404773s" totalImagesPullingTime="3.122317578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:03:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:03:12.111914743 +0000 UTC m=+588.897789867" watchObservedRunningTime="2026-07-28 13:03:12.113404773 +0000 UTC m=+588.899279884"
Jul 28 13:03:23.089416 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:03:23.089388    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-75d9cdbddf-7lz7w"
Jul 28 13:03:23.091377 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:03:23.091360    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-jrxz2"
Jul 28 13:04:42.980960 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:04:42.980927    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-4fpn6"]
Jul 28 13:04:42.984351 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:04:42.984336    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-4fpn6"
Jul 28 13:04:42.986781 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:04:42.986753    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Jul 28 13:04:42.991606 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:04:42.991585    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-4fpn6"]
Jul 28 13:04:43.051338 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:04:43.051309    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e391f949-1830-41cd-897e-f677f161b1ce-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-4fpn6\" (UID: \"e391f949-1830-41cd-897e-f677f161b1ce\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-4fpn6"
Jul 28 13:04:43.051485 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:04:43.051373    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g54d\" (UniqueName: \"kubernetes.io/projected/e391f949-1830-41cd-897e-f677f161b1ce-kube-api-access-4g54d\") pod \"seaweedfs-tls-custom-6644bf5d6c-4fpn6\" (UID: \"e391f949-1830-41cd-897e-f677f161b1ce\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-4fpn6"
Jul 28 13:04:43.151944 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:04:43.151912    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g54d\" (UniqueName: \"kubernetes.io/projected/e391f949-1830-41cd-897e-f677f161b1ce-kube-api-access-4g54d\") pod \"seaweedfs-tls-custom-6644bf5d6c-4fpn6\" (UID: \"e391f949-1830-41cd-897e-f677f161b1ce\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-4fpn6"
Jul 28 13:04:43.152093 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:04:43.151960    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e391f949-1830-41cd-897e-f677f161b1ce-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-4fpn6\" (UID: \"e391f949-1830-41cd-897e-f677f161b1ce\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-4fpn6"
Jul 28 13:04:43.152264 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:04:43.152250    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e391f949-1830-41cd-897e-f677f161b1ce-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-4fpn6\" (UID: \"e391f949-1830-41cd-897e-f677f161b1ce\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-4fpn6"
Jul 28 13:04:43.160607 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:04:43.160586    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g54d\" (UniqueName: \"kubernetes.io/projected/e391f949-1830-41cd-897e-f677f161b1ce-kube-api-access-4g54d\") pod \"seaweedfs-tls-custom-6644bf5d6c-4fpn6\" (UID: \"e391f949-1830-41cd-897e-f677f161b1ce\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-4fpn6"
Jul 28 13:04:43.294070 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:04:43.293985    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-4fpn6"
Jul 28 13:04:43.418896 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:04:43.418860    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-4fpn6"]
Jul 28 13:04:43.421650 ip-10-0-129-177 kubenswrapper[2554]: W0728 13:04:43.421625    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode391f949_1830_41cd_897e_f677f161b1ce.slice/crio-1ce9f525cb3f03d2fc07c6e1c5257df487c576c723501fcf0e2e436915af2a36 WatchSource:0}: Error finding container 1ce9f525cb3f03d2fc07c6e1c5257df487c576c723501fcf0e2e436915af2a36: Status 404 returned error can't find the container with id 1ce9f525cb3f03d2fc07c6e1c5257df487c576c723501fcf0e2e436915af2a36
Jul 28 13:04:43.432291 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:04:43.432252    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-4fpn6" event={"ID":"e391f949-1830-41cd-897e-f677f161b1ce","Type":"ContainerStarted","Data":"1ce9f525cb3f03d2fc07c6e1c5257df487c576c723501fcf0e2e436915af2a36"}
Jul 28 13:04:44.437242 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:04:44.437209    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-4fpn6" event={"ID":"e391f949-1830-41cd-897e-f677f161b1ce","Type":"ContainerStarted","Data":"a01aa36e67ed6a7a1cd9e7c2e0932b8fd0707ba755a993bc6f61b9a1bbf0c0db"}
Jul 28 13:04:44.453082 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:04:44.453029    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-4fpn6" podStartSLOduration=2.129640891 podStartE2EDuration="2.453016508s" totalImagesPullingTime="323.375617ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:04:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:04:44.452457584 +0000 UTC m=+681.238332696" watchObservedRunningTime="2026-07-28 13:04:44.453016508 +0000 UTC m=+681.238891618"
Jul 28 13:04:45.637405 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:04:45.637366    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-4fpn6"]
Jul 28 13:04:46.444363 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:04:46.444328    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-4fpn6" podUID="e391f949-1830-41cd-897e-f677f161b1ce" containerName="seaweedfs-tls-custom" containerID="cri-o://a01aa36e67ed6a7a1cd9e7c2e0932b8fd0707ba755a993bc6f61b9a1bbf0c0db" gracePeriod=30
Jul 28 13:04:47.687974 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:04:47.687950    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-4fpn6"
Jul 28 13:04:47.788991 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:04:47.788912    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e391f949-1830-41cd-897e-f677f161b1ce-data\") pod \"e391f949-1830-41cd-897e-f677f161b1ce\" (UID: \"e391f949-1830-41cd-897e-f677f161b1ce\") "
Jul 28 13:04:47.788991 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:04:47.788972    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4g54d\" (UniqueName: \"kubernetes.io/projected/e391f949-1830-41cd-897e-f677f161b1ce-kube-api-access-4g54d\") pod \"e391f949-1830-41cd-897e-f677f161b1ce\" (UID: \"e391f949-1830-41cd-897e-f677f161b1ce\") "
Jul 28 13:04:47.790137 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:04:47.790115    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e391f949-1830-41cd-897e-f677f161b1ce-data" (OuterVolumeSpecName: "data") pod "e391f949-1830-41cd-897e-f677f161b1ce" (UID: "e391f949-1830-41cd-897e-f677f161b1ce"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:04:47.791093 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:04:47.791073    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e391f949-1830-41cd-897e-f677f161b1ce-kube-api-access-4g54d" (OuterVolumeSpecName: "kube-api-access-4g54d") pod "e391f949-1830-41cd-897e-f677f161b1ce" (UID: "e391f949-1830-41cd-897e-f677f161b1ce"). InnerVolumeSpecName "kube-api-access-4g54d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:04:47.889979 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:04:47.889951    2554 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e391f949-1830-41cd-897e-f677f161b1ce-data\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:04:47.889979 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:04:47.889978    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g54d\" (UniqueName: \"kubernetes.io/projected/e391f949-1830-41cd-897e-f677f161b1ce-kube-api-access-4g54d\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:04:48.453558 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:04:48.453515    2554 generic.go:356] "Generic (PLEG): container finished" podID="e391f949-1830-41cd-897e-f677f161b1ce" containerID="a01aa36e67ed6a7a1cd9e7c2e0932b8fd0707ba755a993bc6f61b9a1bbf0c0db" exitCode=0
Jul 28 13:04:48.453756 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:04:48.453580    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-4fpn6"
Jul 28 13:04:48.453756 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:04:48.453589    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-4fpn6" event={"ID":"e391f949-1830-41cd-897e-f677f161b1ce","Type":"ContainerDied","Data":"a01aa36e67ed6a7a1cd9e7c2e0932b8fd0707ba755a993bc6f61b9a1bbf0c0db"}
Jul 28 13:04:48.453756 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:04:48.453658    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-4fpn6" event={"ID":"e391f949-1830-41cd-897e-f677f161b1ce","Type":"ContainerDied","Data":"1ce9f525cb3f03d2fc07c6e1c5257df487c576c723501fcf0e2e436915af2a36"}
Jul 28 13:04:48.453756 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:04:48.453678    2554 scope.go:117] "RemoveContainer" containerID="a01aa36e67ed6a7a1cd9e7c2e0932b8fd0707ba755a993bc6f61b9a1bbf0c0db"
Jul 28 13:04:48.462596 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:04:48.462580    2554 scope.go:117] "RemoveContainer" containerID="a01aa36e67ed6a7a1cd9e7c2e0932b8fd0707ba755a993bc6f61b9a1bbf0c0db"
Jul 28 13:04:48.462823 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:04:48.462805    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a01aa36e67ed6a7a1cd9e7c2e0932b8fd0707ba755a993bc6f61b9a1bbf0c0db\": container with ID starting with a01aa36e67ed6a7a1cd9e7c2e0932b8fd0707ba755a993bc6f61b9a1bbf0c0db not found: ID does not exist" containerID="a01aa36e67ed6a7a1cd9e7c2e0932b8fd0707ba755a993bc6f61b9a1bbf0c0db"
Jul 28 13:04:48.462873 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:04:48.462831    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a01aa36e67ed6a7a1cd9e7c2e0932b8fd0707ba755a993bc6f61b9a1bbf0c0db"} err="failed to get container status \"a01aa36e67ed6a7a1cd9e7c2e0932b8fd0707ba755a993bc6f61b9a1bbf0c0db\": rpc error: code = NotFound desc = could not find container \"a01aa36e67ed6a7a1cd9e7c2e0932b8fd0707ba755a993bc6f61b9a1bbf0c0db\": container with ID starting with a01aa36e67ed6a7a1cd9e7c2e0932b8fd0707ba755a993bc6f61b9a1bbf0c0db not found: ID does not exist"
Jul 28 13:04:48.468605 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:04:48.468583    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-4fpn6"]
Jul 28 13:04:48.470808 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:04:48.470790    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-4fpn6"]
Jul 28 13:04:48.496262 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:04:48.496238    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-rlhbx"]
Jul 28 13:04:48.496550 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:04:48.496538    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e391f949-1830-41cd-897e-f677f161b1ce" containerName="seaweedfs-tls-custom"
Jul 28 13:04:48.496591 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:04:48.496552    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="e391f949-1830-41cd-897e-f677f161b1ce" containerName="seaweedfs-tls-custom"
Jul 28 13:04:48.496624 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:04:48.496597    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="e391f949-1830-41cd-897e-f677f161b1ce" containerName="seaweedfs-tls-custom"
Jul 28 13:04:48.500842 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:04:48.500824    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-rlhbx"
Jul 28 13:04:48.503263 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:04:48.503242    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Jul 28 13:04:48.503352 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:04:48.503321    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom\""
Jul 28 13:04:48.506920 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:04:48.506900    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-rlhbx"]
Jul 28 13:04:48.596038 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:04:48.596007    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkxsd\" (UniqueName: \"kubernetes.io/projected/93f9a4a0-b773-4d3d-bd4f-d72a7089f8b1-kube-api-access-zkxsd\") pod \"seaweedfs-tls-custom-8b44d767d-rlhbx\" (UID: \"93f9a4a0-b773-4d3d-bd4f-d72a7089f8b1\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-rlhbx"
Jul 28 13:04:48.596184 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:04:48.596042    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/93f9a4a0-b773-4d3d-bd4f-d72a7089f8b1-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-rlhbx\" (UID: \"93f9a4a0-b773-4d3d-bd4f-d72a7089f8b1\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-rlhbx"
Jul 28 13:04:48.596184 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:04:48.596115    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/93f9a4a0-b773-4d3d-bd4f-d72a7089f8b1-data\") pod \"seaweedfs-tls-custom-8b44d767d-rlhbx\" (UID: \"93f9a4a0-b773-4d3d-bd4f-d72a7089f8b1\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-rlhbx"
Jul 28 13:04:48.697533 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:04:48.697489    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/93f9a4a0-b773-4d3d-bd4f-d72a7089f8b1-data\") pod \"seaweedfs-tls-custom-8b44d767d-rlhbx\" (UID: \"93f9a4a0-b773-4d3d-bd4f-d72a7089f8b1\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-rlhbx"
Jul 28 13:04:48.697922 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:04:48.697583    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkxsd\" (UniqueName: \"kubernetes.io/projected/93f9a4a0-b773-4d3d-bd4f-d72a7089f8b1-kube-api-access-zkxsd\") pod \"seaweedfs-tls-custom-8b44d767d-rlhbx\" (UID: \"93f9a4a0-b773-4d3d-bd4f-d72a7089f8b1\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-rlhbx"
Jul 28 13:04:48.697922 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:04:48.697617    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/93f9a4a0-b773-4d3d-bd4f-d72a7089f8b1-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-rlhbx\" (UID: \"93f9a4a0-b773-4d3d-bd4f-d72a7089f8b1\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-rlhbx"
Jul 28 13:04:48.697992 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:04:48.697966    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/93f9a4a0-b773-4d3d-bd4f-d72a7089f8b1-data\") pod \"seaweedfs-tls-custom-8b44d767d-rlhbx\" (UID: \"93f9a4a0-b773-4d3d-bd4f-d72a7089f8b1\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-rlhbx"
Jul 28 13:04:48.700197 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:04:48.700176    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/93f9a4a0-b773-4d3d-bd4f-d72a7089f8b1-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-rlhbx\" (UID: \"93f9a4a0-b773-4d3d-bd4f-d72a7089f8b1\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-rlhbx"
Jul 28 13:04:48.707051 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:04:48.706989    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkxsd\" (UniqueName: \"kubernetes.io/projected/93f9a4a0-b773-4d3d-bd4f-d72a7089f8b1-kube-api-access-zkxsd\") pod \"seaweedfs-tls-custom-8b44d767d-rlhbx\" (UID: \"93f9a4a0-b773-4d3d-bd4f-d72a7089f8b1\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-rlhbx"
Jul 28 13:04:48.809604 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:04:48.809580    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-rlhbx"
Jul 28 13:04:48.934827 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:04:48.934786    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-rlhbx"]
Jul 28 13:04:49.459065 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:04:49.459024    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-rlhbx" event={"ID":"93f9a4a0-b773-4d3d-bd4f-d72a7089f8b1","Type":"ContainerStarted","Data":"158d4b1be506f70a94871bf9167a0c64bd6334a8a0031aecbbc808c11a87b961"}
Jul 28 13:04:49.459258 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:04:49.459091    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-rlhbx" event={"ID":"93f9a4a0-b773-4d3d-bd4f-d72a7089f8b1","Type":"ContainerStarted","Data":"a9ccbc9b2510cad41aae0f003f542c1e5ac943ee79145c3c9d39de1cdc1f0210"}
Jul 28 13:04:49.474341 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:04:49.474231    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-8b44d767d-rlhbx" podStartSLOduration=1.191467842 podStartE2EDuration="1.474211785s" totalImagesPullingTime="282.743943ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:04:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:04:49.473373856 +0000 UTC m=+686.259248977" watchObservedRunningTime="2026-07-28 13:04:49.474211785 +0000 UTC m=+686.260086898"
Jul 28 13:04:49.801414 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:04:49.801332    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e391f949-1830-41cd-897e-f677f161b1ce" path="/var/lib/kubelet/pods/e391f949-1830-41cd-897e-f677f161b1ce/volumes"
Jul 28 13:05:01.950525 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:05:01.950492    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-s4k5q"]
Jul 28 13:05:01.953817 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:05:01.953802    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-s4k5q"
Jul 28 13:05:01.956214 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:05:01.956196    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving\""
Jul 28 13:05:01.956284 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:05:01.956229    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Jul 28 13:05:01.961564 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:05:01.961362    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-s4k5q"]
Jul 28 13:05:02.006157 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:05:02.006118    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/a2044a0c-70f6-4d6d-93fc-2468ce691b0d-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-s4k5q\" (UID: \"a2044a0c-70f6-4d6d-93fc-2468ce691b0d\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-s4k5q"
Jul 28 13:05:02.006327 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:05:02.006158    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjcnt\" (UniqueName: \"kubernetes.io/projected/a2044a0c-70f6-4d6d-93fc-2468ce691b0d-kube-api-access-hjcnt\") pod \"seaweedfs-tls-serving-67b7477c6d-s4k5q\" (UID: \"a2044a0c-70f6-4d6d-93fc-2468ce691b0d\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-s4k5q"
Jul 28 13:05:02.006327 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:05:02.006184    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a2044a0c-70f6-4d6d-93fc-2468ce691b0d-data\") pod \"seaweedfs-tls-serving-67b7477c6d-s4k5q\" (UID: \"a2044a0c-70f6-4d6d-93fc-2468ce691b0d\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-s4k5q"
Jul 28 13:05:02.107613 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:05:02.107578    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/a2044a0c-70f6-4d6d-93fc-2468ce691b0d-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-s4k5q\" (UID: \"a2044a0c-70f6-4d6d-93fc-2468ce691b0d\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-s4k5q"
Jul 28 13:05:02.107845 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:05:02.107617    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjcnt\" (UniqueName: \"kubernetes.io/projected/a2044a0c-70f6-4d6d-93fc-2468ce691b0d-kube-api-access-hjcnt\") pod \"seaweedfs-tls-serving-67b7477c6d-s4k5q\" (UID: \"a2044a0c-70f6-4d6d-93fc-2468ce691b0d\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-s4k5q"
Jul 28 13:05:02.107845 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:05:02.107646    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a2044a0c-70f6-4d6d-93fc-2468ce691b0d-data\") pod \"seaweedfs-tls-serving-67b7477c6d-s4k5q\" (UID: \"a2044a0c-70f6-4d6d-93fc-2468ce691b0d\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-s4k5q"
Jul 28 13:05:02.107845 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:05:02.107755    2554 projected.go:266] Couldn't get secret kserve/seaweedfs-tls-serving: secret "seaweedfs-tls-serving" not found
Jul 28 13:05:02.107845 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:05:02.107776    2554 projected.go:196] Error preparing data for projected volume seaweedfs-tls-serving for pod kserve/seaweedfs-tls-serving-67b7477c6d-s4k5q: secret "seaweedfs-tls-serving" not found
Jul 28 13:05:02.107997 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:05:02.107851    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2044a0c-70f6-4d6d-93fc-2468ce691b0d-seaweedfs-tls-serving podName:a2044a0c-70f6-4d6d-93fc-2468ce691b0d nodeName:}" failed. No retries permitted until 2026-07-28 13:05:02.607831403 +0000 UTC m=+699.393706495 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "seaweedfs-tls-serving" (UniqueName: "kubernetes.io/projected/a2044a0c-70f6-4d6d-93fc-2468ce691b0d-seaweedfs-tls-serving") pod "seaweedfs-tls-serving-67b7477c6d-s4k5q" (UID: "a2044a0c-70f6-4d6d-93fc-2468ce691b0d") : secret "seaweedfs-tls-serving" not found
Jul 28 13:05:02.108095 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:05:02.108074    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a2044a0c-70f6-4d6d-93fc-2468ce691b0d-data\") pod \"seaweedfs-tls-serving-67b7477c6d-s4k5q\" (UID: \"a2044a0c-70f6-4d6d-93fc-2468ce691b0d\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-s4k5q"
Jul 28 13:05:02.116111 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:05:02.116082    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjcnt\" (UniqueName: \"kubernetes.io/projected/a2044a0c-70f6-4d6d-93fc-2468ce691b0d-kube-api-access-hjcnt\") pod \"seaweedfs-tls-serving-67b7477c6d-s4k5q\" (UID: \"a2044a0c-70f6-4d6d-93fc-2468ce691b0d\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-s4k5q"
Jul 28 13:05:02.611798 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:05:02.611758    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/a2044a0c-70f6-4d6d-93fc-2468ce691b0d-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-s4k5q\" (UID: \"a2044a0c-70f6-4d6d-93fc-2468ce691b0d\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-s4k5q"
Jul 28 13:05:02.614191 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:05:02.614172    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/a2044a0c-70f6-4d6d-93fc-2468ce691b0d-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-s4k5q\" (UID: \"a2044a0c-70f6-4d6d-93fc-2468ce691b0d\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-s4k5q"
Jul 28 13:05:02.864528 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:05:02.864447    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-s4k5q"
Jul 28 13:05:02.990605 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:05:02.990544    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-s4k5q"]
Jul 28 13:05:02.993086 ip-10-0-129-177 kubenswrapper[2554]: W0728 13:05:02.993060    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2044a0c_70f6_4d6d_93fc_2468ce691b0d.slice/crio-0ad6e10b86666734d6b29c1f3866d06ed3cfc9f3f73c3c9d426560eb00dbb9e4 WatchSource:0}: Error finding container 0ad6e10b86666734d6b29c1f3866d06ed3cfc9f3f73c3c9d426560eb00dbb9e4: Status 404 returned error can't find the container with id 0ad6e10b86666734d6b29c1f3866d06ed3cfc9f3f73c3c9d426560eb00dbb9e4
Jul 28 13:05:03.518918 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:05:03.518836    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-s4k5q" event={"ID":"a2044a0c-70f6-4d6d-93fc-2468ce691b0d","Type":"ContainerStarted","Data":"ea28205ae32206eb7ff22885b87cfcfe7fa17c1affd9541a69521dfb68d41f68"}
Jul 28 13:05:03.518918 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:05:03.518886    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-s4k5q" event={"ID":"a2044a0c-70f6-4d6d-93fc-2468ce691b0d","Type":"ContainerStarted","Data":"0ad6e10b86666734d6b29c1f3866d06ed3cfc9f3f73c3c9d426560eb00dbb9e4"}
Jul 28 13:05:03.535126 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:05:03.535080    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-serving-67b7477c6d-s4k5q" podStartSLOduration=2.277519286 podStartE2EDuration="2.535067592s" totalImagesPullingTime="257.548306ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:05:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:05:03.53321082 +0000 UTC m=+700.319085942" watchObservedRunningTime="2026-07-28 13:05:03.535067592 +0000 UTC m=+700.320942704"
Jul 28 13:25:14.786048 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:25:14.784600    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gp9k9"]
Jul 28 13:25:14.789598 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:25:14.789577    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gp9k9"
Jul 28 13:25:14.792077 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:25:14.792058    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:25:14.792201 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:25:14.792059    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:25:14.792313 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:25:14.792297    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rxfwd\""
Jul 28 13:25:14.792407 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:25:14.792393    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:25:14.792936 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:25:14.792921    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Jul 28 13:25:14.793025 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:25:14.792962    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:25:14.793087 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:25:14.793026    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Jul 28 13:25:14.796797 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:25:14.796771    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gp9k9"]
Jul 28 13:25:14.821873 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:25:14.821842    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/147fa269-0f1e-4d5f-9877-01ef3db834ba-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-86766c4958-gp9k9\" (UID: \"147fa269-0f1e-4d5f-9877-01ef3db834ba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gp9k9"
Jul 28 13:25:14.821998 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:25:14.821876    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/147fa269-0f1e-4d5f-9877-01ef3db834ba-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-86766c4958-gp9k9\" (UID: \"147fa269-0f1e-4d5f-9877-01ef3db834ba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gp9k9"
Jul 28 13:25:14.821998 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:25:14.821915    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/147fa269-0f1e-4d5f-9877-01ef3db834ba-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-86766c4958-gp9k9\" (UID: \"147fa269-0f1e-4d5f-9877-01ef3db834ba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gp9k9"
Jul 28 13:25:14.822122 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:25:14.822019    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/147fa269-0f1e-4d5f-9877-01ef3db834ba-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-86766c4958-gp9k9\" (UID: \"147fa269-0f1e-4d5f-9877-01ef3db834ba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gp9k9"
Jul 28 13:25:14.822178 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:25:14.822136    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxtgc\" (UniqueName: \"kubernetes.io/projected/147fa269-0f1e-4d5f-9877-01ef3db834ba-kube-api-access-qxtgc\") pod \"isvc-canary-promote-v2-predictor-86766c4958-gp9k9\" (UID: \"147fa269-0f1e-4d5f-9877-01ef3db834ba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gp9k9"
Jul 28 13:25:14.923222 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:25:14.923185    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/147fa269-0f1e-4d5f-9877-01ef3db834ba-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-86766c4958-gp9k9\" (UID: \"147fa269-0f1e-4d5f-9877-01ef3db834ba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gp9k9"
Jul 28 13:25:14.923383 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:25:14.923247    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxtgc\" (UniqueName: \"kubernetes.io/projected/147fa269-0f1e-4d5f-9877-01ef3db834ba-kube-api-access-qxtgc\") pod \"isvc-canary-promote-v2-predictor-86766c4958-gp9k9\" (UID: \"147fa269-0f1e-4d5f-9877-01ef3db834ba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gp9k9"
Jul 28 13:25:14.923383 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:25:14.923315    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/147fa269-0f1e-4d5f-9877-01ef3db834ba-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-86766c4958-gp9k9\" (UID: \"147fa269-0f1e-4d5f-9877-01ef3db834ba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gp9k9"
Jul 28 13:25:14.923383 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:25:14.923341    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/147fa269-0f1e-4d5f-9877-01ef3db834ba-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-86766c4958-gp9k9\" (UID: \"147fa269-0f1e-4d5f-9877-01ef3db834ba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gp9k9"
Jul 28 13:25:14.923488 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:25:14.923381    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/147fa269-0f1e-4d5f-9877-01ef3db834ba-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-86766c4958-gp9k9\" (UID: \"147fa269-0f1e-4d5f-9877-01ef3db834ba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gp9k9"
Jul 28 13:25:14.923780 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:25:14.923758    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/147fa269-0f1e-4d5f-9877-01ef3db834ba-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-86766c4958-gp9k9\" (UID: \"147fa269-0f1e-4d5f-9877-01ef3db834ba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gp9k9"
Jul 28 13:25:14.923959 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:25:14.923943    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/147fa269-0f1e-4d5f-9877-01ef3db834ba-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-86766c4958-gp9k9\" (UID: \"147fa269-0f1e-4d5f-9877-01ef3db834ba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gp9k9"
Jul 28 13:25:14.924027 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:25:14.924008    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/147fa269-0f1e-4d5f-9877-01ef3db834ba-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-86766c4958-gp9k9\" (UID: \"147fa269-0f1e-4d5f-9877-01ef3db834ba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gp9k9"
Jul 28 13:25:14.926019 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:25:14.926000    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/147fa269-0f1e-4d5f-9877-01ef3db834ba-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-86766c4958-gp9k9\" (UID: \"147fa269-0f1e-4d5f-9877-01ef3db834ba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gp9k9"
Jul 28 13:25:14.930721 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:25:14.930698    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxtgc\" (UniqueName: \"kubernetes.io/projected/147fa269-0f1e-4d5f-9877-01ef3db834ba-kube-api-access-qxtgc\") pod \"isvc-canary-promote-v2-predictor-86766c4958-gp9k9\" (UID: \"147fa269-0f1e-4d5f-9877-01ef3db834ba\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gp9k9"
Jul 28 13:25:15.101336 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:25:15.101251    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gp9k9"
Jul 28 13:25:15.234027 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:25:15.234003    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gp9k9"]
Jul 28 13:25:15.236820 ip-10-0-129-177 kubenswrapper[2554]: W0728 13:25:15.236796    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod147fa269_0f1e_4d5f_9877_01ef3db834ba.slice/crio-acda67b6717af467e77e26fe5a456cc578336413b4335b13757a05a307d4a703 WatchSource:0}: Error finding container acda67b6717af467e77e26fe5a456cc578336413b4335b13757a05a307d4a703: Status 404 returned error can't find the container with id acda67b6717af467e77e26fe5a456cc578336413b4335b13757a05a307d4a703
Jul 28 13:25:15.239155 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:25:15.239139    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 13:25:16.178095 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:25:16.178052    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gp9k9" event={"ID":"147fa269-0f1e-4d5f-9877-01ef3db834ba","Type":"ContainerStarted","Data":"acda67b6717af467e77e26fe5a456cc578336413b4335b13757a05a307d4a703"}
Jul 28 13:25:20.195575 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:25:20.195540    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gp9k9" event={"ID":"147fa269-0f1e-4d5f-9877-01ef3db834ba","Type":"ContainerStarted","Data":"8a1337b3d04ded9542cd2664566a79fbd6efd4e2e4d1de436737a84c62b36ddb"}
Jul 28 13:25:23.208589 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:25:23.208559    2554 generic.go:356] "Generic (PLEG): container finished" podID="147fa269-0f1e-4d5f-9877-01ef3db834ba" containerID="8a1337b3d04ded9542cd2664566a79fbd6efd4e2e4d1de436737a84c62b36ddb" exitCode=0
Jul 28 13:25:23.208994 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:25:23.208609    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gp9k9" event={"ID":"147fa269-0f1e-4d5f-9877-01ef3db834ba","Type":"ContainerDied","Data":"8a1337b3d04ded9542cd2664566a79fbd6efd4e2e4d1de436737a84c62b36ddb"}
Jul 28 13:25:37.281936 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:25:37.281853    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gp9k9" event={"ID":"147fa269-0f1e-4d5f-9877-01ef3db834ba","Type":"ContainerStarted","Data":"0112271cb5034f58c69afcd681b4f25f3fbdf587b6c5ba4dad2089ae60c368e1"}
Jul 28 13:25:40.300311 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:25:40.300252    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gp9k9" event={"ID":"147fa269-0f1e-4d5f-9877-01ef3db834ba","Type":"ContainerStarted","Data":"fc798691eb775c84a0bd5c37bbba762e9eda04f9445b4c4e040d75b75a96f0ff"}
Jul 28 13:25:40.300685 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:25:40.300497    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gp9k9"
Jul 28 13:25:40.300685 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:25:40.300556    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gp9k9"
Jul 28 13:25:40.301709 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:25:40.301680    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gp9k9" podUID="147fa269-0f1e-4d5f-9877-01ef3db834ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 13:25:40.319352 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:25:40.319304    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gp9k9" podStartSLOduration=2.372455406 podStartE2EDuration="26.319290088s" totalImagesPullingTime="20.387582465s" totalInitContainerRuntime="3.559252217s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:25:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:25:40.317479966 +0000 UTC m=+1937.103355127" watchObservedRunningTime="2026-07-28 13:25:40.319290088 +0000 UTC m=+1937.105165189"
Jul 28 13:25:41.304041 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:25:41.303993    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gp9k9" podUID="147fa269-0f1e-4d5f-9877-01ef3db834ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 13:25:46.309689 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:25:46.309662    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gp9k9"
Jul 28 13:25:46.310140 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:25:46.310114    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gp9k9" podUID="147fa269-0f1e-4d5f-9877-01ef3db834ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 13:25:56.310254 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:25:56.310211    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gp9k9" podUID="147fa269-0f1e-4d5f-9877-01ef3db834ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 13:26:06.310871 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:26:06.310830    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gp9k9" podUID="147fa269-0f1e-4d5f-9877-01ef3db834ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 13:26:16.310963 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:26:16.310873    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gp9k9" podUID="147fa269-0f1e-4d5f-9877-01ef3db834ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 13:26:26.310916 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:26:26.310876    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gp9k9" podUID="147fa269-0f1e-4d5f-9877-01ef3db834ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 13:26:36.310846 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:26:36.310806    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gp9k9" podUID="147fa269-0f1e-4d5f-9877-01ef3db834ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 13:26:46.310211 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:26:46.310172    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gp9k9" podUID="147fa269-0f1e-4d5f-9877-01ef3db834ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 13:26:56.311428 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:26:56.311393    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gp9k9"
Jul 28 13:27:04.963451 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:04.963418    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gp9k9"]
Jul 28 13:27:04.963966 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:04.963677    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gp9k9" podUID="147fa269-0f1e-4d5f-9877-01ef3db834ba" containerName="kserve-container" containerID="cri-o://0112271cb5034f58c69afcd681b4f25f3fbdf587b6c5ba4dad2089ae60c368e1" gracePeriod=30
Jul 28 13:27:04.963966 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:04.963739    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gp9k9" podUID="147fa269-0f1e-4d5f-9877-01ef3db834ba" containerName="kube-rbac-proxy" containerID="cri-o://fc798691eb775c84a0bd5c37bbba762e9eda04f9445b4c4e040d75b75a96f0ff" gracePeriod=30
Jul 28 13:27:05.639153 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:05.639119    2554 generic.go:356] "Generic (PLEG): container finished" podID="147fa269-0f1e-4d5f-9877-01ef3db834ba" containerID="fc798691eb775c84a0bd5c37bbba762e9eda04f9445b4c4e040d75b75a96f0ff" exitCode=2
Jul 28 13:27:05.639335 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:05.639193    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gp9k9" event={"ID":"147fa269-0f1e-4d5f-9877-01ef3db834ba","Type":"ContainerDied","Data":"fc798691eb775c84a0bd5c37bbba762e9eda04f9445b4c4e040d75b75a96f0ff"}
Jul 28 13:27:06.304730 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:06.304689    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gp9k9" podUID="147fa269-0f1e-4d5f-9877-01ef3db834ba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jul 28 13:27:06.310104 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:06.310075    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gp9k9" podUID="147fa269-0f1e-4d5f-9877-01ef3db834ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 13:27:09.659870 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:09.659828    2554 generic.go:356] "Generic (PLEG): container finished" podID="147fa269-0f1e-4d5f-9877-01ef3db834ba" containerID="0112271cb5034f58c69afcd681b4f25f3fbdf587b6c5ba4dad2089ae60c368e1" exitCode=0
Jul 28 13:27:09.660292 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:09.659860    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gp9k9" event={"ID":"147fa269-0f1e-4d5f-9877-01ef3db834ba","Type":"ContainerDied","Data":"0112271cb5034f58c69afcd681b4f25f3fbdf587b6c5ba4dad2089ae60c368e1"}
Jul 28 13:27:09.718296 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:09.718254    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gp9k9"
Jul 28 13:27:09.780332 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:09.780237    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxtgc\" (UniqueName: \"kubernetes.io/projected/147fa269-0f1e-4d5f-9877-01ef3db834ba-kube-api-access-qxtgc\") pod \"147fa269-0f1e-4d5f-9877-01ef3db834ba\" (UID: \"147fa269-0f1e-4d5f-9877-01ef3db834ba\") "
Jul 28 13:27:09.780489 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:09.780361    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/147fa269-0f1e-4d5f-9877-01ef3db834ba-cabundle-cert\") pod \"147fa269-0f1e-4d5f-9877-01ef3db834ba\" (UID: \"147fa269-0f1e-4d5f-9877-01ef3db834ba\") "
Jul 28 13:27:09.780489 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:09.780393    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/147fa269-0f1e-4d5f-9877-01ef3db834ba-kube-rbac-proxy-sar-config\") pod \"147fa269-0f1e-4d5f-9877-01ef3db834ba\" (UID: \"147fa269-0f1e-4d5f-9877-01ef3db834ba\") "
Jul 28 13:27:09.780489 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:09.780421    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/147fa269-0f1e-4d5f-9877-01ef3db834ba-proxy-tls\") pod \"147fa269-0f1e-4d5f-9877-01ef3db834ba\" (UID: \"147fa269-0f1e-4d5f-9877-01ef3db834ba\") "
Jul 28 13:27:09.780489 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:09.780453    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/147fa269-0f1e-4d5f-9877-01ef3db834ba-kserve-provision-location\") pod \"147fa269-0f1e-4d5f-9877-01ef3db834ba\" (UID: \"147fa269-0f1e-4d5f-9877-01ef3db834ba\") "
Jul 28 13:27:09.780775 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:09.780750    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/147fa269-0f1e-4d5f-9877-01ef3db834ba-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "147fa269-0f1e-4d5f-9877-01ef3db834ba" (UID: "147fa269-0f1e-4d5f-9877-01ef3db834ba"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:27:09.780842 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:09.780787    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/147fa269-0f1e-4d5f-9877-01ef3db834ba-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "147fa269-0f1e-4d5f-9877-01ef3db834ba" (UID: "147fa269-0f1e-4d5f-9877-01ef3db834ba"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:27:09.780880 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:09.780834    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/147fa269-0f1e-4d5f-9877-01ef3db834ba-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "147fa269-0f1e-4d5f-9877-01ef3db834ba" (UID: "147fa269-0f1e-4d5f-9877-01ef3db834ba"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:27:09.782682 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:09.782656    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/147fa269-0f1e-4d5f-9877-01ef3db834ba-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "147fa269-0f1e-4d5f-9877-01ef3db834ba" (UID: "147fa269-0f1e-4d5f-9877-01ef3db834ba"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:27:09.782764 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:09.782717    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/147fa269-0f1e-4d5f-9877-01ef3db834ba-kube-api-access-qxtgc" (OuterVolumeSpecName: "kube-api-access-qxtgc") pod "147fa269-0f1e-4d5f-9877-01ef3db834ba" (UID: "147fa269-0f1e-4d5f-9877-01ef3db834ba"). InnerVolumeSpecName "kube-api-access-qxtgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:27:09.881873 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:09.881848    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/147fa269-0f1e-4d5f-9877-01ef3db834ba-cabundle-cert\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:27:09.881873 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:09.881872    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/147fa269-0f1e-4d5f-9877-01ef3db834ba-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:27:09.882010 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:09.881881    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/147fa269-0f1e-4d5f-9877-01ef3db834ba-proxy-tls\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:27:09.882010 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:09.881889    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/147fa269-0f1e-4d5f-9877-01ef3db834ba-kserve-provision-location\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:27:09.882010 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:09.881898    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxtgc\" (UniqueName: \"kubernetes.io/projected/147fa269-0f1e-4d5f-9877-01ef3db834ba-kube-api-access-qxtgc\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:27:10.204370 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:10.204340    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt"]
Jul 28 13:27:10.204709 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:10.204696    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="147fa269-0f1e-4d5f-9877-01ef3db834ba" containerName="kube-rbac-proxy"
Jul 28 13:27:10.204752 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:10.204712    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="147fa269-0f1e-4d5f-9877-01ef3db834ba" containerName="kube-rbac-proxy"
Jul 28 13:27:10.204752 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:10.204730    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="147fa269-0f1e-4d5f-9877-01ef3db834ba" containerName="storage-initializer"
Jul 28 13:27:10.204752 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:10.204736    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="147fa269-0f1e-4d5f-9877-01ef3db834ba" containerName="storage-initializer"
Jul 28 13:27:10.204752 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:10.204746    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="147fa269-0f1e-4d5f-9877-01ef3db834ba" containerName="kserve-container"
Jul 28 13:27:10.204870 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:10.204755    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="147fa269-0f1e-4d5f-9877-01ef3db834ba" containerName="kserve-container"
Jul 28 13:27:10.204870 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:10.204811    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="147fa269-0f1e-4d5f-9877-01ef3db834ba" containerName="kserve-container"
Jul 28 13:27:10.204870 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:10.204818    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="147fa269-0f1e-4d5f-9877-01ef3db834ba" containerName="kube-rbac-proxy"
Jul 28 13:27:10.208228 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:10.208196    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt"
Jul 28 13:27:10.210596 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:10.210573    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Jul 28 13:27:10.210700 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:10.210610    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Jul 28 13:27:10.219411 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:10.219384    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt"]
Jul 28 13:27:10.286477 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:10.286440    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l72w4\" (UniqueName: \"kubernetes.io/projected/24505273-bc06-4fa7-97dd-0d83b1582654-kube-api-access-l72w4\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt\" (UID: \"24505273-bc06-4fa7-97dd-0d83b1582654\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt"
Jul 28 13:27:10.286477 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:10.286477    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24505273-bc06-4fa7-97dd-0d83b1582654-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt\" (UID: \"24505273-bc06-4fa7-97dd-0d83b1582654\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt"
Jul 28 13:27:10.286707 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:10.286543    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24505273-bc06-4fa7-97dd-0d83b1582654-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt\" (UID: \"24505273-bc06-4fa7-97dd-0d83b1582654\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt"
Jul 28 13:27:10.286707 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:10.286585    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24505273-bc06-4fa7-97dd-0d83b1582654-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt\" (UID: \"24505273-bc06-4fa7-97dd-0d83b1582654\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt"
Jul 28 13:27:10.286707 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:10.286625    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24505273-bc06-4fa7-97dd-0d83b1582654-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt\" (UID: \"24505273-bc06-4fa7-97dd-0d83b1582654\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt"
Jul 28 13:27:10.387389 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:10.387354    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24505273-bc06-4fa7-97dd-0d83b1582654-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt\" (UID: \"24505273-bc06-4fa7-97dd-0d83b1582654\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt"
Jul 28 13:27:10.387530 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:10.387395    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24505273-bc06-4fa7-97dd-0d83b1582654-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt\" (UID: \"24505273-bc06-4fa7-97dd-0d83b1582654\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt"
Jul 28 13:27:10.387530 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:10.387436    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l72w4\" (UniqueName: \"kubernetes.io/projected/24505273-bc06-4fa7-97dd-0d83b1582654-kube-api-access-l72w4\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt\" (UID: \"24505273-bc06-4fa7-97dd-0d83b1582654\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt"
Jul 28 13:27:10.387530 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:10.387460    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24505273-bc06-4fa7-97dd-0d83b1582654-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt\" (UID: \"24505273-bc06-4fa7-97dd-0d83b1582654\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt"
Jul 28 13:27:10.387674 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:10.387615    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24505273-bc06-4fa7-97dd-0d83b1582654-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt\" (UID: \"24505273-bc06-4fa7-97dd-0d83b1582654\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt"
Jul 28 13:27:10.388010 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:10.387977    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24505273-bc06-4fa7-97dd-0d83b1582654-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt\" (UID: \"24505273-bc06-4fa7-97dd-0d83b1582654\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt"
Jul 28 13:27:10.388134 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:10.388094    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24505273-bc06-4fa7-97dd-0d83b1582654-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt\" (UID: \"24505273-bc06-4fa7-97dd-0d83b1582654\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt"
Jul 28 13:27:10.388134 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:10.388111    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24505273-bc06-4fa7-97dd-0d83b1582654-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt\" (UID: \"24505273-bc06-4fa7-97dd-0d83b1582654\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt"
Jul 28 13:27:10.390050 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:10.390033    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24505273-bc06-4fa7-97dd-0d83b1582654-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt\" (UID: \"24505273-bc06-4fa7-97dd-0d83b1582654\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt"
Jul 28 13:27:10.397951 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:10.397930    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l72w4\" (UniqueName: \"kubernetes.io/projected/24505273-bc06-4fa7-97dd-0d83b1582654-kube-api-access-l72w4\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt\" (UID: \"24505273-bc06-4fa7-97dd-0d83b1582654\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt"
Jul 28 13:27:10.521767 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:10.521696    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt"
Jul 28 13:27:10.651134 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:10.651109    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt"]
Jul 28 13:27:10.652890 ip-10-0-129-177 kubenswrapper[2554]: W0728 13:27:10.652858    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24505273_bc06_4fa7_97dd_0d83b1582654.slice/crio-a410b63164b1c0245a4cab9ab6c3fbdff10b582a8f03bf45d586007d7f97ded0 WatchSource:0}: Error finding container a410b63164b1c0245a4cab9ab6c3fbdff10b582a8f03bf45d586007d7f97ded0: Status 404 returned error can't find the container with id a410b63164b1c0245a4cab9ab6c3fbdff10b582a8f03bf45d586007d7f97ded0
Jul 28 13:27:10.666641 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:10.666620    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gp9k9"
Jul 28 13:27:10.667034 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:10.666635    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gp9k9" event={"ID":"147fa269-0f1e-4d5f-9877-01ef3db834ba","Type":"ContainerDied","Data":"acda67b6717af467e77e26fe5a456cc578336413b4335b13757a05a307d4a703"}
Jul 28 13:27:10.667034 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:10.666717    2554 scope.go:117] "RemoveContainer" containerID="fc798691eb775c84a0bd5c37bbba762e9eda04f9445b4c4e040d75b75a96f0ff"
Jul 28 13:27:10.668218 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:10.668138    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt" event={"ID":"24505273-bc06-4fa7-97dd-0d83b1582654","Type":"ContainerStarted","Data":"a410b63164b1c0245a4cab9ab6c3fbdff10b582a8f03bf45d586007d7f97ded0"}
Jul 28 13:27:10.680112 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:10.680092    2554 scope.go:117] "RemoveContainer" containerID="0112271cb5034f58c69afcd681b4f25f3fbdf587b6c5ba4dad2089ae60c368e1"
Jul 28 13:27:10.691883 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:10.691868    2554 scope.go:117] "RemoveContainer" containerID="8a1337b3d04ded9542cd2664566a79fbd6efd4e2e4d1de436737a84c62b36ddb"
Jul 28 13:27:10.702094 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:10.702067    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gp9k9"]
Jul 28 13:27:10.706909 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:10.706889    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-gp9k9"]
Jul 28 13:27:11.673891 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:11.673851    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt" event={"ID":"24505273-bc06-4fa7-97dd-0d83b1582654","Type":"ContainerStarted","Data":"3e49c085c0eef6dcb825d9a20cfd08ae65c7c78150486bbc6f45319945b65a43"}
Jul 28 13:27:11.801643 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:11.801610    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="147fa269-0f1e-4d5f-9877-01ef3db834ba" path="/var/lib/kubelet/pods/147fa269-0f1e-4d5f-9877-01ef3db834ba/volumes"
Jul 28 13:27:14.688638 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:14.688549    2554 generic.go:356] "Generic (PLEG): container finished" podID="24505273-bc06-4fa7-97dd-0d83b1582654" containerID="3e49c085c0eef6dcb825d9a20cfd08ae65c7c78150486bbc6f45319945b65a43" exitCode=0
Jul 28 13:27:14.688638 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:14.688614    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt" event={"ID":"24505273-bc06-4fa7-97dd-0d83b1582654","Type":"ContainerDied","Data":"3e49c085c0eef6dcb825d9a20cfd08ae65c7c78150486bbc6f45319945b65a43"}
Jul 28 13:27:15.695701 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:15.695659    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt" event={"ID":"24505273-bc06-4fa7-97dd-0d83b1582654","Type":"ContainerStarted","Data":"bf6d7a31359ab1e6a584bbf44dd3e3f8459390a4876a7bc1affe33e4868dd8e0"}
Jul 28 13:27:15.696185 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:15.695722    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt" event={"ID":"24505273-bc06-4fa7-97dd-0d83b1582654","Type":"ContainerStarted","Data":"bb5e850f621d60e7a0e5d82bfaf516c4b5b72cac157196673478314bca427764"}
Jul 28 13:27:15.696185 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:15.695993    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt"
Jul 28 13:27:15.696185 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:15.696137    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt"
Jul 28 13:27:15.697427 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:15.697401    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt" podUID="24505273-bc06-4fa7-97dd-0d83b1582654" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 13:27:15.716098 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:15.716054    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt" podStartSLOduration=2.098299186 podStartE2EDuration="5.71603997s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.617740784s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:27:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:27:15.715599369 +0000 UTC m=+2032.501474481" watchObservedRunningTime="2026-07-28 13:27:15.71603997 +0000 UTC m=+2032.501915081"
Jul 28 13:27:16.700021 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:16.699984    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt" podUID="24505273-bc06-4fa7-97dd-0d83b1582654" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 13:27:21.704051 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:21.704021    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt"
Jul 28 13:27:21.704646 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:21.704619    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt" podUID="24505273-bc06-4fa7-97dd-0d83b1582654" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 13:27:31.704776 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:31.704738    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt" podUID="24505273-bc06-4fa7-97dd-0d83b1582654" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 13:27:41.705174 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:41.705134    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt" podUID="24505273-bc06-4fa7-97dd-0d83b1582654" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 13:27:51.704635 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:27:51.704552    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt" podUID="24505273-bc06-4fa7-97dd-0d83b1582654" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 13:28:01.705219 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:28:01.705174    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt" podUID="24505273-bc06-4fa7-97dd-0d83b1582654" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 13:28:11.704727 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:28:11.704682    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt" podUID="24505273-bc06-4fa7-97dd-0d83b1582654" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 13:28:21.705009 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:28:21.704967    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt" podUID="24505273-bc06-4fa7-97dd-0d83b1582654" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 13:28:31.705185 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:28:31.705152    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt"
Jul 28 13:30:00.276030 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:30:00.275993    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt"]
Jul 28 13:30:00.276592 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:30:00.276244    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt" podUID="24505273-bc06-4fa7-97dd-0d83b1582654" containerName="kserve-container" containerID="cri-o://bb5e850f621d60e7a0e5d82bfaf516c4b5b72cac157196673478314bca427764" gracePeriod=30
Jul 28 13:30:00.276592 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:30:00.276316    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt" podUID="24505273-bc06-4fa7-97dd-0d83b1582654" containerName="kube-rbac-proxy" containerID="cri-o://bf6d7a31359ab1e6a584bbf44dd3e3f8459390a4876a7bc1affe33e4868dd8e0" gracePeriod=30
Jul 28 13:30:01.346588 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:30:01.346558    2554 generic.go:356] "Generic (PLEG): container finished" podID="24505273-bc06-4fa7-97dd-0d83b1582654" containerID="bf6d7a31359ab1e6a584bbf44dd3e3f8459390a4876a7bc1affe33e4868dd8e0" exitCode=2
Jul 28 13:30:01.346944 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:30:01.346628    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt" event={"ID":"24505273-bc06-4fa7-97dd-0d83b1582654","Type":"ContainerDied","Data":"bf6d7a31359ab1e6a584bbf44dd3e3f8459390a4876a7bc1affe33e4868dd8e0"}
Jul 28 13:30:01.701331 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:30:01.701252    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt" podUID="24505273-bc06-4fa7-97dd-0d83b1582654" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 28 13:30:01.704574 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:30:01.704545    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt" podUID="24505273-bc06-4fa7-97dd-0d83b1582654" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 13:30:04.826729 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:30:04.826703    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt"
Jul 28 13:30:04.931025 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:30:04.930937    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l72w4\" (UniqueName: \"kubernetes.io/projected/24505273-bc06-4fa7-97dd-0d83b1582654-kube-api-access-l72w4\") pod \"24505273-bc06-4fa7-97dd-0d83b1582654\" (UID: \"24505273-bc06-4fa7-97dd-0d83b1582654\") "
Jul 28 13:30:04.931025 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:30:04.930982    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24505273-bc06-4fa7-97dd-0d83b1582654-kube-rbac-proxy-sar-config\") pod \"24505273-bc06-4fa7-97dd-0d83b1582654\" (UID: \"24505273-bc06-4fa7-97dd-0d83b1582654\") "
Jul 28 13:30:04.931025 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:30:04.931015    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24505273-bc06-4fa7-97dd-0d83b1582654-kserve-provision-location\") pod \"24505273-bc06-4fa7-97dd-0d83b1582654\" (UID: \"24505273-bc06-4fa7-97dd-0d83b1582654\") "
Jul 28 13:30:04.931312 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:30:04.931032    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24505273-bc06-4fa7-97dd-0d83b1582654-proxy-tls\") pod \"24505273-bc06-4fa7-97dd-0d83b1582654\" (UID: \"24505273-bc06-4fa7-97dd-0d83b1582654\") "
Jul 28 13:30:04.931312 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:30:04.931078    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24505273-bc06-4fa7-97dd-0d83b1582654-cabundle-cert\") pod \"24505273-bc06-4fa7-97dd-0d83b1582654\" (UID: \"24505273-bc06-4fa7-97dd-0d83b1582654\") "
Jul 28 13:30:04.931476 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:30:04.931446    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24505273-bc06-4fa7-97dd-0d83b1582654-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "24505273-bc06-4fa7-97dd-0d83b1582654" (UID: "24505273-bc06-4fa7-97dd-0d83b1582654"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:30:04.931476 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:30:04.931409    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24505273-bc06-4fa7-97dd-0d83b1582654-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "24505273-bc06-4fa7-97dd-0d83b1582654" (UID: "24505273-bc06-4fa7-97dd-0d83b1582654"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:30:04.931606 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:30:04.931564    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24505273-bc06-4fa7-97dd-0d83b1582654-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "24505273-bc06-4fa7-97dd-0d83b1582654" (UID: "24505273-bc06-4fa7-97dd-0d83b1582654"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:30:04.933328 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:30:04.933299    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24505273-bc06-4fa7-97dd-0d83b1582654-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "24505273-bc06-4fa7-97dd-0d83b1582654" (UID: "24505273-bc06-4fa7-97dd-0d83b1582654"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:30:04.933405 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:30:04.933328    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24505273-bc06-4fa7-97dd-0d83b1582654-kube-api-access-l72w4" (OuterVolumeSpecName: "kube-api-access-l72w4") pod "24505273-bc06-4fa7-97dd-0d83b1582654" (UID: "24505273-bc06-4fa7-97dd-0d83b1582654"). InnerVolumeSpecName "kube-api-access-l72w4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:30:05.032584 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:30:05.032549    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24505273-bc06-4fa7-97dd-0d83b1582654-cabundle-cert\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:30:05.032584 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:30:05.032582    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l72w4\" (UniqueName: \"kubernetes.io/projected/24505273-bc06-4fa7-97dd-0d83b1582654-kube-api-access-l72w4\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:30:05.032584 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:30:05.032599    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24505273-bc06-4fa7-97dd-0d83b1582654-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:30:05.032827 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:30:05.032613    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24505273-bc06-4fa7-97dd-0d83b1582654-kserve-provision-location\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:30:05.032827 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:30:05.032625    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24505273-bc06-4fa7-97dd-0d83b1582654-proxy-tls\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:30:05.369732 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:30:05.369692    2554 generic.go:356] "Generic (PLEG): container finished" podID="24505273-bc06-4fa7-97dd-0d83b1582654" containerID="bb5e850f621d60e7a0e5d82bfaf516c4b5b72cac157196673478314bca427764" exitCode=0
Jul 28 13:30:05.369896 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:30:05.369746    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt" event={"ID":"24505273-bc06-4fa7-97dd-0d83b1582654","Type":"ContainerDied","Data":"bb5e850f621d60e7a0e5d82bfaf516c4b5b72cac157196673478314bca427764"}
Jul 28 13:30:05.369896 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:30:05.369766    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt"
Jul 28 13:30:05.369896 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:30:05.369819    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt" event={"ID":"24505273-bc06-4fa7-97dd-0d83b1582654","Type":"ContainerDied","Data":"a410b63164b1c0245a4cab9ab6c3fbdff10b582a8f03bf45d586007d7f97ded0"}
Jul 28 13:30:05.369896 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:30:05.369848    2554 scope.go:117] "RemoveContainer" containerID="bf6d7a31359ab1e6a584bbf44dd3e3f8459390a4876a7bc1affe33e4868dd8e0"
Jul 28 13:30:05.379117 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:30:05.379100    2554 scope.go:117] "RemoveContainer" containerID="bb5e850f621d60e7a0e5d82bfaf516c4b5b72cac157196673478314bca427764"
Jul 28 13:30:05.389119 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:30:05.389096    2554 scope.go:117] "RemoveContainer" containerID="3e49c085c0eef6dcb825d9a20cfd08ae65c7c78150486bbc6f45319945b65a43"
Jul 28 13:30:05.397955 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:30:05.397929    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt"]
Jul 28 13:30:05.399228 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:30:05.399205    2554 scope.go:117] "RemoveContainer" containerID="bf6d7a31359ab1e6a584bbf44dd3e3f8459390a4876a7bc1affe33e4868dd8e0"
Jul 28 13:30:05.399679 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:30:05.399646    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf6d7a31359ab1e6a584bbf44dd3e3f8459390a4876a7bc1affe33e4868dd8e0\": container with ID starting with bf6d7a31359ab1e6a584bbf44dd3e3f8459390a4876a7bc1affe33e4868dd8e0 not found: ID does not exist" containerID="bf6d7a31359ab1e6a584bbf44dd3e3f8459390a4876a7bc1affe33e4868dd8e0"
Jul 28 13:30:05.399773 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:30:05.399693    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf6d7a31359ab1e6a584bbf44dd3e3f8459390a4876a7bc1affe33e4868dd8e0"} err="failed to get container status \"bf6d7a31359ab1e6a584bbf44dd3e3f8459390a4876a7bc1affe33e4868dd8e0\": rpc error: code = NotFound desc = could not find container \"bf6d7a31359ab1e6a584bbf44dd3e3f8459390a4876a7bc1affe33e4868dd8e0\": container with ID starting with bf6d7a31359ab1e6a584bbf44dd3e3f8459390a4876a7bc1affe33e4868dd8e0 not found: ID does not exist"
Jul 28 13:30:05.399773 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:30:05.399721    2554 scope.go:117] "RemoveContainer" containerID="bb5e850f621d60e7a0e5d82bfaf516c4b5b72cac157196673478314bca427764"
Jul 28 13:30:05.400079 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:30:05.400033    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb5e850f621d60e7a0e5d82bfaf516c4b5b72cac157196673478314bca427764\": container with ID starting with bb5e850f621d60e7a0e5d82bfaf516c4b5b72cac157196673478314bca427764 not found: ID does not exist" containerID="bb5e850f621d60e7a0e5d82bfaf516c4b5b72cac157196673478314bca427764"
Jul 28 13:30:05.400079 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:30:05.400068    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb5e850f621d60e7a0e5d82bfaf516c4b5b72cac157196673478314bca427764"} err="failed to get container status \"bb5e850f621d60e7a0e5d82bfaf516c4b5b72cac157196673478314bca427764\": rpc error: code = NotFound desc = could not find container \"bb5e850f621d60e7a0e5d82bfaf516c4b5b72cac157196673478314bca427764\": container with ID starting with bb5e850f621d60e7a0e5d82bfaf516c4b5b72cac157196673478314bca427764 not found: ID does not exist"
Jul 28 13:30:05.400229 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:30:05.400090    2554 scope.go:117] "RemoveContainer" containerID="3e49c085c0eef6dcb825d9a20cfd08ae65c7c78150486bbc6f45319945b65a43"
Jul 28 13:30:05.400411 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:30:05.400394    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e49c085c0eef6dcb825d9a20cfd08ae65c7c78150486bbc6f45319945b65a43\": container with ID starting with 3e49c085c0eef6dcb825d9a20cfd08ae65c7c78150486bbc6f45319945b65a43 not found: ID does not exist" containerID="3e49c085c0eef6dcb825d9a20cfd08ae65c7c78150486bbc6f45319945b65a43"
Jul 28 13:30:05.400474 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:30:05.400417    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e49c085c0eef6dcb825d9a20cfd08ae65c7c78150486bbc6f45319945b65a43"} err="failed to get container status \"3e49c085c0eef6dcb825d9a20cfd08ae65c7c78150486bbc6f45319945b65a43\": rpc error: code = NotFound desc = could not find container \"3e49c085c0eef6dcb825d9a20cfd08ae65c7c78150486bbc6f45319945b65a43\": container with ID starting with 3e49c085c0eef6dcb825d9a20cfd08ae65c7c78150486bbc6f45319945b65a43 not found: ID does not exist"
Jul 28 13:30:05.407910 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:30:05.407879    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-7gtxt"]
Jul 28 13:30:05.802057 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:30:05.802023    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24505273-bc06-4fa7-97dd-0d83b1582654" path="/var/lib/kubelet/pods/24505273-bc06-4fa7-97dd-0d83b1582654/volumes"
Jul 28 13:31:45.824655 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:31:45.824622    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dsjn6"]
Jul 28 13:31:45.825049 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:31:45.824955    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24505273-bc06-4fa7-97dd-0d83b1582654" containerName="kserve-container"
Jul 28 13:31:45.825049 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:31:45.824967    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="24505273-bc06-4fa7-97dd-0d83b1582654" containerName="kserve-container"
Jul 28 13:31:45.825049 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:31:45.824981    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24505273-bc06-4fa7-97dd-0d83b1582654" containerName="kube-rbac-proxy"
Jul 28 13:31:45.825049 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:31:45.824986    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="24505273-bc06-4fa7-97dd-0d83b1582654" containerName="kube-rbac-proxy"
Jul 28 13:31:45.825049 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:31:45.824993    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24505273-bc06-4fa7-97dd-0d83b1582654" containerName="storage-initializer"
Jul 28 13:31:45.825049 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:31:45.824999    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="24505273-bc06-4fa7-97dd-0d83b1582654" containerName="storage-initializer"
Jul 28 13:31:45.825049 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:31:45.825052    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="24505273-bc06-4fa7-97dd-0d83b1582654" containerName="kube-rbac-proxy"
Jul 28 13:31:45.825259 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:31:45.825062    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="24505273-bc06-4fa7-97dd-0d83b1582654" containerName="kserve-container"
Jul 28 13:31:45.828183 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:31:45.828164    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dsjn6"
Jul 28 13:31:45.830574 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:31:45.830513    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\""
Jul 28 13:31:45.830574 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:31:45.830541    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
Jul 28 13:31:45.830574 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:31:45.830555    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rxfwd\""
Jul 28 13:31:45.830775 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:31:45.830654    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:31:45.830832 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:31:45.830787    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:31:45.830832 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:31:45.830804    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:31:45.831456 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:31:45.831437    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:31:45.837724 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:31:45.837698    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dsjn6"]
Jul 28 13:31:45.932954 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:31:45.932917    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-dsjn6\" (UID: \"f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dsjn6"
Jul 28 13:31:45.933113 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:31:45.932956    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-dsjn6\" (UID: \"f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dsjn6"
Jul 28 13:31:45.933113 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:31:45.933060    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4mfh\" (UniqueName: \"kubernetes.io/projected/f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c-kube-api-access-x4mfh\") pod \"isvc-lightgbm-predictor-54f85ddf88-dsjn6\" (UID: \"f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dsjn6"
Jul 28 13:31:45.933113 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:31:45.933095    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-dsjn6\" (UID: \"f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dsjn6"
Jul 28 13:31:45.933224 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:31:45.933124    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-dsjn6\" (UID: \"f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dsjn6"
Jul 28 13:31:46.033975 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:31:46.033940    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4mfh\" (UniqueName: \"kubernetes.io/projected/f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c-kube-api-access-x4mfh\") pod \"isvc-lightgbm-predictor-54f85ddf88-dsjn6\" (UID: \"f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dsjn6"
Jul 28 13:31:46.033975 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:31:46.033975    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-dsjn6\" (UID: \"f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dsjn6"
Jul 28 13:31:46.034238 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:31:46.033995    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-dsjn6\" (UID: \"f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dsjn6"
Jul 28 13:31:46.034238 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:31:46.034041    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-dsjn6\" (UID: \"f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dsjn6"
Jul 28 13:31:46.034238 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:31:46.034058    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-dsjn6\" (UID: \"f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dsjn6"
Jul 28 13:31:46.034533 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:31:46.034510    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-dsjn6\" (UID: \"f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dsjn6"
Jul 28 13:31:46.034756 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:31:46.034735    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-dsjn6\" (UID: \"f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dsjn6"
Jul 28 13:31:46.034756 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:31:46.034752    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-dsjn6\" (UID: \"f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dsjn6"
Jul 28 13:31:46.036626 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:31:46.036606    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-dsjn6\" (UID: \"f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dsjn6"
Jul 28 13:31:46.043088 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:31:46.043068    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4mfh\" (UniqueName: \"kubernetes.io/projected/f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c-kube-api-access-x4mfh\") pod \"isvc-lightgbm-predictor-54f85ddf88-dsjn6\" (UID: \"f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dsjn6"
Jul 28 13:31:46.140425 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:31:46.140358    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dsjn6"
Jul 28 13:31:46.267135 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:31:46.267112    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dsjn6"]
Jul 28 13:31:46.269428 ip-10-0-129-177 kubenswrapper[2554]: W0728 13:31:46.269401    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9db2e34_b3d5_48ec_8e5b_c077bf6d6f7c.slice/crio-359dab112bd13beb1b01796ab03a7bb6f8724236fafffb8d9cb19083e747d965 WatchSource:0}: Error finding container 359dab112bd13beb1b01796ab03a7bb6f8724236fafffb8d9cb19083e747d965: Status 404 returned error can't find the container with id 359dab112bd13beb1b01796ab03a7bb6f8724236fafffb8d9cb19083e747d965
Jul 28 13:31:46.757329 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:31:46.757295    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dsjn6" event={"ID":"f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c","Type":"ContainerStarted","Data":"f008bd7f265d9bdfbd84f2f51f964aebcd104092f5b75e60c49e7265b8f9c565"}
Jul 28 13:31:46.757490 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:31:46.757355    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dsjn6" event={"ID":"f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c","Type":"ContainerStarted","Data":"359dab112bd13beb1b01796ab03a7bb6f8724236fafffb8d9cb19083e747d965"}
Jul 28 13:31:50.773370 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:31:50.773295    2554 generic.go:356] "Generic (PLEG): container finished" podID="f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c" containerID="f008bd7f265d9bdfbd84f2f51f964aebcd104092f5b75e60c49e7265b8f9c565" exitCode=0
Jul 28 13:31:50.773370 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:31:50.773308    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dsjn6" event={"ID":"f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c","Type":"ContainerDied","Data":"f008bd7f265d9bdfbd84f2f51f964aebcd104092f5b75e60c49e7265b8f9c565"}
Jul 28 13:31:50.774810 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:31:50.774791    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 13:31:58.812611 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:31:58.812577    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dsjn6" event={"ID":"f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c","Type":"ContainerStarted","Data":"a96ec67ddac198cb26911076937663277a4c574c45097f2bf2649b86a4498dac"}
Jul 28 13:31:58.813086 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:31:58.812649    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dsjn6" event={"ID":"f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c","Type":"ContainerStarted","Data":"33e2fee9cdbecd224607eedc7fa0a795af0ba2eb4d59c8ca886f0bebb226d48e"}
Jul 28 13:31:58.813086 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:31:58.812875    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dsjn6"
Jul 28 13:31:58.832787 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:31:58.832658    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dsjn6" podStartSLOduration=3.045350701 podStartE2EDuration="13.832640434s" totalImagesPullingTime="7.030836831s" totalInitContainerRuntime="3.756452902s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:31:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:31:58.83206506 +0000 UTC m=+2315.617940171" watchObservedRunningTime="2026-07-28 13:31:58.832640434 +0000 UTC m=+2315.618515546"
Jul 28 13:31:59.816921 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:31:59.816887    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dsjn6"
Jul 28 13:31:59.818052 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:31:59.818028    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dsjn6" podUID="f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 13:32:00.820784 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:32:00.820740    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dsjn6" podUID="f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 13:32:05.825832 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:32:05.825804    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dsjn6"
Jul 28 13:32:05.826396 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:32:05.826367    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dsjn6" podUID="f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 13:32:15.826741 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:32:15.826661    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dsjn6" podUID="f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 13:32:25.826387 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:32:25.826349    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dsjn6" podUID="f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 13:32:35.827223 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:32:35.827183    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dsjn6" podUID="f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 13:32:45.827143 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:32:45.827103    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dsjn6" podUID="f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 13:32:55.826864 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:32:55.826827    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dsjn6" podUID="f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 13:33:05.826659 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:33:05.826616    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dsjn6" podUID="f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 13:33:15.826955 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:33:15.826926    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dsjn6"
Jul 28 13:33:25.957144 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:33:25.957104    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dsjn6"]
Jul 28 13:33:25.957566 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:33:25.957417    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dsjn6" podUID="f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c" containerName="kserve-container" containerID="cri-o://33e2fee9cdbecd224607eedc7fa0a795af0ba2eb4d59c8ca886f0bebb226d48e" gracePeriod=30
Jul 28 13:33:25.957566 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:33:25.957459    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dsjn6" podUID="f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c" containerName="kube-rbac-proxy" containerID="cri-o://a96ec67ddac198cb26911076937663277a4c574c45097f2bf2649b86a4498dac" gracePeriod=30
Jul 28 13:33:26.169507 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:33:26.169473    2554 generic.go:356] "Generic (PLEG): container finished" podID="f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c" containerID="a96ec67ddac198cb26911076937663277a4c574c45097f2bf2649b86a4498dac" exitCode=2
Jul 28 13:33:26.169673 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:33:26.169555    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dsjn6" event={"ID":"f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c","Type":"ContainerDied","Data":"a96ec67ddac198cb26911076937663277a4c574c45097f2bf2649b86a4498dac"}
Jul 28 13:33:30.821723 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:33:30.821685    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dsjn6" podUID="f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Jul 28 13:33:31.007235 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:33:31.007205    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dsjn6"
Jul 28 13:33:31.116789 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:33:31.116703    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c-kserve-provision-location\") pod \"f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c\" (UID: \"f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c\") "
Jul 28 13:33:31.116789 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:33:31.116744    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c-kube-rbac-proxy-sar-config\") pod \"f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c\" (UID: \"f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c\") "
Jul 28 13:33:31.116789 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:33:31.116773    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4mfh\" (UniqueName: \"kubernetes.io/projected/f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c-kube-api-access-x4mfh\") pod \"f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c\" (UID: \"f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c\") "
Jul 28 13:33:31.117066 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:33:31.116929    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c-proxy-tls\") pod \"f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c\" (UID: \"f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c\") "
Jul 28 13:33:31.117066 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:33:31.116994    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c-cabundle-cert\") pod \"f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c\" (UID: \"f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c\") "
Jul 28 13:33:31.117168 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:33:31.117059    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c" (UID: "f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:33:31.117168 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:33:31.117090    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c" (UID: "f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:33:31.117391 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:33:31.117370    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c-kserve-provision-location\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:33:31.117465 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:33:31.117394    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:33:31.117465 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:33:31.117390    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c" (UID: "f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:33:31.119204 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:33:31.119177    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c-kube-api-access-x4mfh" (OuterVolumeSpecName: "kube-api-access-x4mfh") pod "f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c" (UID: "f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c"). InnerVolumeSpecName "kube-api-access-x4mfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:33:31.119421 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:33:31.119402    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c" (UID: "f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:33:31.190746 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:33:31.190713    2554 generic.go:356] "Generic (PLEG): container finished" podID="f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c" containerID="33e2fee9cdbecd224607eedc7fa0a795af0ba2eb4d59c8ca886f0bebb226d48e" exitCode=0
Jul 28 13:33:31.190875 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:33:31.190771    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dsjn6"
Jul 28 13:33:31.190875 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:33:31.190784    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dsjn6" event={"ID":"f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c","Type":"ContainerDied","Data":"33e2fee9cdbecd224607eedc7fa0a795af0ba2eb4d59c8ca886f0bebb226d48e"}
Jul 28 13:33:31.190875 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:33:31.190834    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dsjn6" event={"ID":"f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c","Type":"ContainerDied","Data":"359dab112bd13beb1b01796ab03a7bb6f8724236fafffb8d9cb19083e747d965"}
Jul 28 13:33:31.190875 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:33:31.190870    2554 scope.go:117] "RemoveContainer" containerID="a96ec67ddac198cb26911076937663277a4c574c45097f2bf2649b86a4498dac"
Jul 28 13:33:31.200720 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:33:31.200698    2554 scope.go:117] "RemoveContainer" containerID="33e2fee9cdbecd224607eedc7fa0a795af0ba2eb4d59c8ca886f0bebb226d48e"
Jul 28 13:33:31.208153 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:33:31.208095    2554 scope.go:117] "RemoveContainer" containerID="f008bd7f265d9bdfbd84f2f51f964aebcd104092f5b75e60c49e7265b8f9c565"
Jul 28 13:33:31.217038 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:33:31.217018    2554 scope.go:117] "RemoveContainer" containerID="a96ec67ddac198cb26911076937663277a4c574c45097f2bf2649b86a4498dac"
Jul 28 13:33:31.217355 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:33:31.217326    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a96ec67ddac198cb26911076937663277a4c574c45097f2bf2649b86a4498dac\": container with ID starting with a96ec67ddac198cb26911076937663277a4c574c45097f2bf2649b86a4498dac not found: ID does not exist" containerID="a96ec67ddac198cb26911076937663277a4c574c45097f2bf2649b86a4498dac"
Jul 28 13:33:31.217439 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:33:31.217364    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a96ec67ddac198cb26911076937663277a4c574c45097f2bf2649b86a4498dac"} err="failed to get container status \"a96ec67ddac198cb26911076937663277a4c574c45097f2bf2649b86a4498dac\": rpc error: code = NotFound desc = could not find container \"a96ec67ddac198cb26911076937663277a4c574c45097f2bf2649b86a4498dac\": container with ID starting with a96ec67ddac198cb26911076937663277a4c574c45097f2bf2649b86a4498dac not found: ID does not exist"
Jul 28 13:33:31.217439 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:33:31.217384    2554 scope.go:117] "RemoveContainer" containerID="33e2fee9cdbecd224607eedc7fa0a795af0ba2eb4d59c8ca886f0bebb226d48e"
Jul 28 13:33:31.217706 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:33:31.217682    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33e2fee9cdbecd224607eedc7fa0a795af0ba2eb4d59c8ca886f0bebb226d48e\": container with ID starting with 33e2fee9cdbecd224607eedc7fa0a795af0ba2eb4d59c8ca886f0bebb226d48e not found: ID does not exist" containerID="33e2fee9cdbecd224607eedc7fa0a795af0ba2eb4d59c8ca886f0bebb226d48e"
Jul 28 13:33:31.217757 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:33:31.217719    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33e2fee9cdbecd224607eedc7fa0a795af0ba2eb4d59c8ca886f0bebb226d48e"} err="failed to get container status \"33e2fee9cdbecd224607eedc7fa0a795af0ba2eb4d59c8ca886f0bebb226d48e\": rpc error: code = NotFound desc = could not find container \"33e2fee9cdbecd224607eedc7fa0a795af0ba2eb4d59c8ca886f0bebb226d48e\": container with ID starting with 33e2fee9cdbecd224607eedc7fa0a795af0ba2eb4d59c8ca886f0bebb226d48e not found: ID does not exist"
Jul 28 13:33:31.217757 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:33:31.217743    2554 scope.go:117] "RemoveContainer" containerID="f008bd7f265d9bdfbd84f2f51f964aebcd104092f5b75e60c49e7265b8f9c565"
Jul 28 13:33:31.217841 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:33:31.217785    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c-proxy-tls\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:33:31.217841 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:33:31.217800    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c-cabundle-cert\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:33:31.217841 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:33:31.217814    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4mfh\" (UniqueName: \"kubernetes.io/projected/f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c-kube-api-access-x4mfh\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:33:31.218042 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:33:31.218025    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f008bd7f265d9bdfbd84f2f51f964aebcd104092f5b75e60c49e7265b8f9c565\": container with ID starting with f008bd7f265d9bdfbd84f2f51f964aebcd104092f5b75e60c49e7265b8f9c565 not found: ID does not exist" containerID="f008bd7f265d9bdfbd84f2f51f964aebcd104092f5b75e60c49e7265b8f9c565"
Jul 28 13:33:31.218080 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:33:31.218047    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f008bd7f265d9bdfbd84f2f51f964aebcd104092f5b75e60c49e7265b8f9c565"} err="failed to get container status \"f008bd7f265d9bdfbd84f2f51f964aebcd104092f5b75e60c49e7265b8f9c565\": rpc error: code = NotFound desc = could not find container \"f008bd7f265d9bdfbd84f2f51f964aebcd104092f5b75e60c49e7265b8f9c565\": container with ID starting with f008bd7f265d9bdfbd84f2f51f964aebcd104092f5b75e60c49e7265b8f9c565 not found: ID does not exist"
Jul 28 13:33:31.218250 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:33:31.218237    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dsjn6"]
Jul 28 13:33:31.223990 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:33:31.223968    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dsjn6"]
Jul 28 13:33:31.801926 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:33:31.801893    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c" path="/var/lib/kubelet/pods/f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c/volumes"
Jul 28 13:35:06.908434 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:35:06.908378    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-hlg5s"]
Jul 28 13:35:06.909013 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:35:06.908707    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c" containerName="storage-initializer"
Jul 28 13:35:06.909013 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:35:06.908718    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c" containerName="storage-initializer"
Jul 28 13:35:06.909013 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:35:06.908729    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c" containerName="kube-rbac-proxy"
Jul 28 13:35:06.909013 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:35:06.908734    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c" containerName="kube-rbac-proxy"
Jul 28 13:35:06.909013 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:35:06.908746    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c" containerName="kserve-container"
Jul 28 13:35:06.909013 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:35:06.908752    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c" containerName="kserve-container"
Jul 28 13:35:06.909013 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:35:06.908815    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c" containerName="kserve-container"
Jul 28 13:35:06.909013 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:35:06.908824    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9db2e34-b3d5-48ec-8e5b-c077bf6d6f7c" containerName="kube-rbac-proxy"
Jul 28 13:35:06.912033 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:35:06.912016    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-hlg5s"
Jul 28 13:35:06.914356 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:35:06.914330    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
Jul 28 13:35:06.914549 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:35:06.914529    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:35:06.914662 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:35:06.914561    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rxfwd\""
Jul 28 13:35:06.914662 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:35:06.914596    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:35:06.914845 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:35:06.914829    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:35:06.915387 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:35:06.915371    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 28 13:35:06.915680 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:35:06.915663    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:35:06.925739 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:35:06.925717    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-hlg5s"]
Jul 28 13:35:07.003946 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:35:07.003909    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-hlg5s\" (UID: \"a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-hlg5s"
Jul 28 13:35:07.004120 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:35:07.003956    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-hlg5s\" (UID: \"a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-hlg5s"
Jul 28 13:35:07.004120 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:35:07.003982    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-hlg5s\" (UID: \"a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-hlg5s"
Jul 28 13:35:07.004120 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:35:07.004028    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-przsh\" (UniqueName: \"kubernetes.io/projected/a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd-kube-api-access-przsh\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-hlg5s\" (UID: \"a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-hlg5s"
Jul 28 13:35:07.004120 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:35:07.004096    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-hlg5s\" (UID: \"a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-hlg5s"
Jul 28 13:35:07.105122 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:35:07.105082    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-hlg5s\" (UID: \"a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-hlg5s"
Jul 28 13:35:07.105339 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:35:07.105183    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-hlg5s\" (UID: \"a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-hlg5s"
Jul 28 13:35:07.105339 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:35:07.105218    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-hlg5s\" (UID: \"a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-hlg5s"
Jul 28 13:35:07.105339 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:35:07.105239    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-hlg5s\" (UID: \"a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-hlg5s"
Jul 28 13:35:07.105529 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:35:07.105401    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-przsh\" (UniqueName: \"kubernetes.io/projected/a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd-kube-api-access-przsh\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-hlg5s\" (UID: \"a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-hlg5s"
Jul 28 13:35:07.105698 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:35:07.105673    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-hlg5s\" (UID: \"a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-hlg5s"
Jul 28 13:35:07.105883 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:35:07.105862    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-hlg5s\" (UID: \"a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-hlg5s"
Jul 28 13:35:07.105883 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:35:07.105875    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-hlg5s\" (UID: \"a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-hlg5s"
Jul 28 13:35:07.108051 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:35:07.108030    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-hlg5s\" (UID: \"a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-hlg5s"
Jul 28 13:35:07.114125 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:35:07.114106    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-przsh\" (UniqueName: \"kubernetes.io/projected/a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd-kube-api-access-przsh\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-hlg5s\" (UID: \"a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-hlg5s"
Jul 28 13:35:07.222466 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:35:07.222427    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-hlg5s"
Jul 28 13:35:07.355187 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:35:07.355154    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-hlg5s"]
Jul 28 13:35:07.358039 ip-10-0-129-177 kubenswrapper[2554]: W0728 13:35:07.358012    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6f0c853_0d99_4da3_a5c1_4d7daf58f1fd.slice/crio-dec951eeaf813f5e982cb2c5e3be6a03151cb1cb286acb35eea5219d49b8345c WatchSource:0}: Error finding container dec951eeaf813f5e982cb2c5e3be6a03151cb1cb286acb35eea5219d49b8345c: Status 404 returned error can't find the container with id dec951eeaf813f5e982cb2c5e3be6a03151cb1cb286acb35eea5219d49b8345c
Jul 28 13:35:07.565465 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:35:07.565356    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-hlg5s" event={"ID":"a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd","Type":"ContainerStarted","Data":"9c585869b0bf1f6149336156dc2e65c9584cf604db0e516e01f23716494a796b"}
Jul 28 13:35:07.565465 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:35:07.565426    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-hlg5s" event={"ID":"a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd","Type":"ContainerStarted","Data":"dec951eeaf813f5e982cb2c5e3be6a03151cb1cb286acb35eea5219d49b8345c"}
Jul 28 13:35:11.582145 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:35:11.582110    2554 generic.go:356] "Generic (PLEG): container finished" podID="a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd" containerID="9c585869b0bf1f6149336156dc2e65c9584cf604db0e516e01f23716494a796b" exitCode=0
Jul 28 13:35:11.582145 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:35:11.582148    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-hlg5s" event={"ID":"a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd","Type":"ContainerDied","Data":"9c585869b0bf1f6149336156dc2e65c9584cf604db0e516e01f23716494a796b"}
Jul 28 13:37:26.221492 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:37:26.221456    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-hlg5s" event={"ID":"a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd","Type":"ContainerStarted","Data":"0b191ac70ada938d294d20a30bec7ba52e8ccf5f4dc21742649885248b938e63"}
Jul 28 13:37:26.222011 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:37:26.221541    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-hlg5s" event={"ID":"a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd","Type":"ContainerStarted","Data":"2a187f9541b8ce97c94ad7527116124b775ed5e3794df94017de59027fda9cca"}
Jul 28 13:37:26.222011 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:37:26.221575    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-hlg5s"
Jul 28 13:37:26.269033 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:37:26.268969    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-hlg5s" podStartSLOduration=2.836938436 podStartE2EDuration="2m20.268948196s" totalImagesPullingTime="2m13.684543532s" totalInitContainerRuntime="3.747466228s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:35:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:37:26.268149733 +0000 UTC m=+2643.054024873" watchObservedRunningTime="2026-07-28 13:37:26.268948196 +0000 UTC m=+2643.054823310"
Jul 28 13:37:27.226389 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:37:27.226350    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-hlg5s"
Jul 28 13:37:33.235963 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:37:33.235927    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-hlg5s"
Jul 28 13:38:03.240356 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:03.240325    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-hlg5s"
Jul 28 13:38:07.145233 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:07.140338    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-hlg5s"]
Jul 28 13:38:07.145233 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:07.140833    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-hlg5s" podUID="a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd" containerName="kserve-container" containerID="cri-o://2a187f9541b8ce97c94ad7527116124b775ed5e3794df94017de59027fda9cca" gracePeriod=30
Jul 28 13:38:07.145233 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:07.141296    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-hlg5s" podUID="a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd" containerName="kube-rbac-proxy" containerID="cri-o://0b191ac70ada938d294d20a30bec7ba52e8ccf5f4dc21742649885248b938e63" gracePeriod=30
Jul 28 13:38:07.390739 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:07.390702    2554 generic.go:356] "Generic (PLEG): container finished" podID="a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd" containerID="0b191ac70ada938d294d20a30bec7ba52e8ccf5f4dc21742649885248b938e63" exitCode=2
Jul 28 13:38:07.390739 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:07.390752    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-hlg5s" event={"ID":"a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd","Type":"ContainerDied","Data":"0b191ac70ada938d294d20a30bec7ba52e8ccf5f4dc21742649885248b938e63"}
Jul 28 13:38:08.197871 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:08.197847    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-hlg5s"
Jul 28 13:38:08.324551 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:08.324476    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-przsh\" (UniqueName: \"kubernetes.io/projected/a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd-kube-api-access-przsh\") pod \"a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd\" (UID: \"a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd\") "
Jul 28 13:38:08.324551 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:08.324515    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd-cabundle-cert\") pod \"a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd\" (UID: \"a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd\") "
Jul 28 13:38:08.324787 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:08.324562    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd-proxy-tls\") pod \"a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd\" (UID: \"a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd\") "
Jul 28 13:38:08.324787 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:08.324621    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd-kube-rbac-proxy-sar-config\") pod \"a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd\" (UID: \"a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd\") "
Jul 28 13:38:08.324787 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:08.324657    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd-kserve-provision-location\") pod \"a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd\" (UID: \"a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd\") "
Jul 28 13:38:08.325069 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:08.325036    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd" (UID: "a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:38:08.325069 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:08.325045    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd" (UID: "a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:38:08.325069 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:08.325048    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd" (UID: "a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:38:08.326869 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:08.326844    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd" (UID: "a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:38:08.326947 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:08.326907    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd-kube-api-access-przsh" (OuterVolumeSpecName: "kube-api-access-przsh") pod "a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd" (UID: "a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd"). InnerVolumeSpecName "kube-api-access-przsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:38:08.396076 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:08.396036    2554 generic.go:356] "Generic (PLEG): container finished" podID="a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd" containerID="2a187f9541b8ce97c94ad7527116124b775ed5e3794df94017de59027fda9cca" exitCode=0
Jul 28 13:38:08.396223 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:08.396093    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-hlg5s"
Jul 28 13:38:08.396223 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:08.396110    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-hlg5s" event={"ID":"a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd","Type":"ContainerDied","Data":"2a187f9541b8ce97c94ad7527116124b775ed5e3794df94017de59027fda9cca"}
Jul 28 13:38:08.396223 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:08.396166    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-hlg5s" event={"ID":"a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd","Type":"ContainerDied","Data":"dec951eeaf813f5e982cb2c5e3be6a03151cb1cb286acb35eea5219d49b8345c"}
Jul 28 13:38:08.396223 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:08.396196    2554 scope.go:117] "RemoveContainer" containerID="0b191ac70ada938d294d20a30bec7ba52e8ccf5f4dc21742649885248b938e63"
Jul 28 13:38:08.406625 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:08.406545    2554 scope.go:117] "RemoveContainer" containerID="2a187f9541b8ce97c94ad7527116124b775ed5e3794df94017de59027fda9cca"
Jul 28 13:38:08.414314 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:08.414298    2554 scope.go:117] "RemoveContainer" containerID="9c585869b0bf1f6149336156dc2e65c9584cf604db0e516e01f23716494a796b"
Jul 28 13:38:08.421526 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:08.421502    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-hlg5s"]
Jul 28 13:38:08.421760 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:08.421748    2554 scope.go:117] "RemoveContainer" containerID="0b191ac70ada938d294d20a30bec7ba52e8ccf5f4dc21742649885248b938e63"
Jul 28 13:38:08.422021 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:38:08.422004    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b191ac70ada938d294d20a30bec7ba52e8ccf5f4dc21742649885248b938e63\": container with ID starting with 0b191ac70ada938d294d20a30bec7ba52e8ccf5f4dc21742649885248b938e63 not found: ID does not exist" containerID="0b191ac70ada938d294d20a30bec7ba52e8ccf5f4dc21742649885248b938e63"
Jul 28 13:38:08.422086 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:08.422029    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b191ac70ada938d294d20a30bec7ba52e8ccf5f4dc21742649885248b938e63"} err="failed to get container status \"0b191ac70ada938d294d20a30bec7ba52e8ccf5f4dc21742649885248b938e63\": rpc error: code = NotFound desc = could not find container \"0b191ac70ada938d294d20a30bec7ba52e8ccf5f4dc21742649885248b938e63\": container with ID starting with 0b191ac70ada938d294d20a30bec7ba52e8ccf5f4dc21742649885248b938e63 not found: ID does not exist"
Jul 28 13:38:08.422086 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:08.422044    2554 scope.go:117] "RemoveContainer" containerID="2a187f9541b8ce97c94ad7527116124b775ed5e3794df94017de59027fda9cca"
Jul 28 13:38:08.422288 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:38:08.422250    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a187f9541b8ce97c94ad7527116124b775ed5e3794df94017de59027fda9cca\": container with ID starting with 2a187f9541b8ce97c94ad7527116124b775ed5e3794df94017de59027fda9cca not found: ID does not exist" containerID="2a187f9541b8ce97c94ad7527116124b775ed5e3794df94017de59027fda9cca"
Jul 28 13:38:08.422412 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:08.422388    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a187f9541b8ce97c94ad7527116124b775ed5e3794df94017de59027fda9cca"} err="failed to get container status \"2a187f9541b8ce97c94ad7527116124b775ed5e3794df94017de59027fda9cca\": rpc error: code = NotFound desc = could not find container \"2a187f9541b8ce97c94ad7527116124b775ed5e3794df94017de59027fda9cca\": container with ID starting with 2a187f9541b8ce97c94ad7527116124b775ed5e3794df94017de59027fda9cca not found: ID does not exist"
Jul 28 13:38:08.422412 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:08.422413    2554 scope.go:117] "RemoveContainer" containerID="9c585869b0bf1f6149336156dc2e65c9584cf604db0e516e01f23716494a796b"
Jul 28 13:38:08.422728 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:38:08.422660    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c585869b0bf1f6149336156dc2e65c9584cf604db0e516e01f23716494a796b\": container with ID starting with 9c585869b0bf1f6149336156dc2e65c9584cf604db0e516e01f23716494a796b not found: ID does not exist" containerID="9c585869b0bf1f6149336156dc2e65c9584cf604db0e516e01f23716494a796b"
Jul 28 13:38:08.422797 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:08.422734    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c585869b0bf1f6149336156dc2e65c9584cf604db0e516e01f23716494a796b"} err="failed to get container status \"9c585869b0bf1f6149336156dc2e65c9584cf604db0e516e01f23716494a796b\": rpc error: code = NotFound desc = could not find container \"9c585869b0bf1f6149336156dc2e65c9584cf604db0e516e01f23716494a796b\": container with ID starting with 9c585869b0bf1f6149336156dc2e65c9584cf604db0e516e01f23716494a796b not found: ID does not exist"
Jul 28 13:38:08.425676 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:08.425655    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:38:08.425752 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:08.425681    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd-kserve-provision-location\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:38:08.425752 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:08.425698    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-przsh\" (UniqueName: \"kubernetes.io/projected/a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd-kube-api-access-przsh\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:38:08.425752 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:08.425710    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd-cabundle-cert\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:38:08.425752 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:08.425723    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd-proxy-tls\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:38:08.426000 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:08.425984    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-hlg5s"]
Jul 28 13:38:09.802360 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:09.802330    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd" path="/var/lib/kubelet/pods/a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd/volumes"
Jul 28 13:38:12.255042 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:12.255001    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-8pv6c"]
Jul 28 13:38:12.255545 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:12.255525    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd" containerName="kserve-container"
Jul 28 13:38:12.255628 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:12.255548    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd" containerName="kserve-container"
Jul 28 13:38:12.255628 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:12.255573    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd" containerName="storage-initializer"
Jul 28 13:38:12.255628 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:12.255582    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd" containerName="storage-initializer"
Jul 28 13:38:12.255628 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:12.255607    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd" containerName="kube-rbac-proxy"
Jul 28 13:38:12.255628 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:12.255615    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd" containerName="kube-rbac-proxy"
Jul 28 13:38:12.255877 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:12.255740    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd" containerName="kube-rbac-proxy"
Jul 28 13:38:12.255877 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:12.255756    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6f0c853-0d99-4da3-a5c1-4d7daf58f1fd" containerName="kserve-container"
Jul 28 13:38:12.260818 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:12.260797    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-8pv6c"
Jul 28 13:38:12.264680 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:12.264653    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:38:12.264807 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:12.264750    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:38:12.265057 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:12.265042    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:38:12.265112 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:12.265043    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
Jul 28 13:38:12.265533 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:12.265515    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
Jul 28 13:38:12.265651 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:12.265559    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:38:12.265651 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:12.265559    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rxfwd\""
Jul 28 13:38:12.269637 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:12.269615    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-8pv6c"]
Jul 28 13:38:12.361487 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:12.361460    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq5nn\" (UniqueName: \"kubernetes.io/projected/02f6c7f8-0c27-4fcd-a3d8-23b081e01e42-kube-api-access-zq5nn\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-8pv6c\" (UID: \"02f6c7f8-0c27-4fcd-a3d8-23b081e01e42\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-8pv6c"
Jul 28 13:38:12.361645 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:12.361522    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02f6c7f8-0c27-4fcd-a3d8-23b081e01e42-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-8pv6c\" (UID: \"02f6c7f8-0c27-4fcd-a3d8-23b081e01e42\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-8pv6c"
Jul 28 13:38:12.361645 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:12.361584    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02f6c7f8-0c27-4fcd-a3d8-23b081e01e42-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-8pv6c\" (UID: \"02f6c7f8-0c27-4fcd-a3d8-23b081e01e42\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-8pv6c"
Jul 28 13:38:12.361645 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:12.361627    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/02f6c7f8-0c27-4fcd-a3d8-23b081e01e42-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-8pv6c\" (UID: \"02f6c7f8-0c27-4fcd-a3d8-23b081e01e42\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-8pv6c"
Jul 28 13:38:12.361745 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:12.361662    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02f6c7f8-0c27-4fcd-a3d8-23b081e01e42-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-8pv6c\" (UID: \"02f6c7f8-0c27-4fcd-a3d8-23b081e01e42\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-8pv6c"
Jul 28 13:38:12.462164 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:12.462127    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02f6c7f8-0c27-4fcd-a3d8-23b081e01e42-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-8pv6c\" (UID: \"02f6c7f8-0c27-4fcd-a3d8-23b081e01e42\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-8pv6c"
Jul 28 13:38:12.462388 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:12.462171    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02f6c7f8-0c27-4fcd-a3d8-23b081e01e42-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-8pv6c\" (UID: \"02f6c7f8-0c27-4fcd-a3d8-23b081e01e42\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-8pv6c"
Jul 28 13:38:12.462388 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:12.462212    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/02f6c7f8-0c27-4fcd-a3d8-23b081e01e42-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-8pv6c\" (UID: \"02f6c7f8-0c27-4fcd-a3d8-23b081e01e42\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-8pv6c"
Jul 28 13:38:12.462388 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:12.462253    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02f6c7f8-0c27-4fcd-a3d8-23b081e01e42-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-8pv6c\" (UID: \"02f6c7f8-0c27-4fcd-a3d8-23b081e01e42\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-8pv6c"
Jul 28 13:38:12.462388 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:12.462302    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq5nn\" (UniqueName: \"kubernetes.io/projected/02f6c7f8-0c27-4fcd-a3d8-23b081e01e42-kube-api-access-zq5nn\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-8pv6c\" (UID: \"02f6c7f8-0c27-4fcd-a3d8-23b081e01e42\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-8pv6c"
Jul 28 13:38:12.462830 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:12.462799    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02f6c7f8-0c27-4fcd-a3d8-23b081e01e42-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-8pv6c\" (UID: \"02f6c7f8-0c27-4fcd-a3d8-23b081e01e42\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-8pv6c"
Jul 28 13:38:12.463014 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:12.462965    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02f6c7f8-0c27-4fcd-a3d8-23b081e01e42-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-8pv6c\" (UID: \"02f6c7f8-0c27-4fcd-a3d8-23b081e01e42\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-8pv6c"
Jul 28 13:38:12.463014 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:12.462997    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/02f6c7f8-0c27-4fcd-a3d8-23b081e01e42-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-8pv6c\" (UID: \"02f6c7f8-0c27-4fcd-a3d8-23b081e01e42\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-8pv6c"
Jul 28 13:38:12.464912 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:12.464884    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02f6c7f8-0c27-4fcd-a3d8-23b081e01e42-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-8pv6c\" (UID: \"02f6c7f8-0c27-4fcd-a3d8-23b081e01e42\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-8pv6c"
Jul 28 13:38:12.471111 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:12.471086    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq5nn\" (UniqueName: \"kubernetes.io/projected/02f6c7f8-0c27-4fcd-a3d8-23b081e01e42-kube-api-access-zq5nn\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-8pv6c\" (UID: \"02f6c7f8-0c27-4fcd-a3d8-23b081e01e42\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-8pv6c"
Jul 28 13:38:12.572947 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:12.572872    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-8pv6c"
Jul 28 13:38:12.706724 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:12.706698    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-8pv6c"]
Jul 28 13:38:12.709159 ip-10-0-129-177 kubenswrapper[2554]: W0728 13:38:12.709132    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02f6c7f8_0c27_4fcd_a3d8_23b081e01e42.slice/crio-51ada1ac623611d0976bb161a00d23353db6880b4d726d7a5ec2d914223851e8 WatchSource:0}: Error finding container 51ada1ac623611d0976bb161a00d23353db6880b4d726d7a5ec2d914223851e8: Status 404 returned error can't find the container with id 51ada1ac623611d0976bb161a00d23353db6880b4d726d7a5ec2d914223851e8
Jul 28 13:38:13.418997 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:13.418955    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-8pv6c" event={"ID":"02f6c7f8-0c27-4fcd-a3d8-23b081e01e42","Type":"ContainerStarted","Data":"44f3c73602bb971afb9740cb1b555298c185e24bd8e7de2ebd026624044a5961"}
Jul 28 13:38:13.419437 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:13.419031    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-8pv6c" event={"ID":"02f6c7f8-0c27-4fcd-a3d8-23b081e01e42","Type":"ContainerStarted","Data":"51ada1ac623611d0976bb161a00d23353db6880b4d726d7a5ec2d914223851e8"}
Jul 28 13:38:17.437463 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:17.437432    2554 generic.go:356] "Generic (PLEG): container finished" podID="02f6c7f8-0c27-4fcd-a3d8-23b081e01e42" containerID="44f3c73602bb971afb9740cb1b555298c185e24bd8e7de2ebd026624044a5961" exitCode=0
Jul 28 13:38:17.437953 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:17.437488    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-8pv6c" event={"ID":"02f6c7f8-0c27-4fcd-a3d8-23b081e01e42","Type":"ContainerDied","Data":"44f3c73602bb971afb9740cb1b555298c185e24bd8e7de2ebd026624044a5961"}
Jul 28 13:38:18.443166 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:18.443128    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-8pv6c" event={"ID":"02f6c7f8-0c27-4fcd-a3d8-23b081e01e42","Type":"ContainerStarted","Data":"d8e1e93e5d19f89b2095dfa4a0c8c519e58681a33d7730635fc3e9ddb5d0c5ed"}
Jul 28 13:38:18.443565 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:18.443188    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-8pv6c" event={"ID":"02f6c7f8-0c27-4fcd-a3d8-23b081e01e42","Type":"ContainerStarted","Data":"5227ed3934cc6b4d64dbe9d25db3dad66079fe6571871d4be901baf71a642c7c"}
Jul 28 13:38:18.443565 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:18.443416    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-8pv6c"
Jul 28 13:38:18.465213 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:18.465169    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-8pv6c" podStartSLOduration=2.75903598 podStartE2EDuration="6.465156113s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.706120133s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:38:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:38:18.463445123 +0000 UTC m=+2695.249320284" watchObservedRunningTime="2026-07-28 13:38:18.465156113 +0000 UTC m=+2695.251031224"
Jul 28 13:38:19.447943 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:19.447909    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-8pv6c"
Jul 28 13:38:19.449004 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:19.448971    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-8pv6c" podUID="02f6c7f8-0c27-4fcd-a3d8-23b081e01e42" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 13:38:20.452346 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:20.452308    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-8pv6c" podUID="02f6c7f8-0c27-4fcd-a3d8-23b081e01e42" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 13:38:25.456708 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:25.456675    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-8pv6c"
Jul 28 13:38:25.457411 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:25.457393    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-8pv6c"
Jul 28 13:38:32.398792 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:32.398759    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-8pv6c"]
Jul 28 13:38:32.399246 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:32.399007    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-8pv6c" podUID="02f6c7f8-0c27-4fcd-a3d8-23b081e01e42" containerName="kserve-container" containerID="cri-o://5227ed3934cc6b4d64dbe9d25db3dad66079fe6571871d4be901baf71a642c7c" gracePeriod=30
Jul 28 13:38:32.399246 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:32.399049    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-8pv6c" podUID="02f6c7f8-0c27-4fcd-a3d8-23b081e01e42" containerName="kube-rbac-proxy" containerID="cri-o://d8e1e93e5d19f89b2095dfa4a0c8c519e58681a33d7730635fc3e9ddb5d0c5ed" gracePeriod=30
Jul 28 13:38:33.045596 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:33.045570    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-8pv6c"
Jul 28 13:38:33.130127 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:33.130049    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02f6c7f8-0c27-4fcd-a3d8-23b081e01e42-kube-rbac-proxy-sar-config\") pod \"02f6c7f8-0c27-4fcd-a3d8-23b081e01e42\" (UID: \"02f6c7f8-0c27-4fcd-a3d8-23b081e01e42\") "
Jul 28 13:38:33.130127 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:33.130090    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02f6c7f8-0c27-4fcd-a3d8-23b081e01e42-proxy-tls\") pod \"02f6c7f8-0c27-4fcd-a3d8-23b081e01e42\" (UID: \"02f6c7f8-0c27-4fcd-a3d8-23b081e01e42\") "
Jul 28 13:38:33.130127 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:33.130118    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/02f6c7f8-0c27-4fcd-a3d8-23b081e01e42-cabundle-cert\") pod \"02f6c7f8-0c27-4fcd-a3d8-23b081e01e42\" (UID: \"02f6c7f8-0c27-4fcd-a3d8-23b081e01e42\") "
Jul 28 13:38:33.130398 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:33.130244    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zq5nn\" (UniqueName: \"kubernetes.io/projected/02f6c7f8-0c27-4fcd-a3d8-23b081e01e42-kube-api-access-zq5nn\") pod \"02f6c7f8-0c27-4fcd-a3d8-23b081e01e42\" (UID: \"02f6c7f8-0c27-4fcd-a3d8-23b081e01e42\") "
Jul 28 13:38:33.130398 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:33.130355    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02f6c7f8-0c27-4fcd-a3d8-23b081e01e42-kserve-provision-location\") pod \"02f6c7f8-0c27-4fcd-a3d8-23b081e01e42\" (UID: \"02f6c7f8-0c27-4fcd-a3d8-23b081e01e42\") "
Jul 28 13:38:33.130502 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:33.130474    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02f6c7f8-0c27-4fcd-a3d8-23b081e01e42-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "02f6c7f8-0c27-4fcd-a3d8-23b081e01e42" (UID: "02f6c7f8-0c27-4fcd-a3d8-23b081e01e42"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:38:33.130558 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:33.130481    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02f6c7f8-0c27-4fcd-a3d8-23b081e01e42-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "02f6c7f8-0c27-4fcd-a3d8-23b081e01e42" (UID: "02f6c7f8-0c27-4fcd-a3d8-23b081e01e42"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:38:33.130676 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:33.130660    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/02f6c7f8-0c27-4fcd-a3d8-23b081e01e42-cabundle-cert\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:38:33.130744 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:33.130677    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02f6c7f8-0c27-4fcd-a3d8-23b081e01e42-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:38:33.130744 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:33.130713    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02f6c7f8-0c27-4fcd-a3d8-23b081e01e42-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "02f6c7f8-0c27-4fcd-a3d8-23b081e01e42" (UID: "02f6c7f8-0c27-4fcd-a3d8-23b081e01e42"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:38:33.132357 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:33.132337    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02f6c7f8-0c27-4fcd-a3d8-23b081e01e42-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "02f6c7f8-0c27-4fcd-a3d8-23b081e01e42" (UID: "02f6c7f8-0c27-4fcd-a3d8-23b081e01e42"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:38:33.132448 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:33.132382    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02f6c7f8-0c27-4fcd-a3d8-23b081e01e42-kube-api-access-zq5nn" (OuterVolumeSpecName: "kube-api-access-zq5nn") pod "02f6c7f8-0c27-4fcd-a3d8-23b081e01e42" (UID: "02f6c7f8-0c27-4fcd-a3d8-23b081e01e42"). InnerVolumeSpecName "kube-api-access-zq5nn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:38:33.231515 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:33.231484    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq5nn\" (UniqueName: \"kubernetes.io/projected/02f6c7f8-0c27-4fcd-a3d8-23b081e01e42-kube-api-access-zq5nn\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:38:33.231515 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:33.231510    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02f6c7f8-0c27-4fcd-a3d8-23b081e01e42-kserve-provision-location\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:38:33.231515 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:33.231520    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02f6c7f8-0c27-4fcd-a3d8-23b081e01e42-proxy-tls\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:38:33.505289 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:33.505242    2554 generic.go:356] "Generic (PLEG): container finished" podID="02f6c7f8-0c27-4fcd-a3d8-23b081e01e42" containerID="d8e1e93e5d19f89b2095dfa4a0c8c519e58681a33d7730635fc3e9ddb5d0c5ed" exitCode=2
Jul 28 13:38:33.505289 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:33.505267    2554 generic.go:356] "Generic (PLEG): container finished" podID="02f6c7f8-0c27-4fcd-a3d8-23b081e01e42" containerID="5227ed3934cc6b4d64dbe9d25db3dad66079fe6571871d4be901baf71a642c7c" exitCode=0
Jul 28 13:38:33.505716 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:33.505316    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-8pv6c"
Jul 28 13:38:33.505716 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:33.505312    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-8pv6c" event={"ID":"02f6c7f8-0c27-4fcd-a3d8-23b081e01e42","Type":"ContainerDied","Data":"d8e1e93e5d19f89b2095dfa4a0c8c519e58681a33d7730635fc3e9ddb5d0c5ed"}
Jul 28 13:38:33.505716 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:33.505445    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-8pv6c" event={"ID":"02f6c7f8-0c27-4fcd-a3d8-23b081e01e42","Type":"ContainerDied","Data":"5227ed3934cc6b4d64dbe9d25db3dad66079fe6571871d4be901baf71a642c7c"}
Jul 28 13:38:33.505716 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:33.505467    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-8pv6c" event={"ID":"02f6c7f8-0c27-4fcd-a3d8-23b081e01e42","Type":"ContainerDied","Data":"51ada1ac623611d0976bb161a00d23353db6880b4d726d7a5ec2d914223851e8"}
Jul 28 13:38:33.505716 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:33.505493    2554 scope.go:117] "RemoveContainer" containerID="d8e1e93e5d19f89b2095dfa4a0c8c519e58681a33d7730635fc3e9ddb5d0c5ed"
Jul 28 13:38:33.513652 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:33.513636    2554 scope.go:117] "RemoveContainer" containerID="5227ed3934cc6b4d64dbe9d25db3dad66079fe6571871d4be901baf71a642c7c"
Jul 28 13:38:33.520887 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:33.520871    2554 scope.go:117] "RemoveContainer" containerID="44f3c73602bb971afb9740cb1b555298c185e24bd8e7de2ebd026624044a5961"
Jul 28 13:38:33.528134 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:33.528110    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-8pv6c"]
Jul 28 13:38:33.528347 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:33.528333    2554 scope.go:117] "RemoveContainer" containerID="d8e1e93e5d19f89b2095dfa4a0c8c519e58681a33d7730635fc3e9ddb5d0c5ed"
Jul 28 13:38:33.528584 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:38:33.528565    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8e1e93e5d19f89b2095dfa4a0c8c519e58681a33d7730635fc3e9ddb5d0c5ed\": container with ID starting with d8e1e93e5d19f89b2095dfa4a0c8c519e58681a33d7730635fc3e9ddb5d0c5ed not found: ID does not exist" containerID="d8e1e93e5d19f89b2095dfa4a0c8c519e58681a33d7730635fc3e9ddb5d0c5ed"
Jul 28 13:38:33.528655 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:33.528589    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8e1e93e5d19f89b2095dfa4a0c8c519e58681a33d7730635fc3e9ddb5d0c5ed"} err="failed to get container status \"d8e1e93e5d19f89b2095dfa4a0c8c519e58681a33d7730635fc3e9ddb5d0c5ed\": rpc error: code = NotFound desc = could not find container \"d8e1e93e5d19f89b2095dfa4a0c8c519e58681a33d7730635fc3e9ddb5d0c5ed\": container with ID starting with d8e1e93e5d19f89b2095dfa4a0c8c519e58681a33d7730635fc3e9ddb5d0c5ed not found: ID does not exist"
Jul 28 13:38:33.528655 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:33.528604    2554 scope.go:117] "RemoveContainer" containerID="5227ed3934cc6b4d64dbe9d25db3dad66079fe6571871d4be901baf71a642c7c"
Jul 28 13:38:33.528824 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:38:33.528806    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5227ed3934cc6b4d64dbe9d25db3dad66079fe6571871d4be901baf71a642c7c\": container with ID starting with 5227ed3934cc6b4d64dbe9d25db3dad66079fe6571871d4be901baf71a642c7c not found: ID does not exist" containerID="5227ed3934cc6b4d64dbe9d25db3dad66079fe6571871d4be901baf71a642c7c"
Jul 28 13:38:33.528865 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:33.528833    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5227ed3934cc6b4d64dbe9d25db3dad66079fe6571871d4be901baf71a642c7c"} err="failed to get container status \"5227ed3934cc6b4d64dbe9d25db3dad66079fe6571871d4be901baf71a642c7c\": rpc error: code = NotFound desc = could not find container \"5227ed3934cc6b4d64dbe9d25db3dad66079fe6571871d4be901baf71a642c7c\": container with ID starting with 5227ed3934cc6b4d64dbe9d25db3dad66079fe6571871d4be901baf71a642c7c not found: ID does not exist"
Jul 28 13:38:33.528865 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:33.528855    2554 scope.go:117] "RemoveContainer" containerID="44f3c73602bb971afb9740cb1b555298c185e24bd8e7de2ebd026624044a5961"
Jul 28 13:38:33.529088 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:38:33.529072    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44f3c73602bb971afb9740cb1b555298c185e24bd8e7de2ebd026624044a5961\": container with ID starting with 44f3c73602bb971afb9740cb1b555298c185e24bd8e7de2ebd026624044a5961 not found: ID does not exist" containerID="44f3c73602bb971afb9740cb1b555298c185e24bd8e7de2ebd026624044a5961"
Jul 28 13:38:33.529130 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:33.529093    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44f3c73602bb971afb9740cb1b555298c185e24bd8e7de2ebd026624044a5961"} err="failed to get container status \"44f3c73602bb971afb9740cb1b555298c185e24bd8e7de2ebd026624044a5961\": rpc error: code = NotFound desc = could not find container \"44f3c73602bb971afb9740cb1b555298c185e24bd8e7de2ebd026624044a5961\": container with ID starting with 44f3c73602bb971afb9740cb1b555298c185e24bd8e7de2ebd026624044a5961 not found: ID does not exist"
Jul 28 13:38:33.529130 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:33.529106    2554 scope.go:117] "RemoveContainer" containerID="d8e1e93e5d19f89b2095dfa4a0c8c519e58681a33d7730635fc3e9ddb5d0c5ed"
Jul 28 13:38:33.529322 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:33.529306    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8e1e93e5d19f89b2095dfa4a0c8c519e58681a33d7730635fc3e9ddb5d0c5ed"} err="failed to get container status \"d8e1e93e5d19f89b2095dfa4a0c8c519e58681a33d7730635fc3e9ddb5d0c5ed\": rpc error: code = NotFound desc = could not find container \"d8e1e93e5d19f89b2095dfa4a0c8c519e58681a33d7730635fc3e9ddb5d0c5ed\": container with ID starting with d8e1e93e5d19f89b2095dfa4a0c8c519e58681a33d7730635fc3e9ddb5d0c5ed not found: ID does not exist"
Jul 28 13:38:33.529372 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:33.529322    2554 scope.go:117] "RemoveContainer" containerID="5227ed3934cc6b4d64dbe9d25db3dad66079fe6571871d4be901baf71a642c7c"
Jul 28 13:38:33.529523 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:33.529505    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5227ed3934cc6b4d64dbe9d25db3dad66079fe6571871d4be901baf71a642c7c"} err="failed to get container status \"5227ed3934cc6b4d64dbe9d25db3dad66079fe6571871d4be901baf71a642c7c\": rpc error: code = NotFound desc = could not find container \"5227ed3934cc6b4d64dbe9d25db3dad66079fe6571871d4be901baf71a642c7c\": container with ID starting with 5227ed3934cc6b4d64dbe9d25db3dad66079fe6571871d4be901baf71a642c7c not found: ID does not exist"
Jul 28 13:38:33.529566 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:33.529525    2554 scope.go:117] "RemoveContainer" containerID="44f3c73602bb971afb9740cb1b555298c185e24bd8e7de2ebd026624044a5961"
Jul 28 13:38:33.529737 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:33.529713    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44f3c73602bb971afb9740cb1b555298c185e24bd8e7de2ebd026624044a5961"} err="failed to get container status \"44f3c73602bb971afb9740cb1b555298c185e24bd8e7de2ebd026624044a5961\": rpc error: code = NotFound desc = could not find container \"44f3c73602bb971afb9740cb1b555298c185e24bd8e7de2ebd026624044a5961\": container with ID starting with 44f3c73602bb971afb9740cb1b555298c185e24bd8e7de2ebd026624044a5961 not found: ID does not exist"
Jul 28 13:38:33.533713 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:33.533693    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-8pv6c"]
Jul 28 13:38:33.803492 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:33.803415    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02f6c7f8-0c27-4fcd-a3d8-23b081e01e42" path="/var/lib/kubelet/pods/02f6c7f8-0c27-4fcd-a3d8-23b081e01e42/volumes"
Jul 28 13:38:37.493381 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:37.493343    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pcq24"]
Jul 28 13:38:37.493748 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:37.493655    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02f6c7f8-0c27-4fcd-a3d8-23b081e01e42" containerName="kube-rbac-proxy"
Jul 28 13:38:37.493748 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:37.493665    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="02f6c7f8-0c27-4fcd-a3d8-23b081e01e42" containerName="kube-rbac-proxy"
Jul 28 13:38:37.493748 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:37.493675    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02f6c7f8-0c27-4fcd-a3d8-23b081e01e42" containerName="kserve-container"
Jul 28 13:38:37.493748 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:37.493680    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="02f6c7f8-0c27-4fcd-a3d8-23b081e01e42" containerName="kserve-container"
Jul 28 13:38:37.493748 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:37.493694    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02f6c7f8-0c27-4fcd-a3d8-23b081e01e42" containerName="storage-initializer"
Jul 28 13:38:37.493748 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:37.493701    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="02f6c7f8-0c27-4fcd-a3d8-23b081e01e42" containerName="storage-initializer"
Jul 28 13:38:37.493927 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:37.493753    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="02f6c7f8-0c27-4fcd-a3d8-23b081e01e42" containerName="kube-rbac-proxy"
Jul 28 13:38:37.493927 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:37.493761    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="02f6c7f8-0c27-4fcd-a3d8-23b081e01e42" containerName="kserve-container"
Jul 28 13:38:37.498777 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:37.498755    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pcq24"
Jul 28 13:38:37.501700 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:37.501674    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:38:37.501932 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:37.501912    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:38:37.502038 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:37.501992    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:38:37.502095 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:37.502054    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rxfwd\""
Jul 28 13:38:37.502095 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:37.501916    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 28 13:38:37.502197 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:37.502161    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:38:37.502494 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:37.502477    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
Jul 28 13:38:37.510104 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:37.510084    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pcq24"]
Jul 28 13:38:37.570548 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:37.570517    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23faf56e-41c8-4354-92c1-0859ef676070-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-pcq24\" (UID: \"23faf56e-41c8-4354-92c1-0859ef676070\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pcq24"
Jul 28 13:38:37.570548 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:37.570551    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23faf56e-41c8-4354-92c1-0859ef676070-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-pcq24\" (UID: \"23faf56e-41c8-4354-92c1-0859ef676070\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pcq24"
Jul 28 13:38:37.570751 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:37.570577    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f87cz\" (UniqueName: \"kubernetes.io/projected/23faf56e-41c8-4354-92c1-0859ef676070-kube-api-access-f87cz\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-pcq24\" (UID: \"23faf56e-41c8-4354-92c1-0859ef676070\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pcq24"
Jul 28 13:38:37.570751 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:37.570647    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23faf56e-41c8-4354-92c1-0859ef676070-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-pcq24\" (UID: \"23faf56e-41c8-4354-92c1-0859ef676070\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pcq24"
Jul 28 13:38:37.570751 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:37.570674    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23faf56e-41c8-4354-92c1-0859ef676070-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-pcq24\" (UID: \"23faf56e-41c8-4354-92c1-0859ef676070\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pcq24"
Jul 28 13:38:37.671988 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:37.671954    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23faf56e-41c8-4354-92c1-0859ef676070-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-pcq24\" (UID: \"23faf56e-41c8-4354-92c1-0859ef676070\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pcq24"
Jul 28 13:38:37.671988 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:37.671991    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23faf56e-41c8-4354-92c1-0859ef676070-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-pcq24\" (UID: \"23faf56e-41c8-4354-92c1-0859ef676070\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pcq24"
Jul 28 13:38:37.672210 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:37.672050    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23faf56e-41c8-4354-92c1-0859ef676070-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-pcq24\" (UID: \"23faf56e-41c8-4354-92c1-0859ef676070\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pcq24"
Jul 28 13:38:37.672210 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:37.672069    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23faf56e-41c8-4354-92c1-0859ef676070-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-pcq24\" (UID: \"23faf56e-41c8-4354-92c1-0859ef676070\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pcq24"
Jul 28 13:38:37.672210 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:37.672094    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f87cz\" (UniqueName: \"kubernetes.io/projected/23faf56e-41c8-4354-92c1-0859ef676070-kube-api-access-f87cz\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-pcq24\" (UID: \"23faf56e-41c8-4354-92c1-0859ef676070\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pcq24"
Jul 28 13:38:37.672581 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:37.672557    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23faf56e-41c8-4354-92c1-0859ef676070-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-pcq24\" (UID: \"23faf56e-41c8-4354-92c1-0859ef676070\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pcq24"
Jul 28 13:38:37.672840 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:37.672819    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23faf56e-41c8-4354-92c1-0859ef676070-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-pcq24\" (UID: \"23faf56e-41c8-4354-92c1-0859ef676070\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pcq24"
Jul 28 13:38:37.672878 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:37.672819    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23faf56e-41c8-4354-92c1-0859ef676070-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-pcq24\" (UID: \"23faf56e-41c8-4354-92c1-0859ef676070\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pcq24"
Jul 28 13:38:37.674817 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:37.674798    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23faf56e-41c8-4354-92c1-0859ef676070-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-pcq24\" (UID: \"23faf56e-41c8-4354-92c1-0859ef676070\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pcq24"
Jul 28 13:38:37.681994 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:37.681974    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f87cz\" (UniqueName: \"kubernetes.io/projected/23faf56e-41c8-4354-92c1-0859ef676070-kube-api-access-f87cz\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-pcq24\" (UID: \"23faf56e-41c8-4354-92c1-0859ef676070\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pcq24"
Jul 28 13:38:37.811323 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:37.811232    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pcq24"
Jul 28 13:38:37.945297 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:37.945210    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pcq24"]
Jul 28 13:38:37.948347 ip-10-0-129-177 kubenswrapper[2554]: W0728 13:38:37.948315    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23faf56e_41c8_4354_92c1_0859ef676070.slice/crio-7004ad0ecf38b3f080edda21231277ed2fdf077e51e37544ff0a08a70a067dcb WatchSource:0}: Error finding container 7004ad0ecf38b3f080edda21231277ed2fdf077e51e37544ff0a08a70a067dcb: Status 404 returned error can't find the container with id 7004ad0ecf38b3f080edda21231277ed2fdf077e51e37544ff0a08a70a067dcb
Jul 28 13:38:38.528259 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:38.528219    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pcq24" event={"ID":"23faf56e-41c8-4354-92c1-0859ef676070","Type":"ContainerStarted","Data":"b0138580048bd8e3816164a3828243c9c1951411bd10db9e4dbb2147c9cca12a"}
Jul 28 13:38:38.528693 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:38.528298    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pcq24" event={"ID":"23faf56e-41c8-4354-92c1-0859ef676070","Type":"ContainerStarted","Data":"7004ad0ecf38b3f080edda21231277ed2fdf077e51e37544ff0a08a70a067dcb"}
Jul 28 13:38:42.546191 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:42.546155    2554 generic.go:356] "Generic (PLEG): container finished" podID="23faf56e-41c8-4354-92c1-0859ef676070" containerID="b0138580048bd8e3816164a3828243c9c1951411bd10db9e4dbb2147c9cca12a" exitCode=0
Jul 28 13:38:42.546677 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:42.546234    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pcq24" event={"ID":"23faf56e-41c8-4354-92c1-0859ef676070","Type":"ContainerDied","Data":"b0138580048bd8e3816164a3828243c9c1951411bd10db9e4dbb2147c9cca12a"}
Jul 28 13:38:43.553823 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:43.553791    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pcq24" event={"ID":"23faf56e-41c8-4354-92c1-0859ef676070","Type":"ContainerStarted","Data":"729aac710bb600fe065d50d0cd93123528ae92dd0eaa14b6f18ddca3ade6a475"}
Jul 28 13:38:43.554266 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:43.553851    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pcq24" event={"ID":"23faf56e-41c8-4354-92c1-0859ef676070","Type":"ContainerStarted","Data":"25b7b07308e43f3e409bec91ecaf36414ce74061f96cdf14e1f7dbc98c1c94ac"}
Jul 28 13:38:43.554266 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:43.554069    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pcq24"
Jul 28 13:38:43.554266 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:43.554121    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pcq24"
Jul 28 13:38:43.575651 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:43.575603    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pcq24" podStartSLOduration=2.7013103899999997 podStartE2EDuration="6.57558839s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.874278s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:38:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:38:43.574293621 +0000 UTC m=+2720.360168726" watchObservedRunningTime="2026-07-28 13:38:43.57558839 +0000 UTC m=+2720.361463501"
Jul 28 13:38:49.563644 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:38:49.563609    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pcq24"
Jul 28 13:39:19.567195 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:19.567164    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pcq24"
Jul 28 13:39:27.604961 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:27.604928    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pcq24"]
Jul 28 13:39:27.605383 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:27.605234    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pcq24" podUID="23faf56e-41c8-4354-92c1-0859ef676070" containerName="kserve-container" containerID="cri-o://25b7b07308e43f3e409bec91ecaf36414ce74061f96cdf14e1f7dbc98c1c94ac" gracePeriod=30
Jul 28 13:39:27.605383 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:27.605293    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pcq24" podUID="23faf56e-41c8-4354-92c1-0859ef676070" containerName="kube-rbac-proxy" containerID="cri-o://729aac710bb600fe065d50d0cd93123528ae92dd0eaa14b6f18ddca3ade6a475" gracePeriod=30
Jul 28 13:39:27.738699 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:27.738666    2554 generic.go:356] "Generic (PLEG): container finished" podID="23faf56e-41c8-4354-92c1-0859ef676070" containerID="729aac710bb600fe065d50d0cd93123528ae92dd0eaa14b6f18ddca3ade6a475" exitCode=2
Jul 28 13:39:27.738852 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:27.738714    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pcq24" event={"ID":"23faf56e-41c8-4354-92c1-0859ef676070","Type":"ContainerDied","Data":"729aac710bb600fe065d50d0cd93123528ae92dd0eaa14b6f18ddca3ade6a475"}
Jul 28 13:39:28.745255 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:28.745223    2554 generic.go:356] "Generic (PLEG): container finished" podID="23faf56e-41c8-4354-92c1-0859ef676070" containerID="25b7b07308e43f3e409bec91ecaf36414ce74061f96cdf14e1f7dbc98c1c94ac" exitCode=0
Jul 28 13:39:28.745666 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:28.745295    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pcq24" event={"ID":"23faf56e-41c8-4354-92c1-0859ef676070","Type":"ContainerDied","Data":"25b7b07308e43f3e409bec91ecaf36414ce74061f96cdf14e1f7dbc98c1c94ac"}
Jul 28 13:39:28.745666 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:28.745370    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pcq24" event={"ID":"23faf56e-41c8-4354-92c1-0859ef676070","Type":"ContainerDied","Data":"7004ad0ecf38b3f080edda21231277ed2fdf077e51e37544ff0a08a70a067dcb"}
Jul 28 13:39:28.745666 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:28.745411    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7004ad0ecf38b3f080edda21231277ed2fdf077e51e37544ff0a08a70a067dcb"
Jul 28 13:39:28.758984 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:28.758966    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pcq24"
Jul 28 13:39:28.792020 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:28.791990    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23faf56e-41c8-4354-92c1-0859ef676070-kube-rbac-proxy-sar-config\") pod \"23faf56e-41c8-4354-92c1-0859ef676070\" (UID: \"23faf56e-41c8-4354-92c1-0859ef676070\") "
Jul 28 13:39:28.792180 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:28.792045    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f87cz\" (UniqueName: \"kubernetes.io/projected/23faf56e-41c8-4354-92c1-0859ef676070-kube-api-access-f87cz\") pod \"23faf56e-41c8-4354-92c1-0859ef676070\" (UID: \"23faf56e-41c8-4354-92c1-0859ef676070\") "
Jul 28 13:39:28.792180 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:28.792120    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23faf56e-41c8-4354-92c1-0859ef676070-proxy-tls\") pod \"23faf56e-41c8-4354-92c1-0859ef676070\" (UID: \"23faf56e-41c8-4354-92c1-0859ef676070\") "
Jul 28 13:39:28.792316 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:28.792182    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23faf56e-41c8-4354-92c1-0859ef676070-cabundle-cert\") pod \"23faf56e-41c8-4354-92c1-0859ef676070\" (UID: \"23faf56e-41c8-4354-92c1-0859ef676070\") "
Jul 28 13:39:28.792316 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:28.792229    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23faf56e-41c8-4354-92c1-0859ef676070-kserve-provision-location\") pod \"23faf56e-41c8-4354-92c1-0859ef676070\" (UID: \"23faf56e-41c8-4354-92c1-0859ef676070\") "
Jul 28 13:39:28.792437 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:28.792407    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23faf56e-41c8-4354-92c1-0859ef676070-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "23faf56e-41c8-4354-92c1-0859ef676070" (UID: "23faf56e-41c8-4354-92c1-0859ef676070"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:39:28.792497 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:28.792482    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23faf56e-41c8-4354-92c1-0859ef676070-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:39:28.792629 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:28.792605    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23faf56e-41c8-4354-92c1-0859ef676070-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "23faf56e-41c8-4354-92c1-0859ef676070" (UID: "23faf56e-41c8-4354-92c1-0859ef676070"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:39:28.792773 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:28.792753    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23faf56e-41c8-4354-92c1-0859ef676070-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "23faf56e-41c8-4354-92c1-0859ef676070" (UID: "23faf56e-41c8-4354-92c1-0859ef676070"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:39:28.794834 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:28.794813    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23faf56e-41c8-4354-92c1-0859ef676070-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "23faf56e-41c8-4354-92c1-0859ef676070" (UID: "23faf56e-41c8-4354-92c1-0859ef676070"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:39:28.794916 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:28.794849    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23faf56e-41c8-4354-92c1-0859ef676070-kube-api-access-f87cz" (OuterVolumeSpecName: "kube-api-access-f87cz") pod "23faf56e-41c8-4354-92c1-0859ef676070" (UID: "23faf56e-41c8-4354-92c1-0859ef676070"). InnerVolumeSpecName "kube-api-access-f87cz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:39:28.893919 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:28.893827    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23faf56e-41c8-4354-92c1-0859ef676070-kserve-provision-location\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:39:28.893919 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:28.893868    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f87cz\" (UniqueName: \"kubernetes.io/projected/23faf56e-41c8-4354-92c1-0859ef676070-kube-api-access-f87cz\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:39:28.893919 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:28.893884    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23faf56e-41c8-4354-92c1-0859ef676070-proxy-tls\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:39:28.893919 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:28.893896    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23faf56e-41c8-4354-92c1-0859ef676070-cabundle-cert\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:39:29.748935 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:29.748901    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pcq24"
Jul 28 13:39:29.770684 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:29.770655    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pcq24"]
Jul 28 13:39:29.774551 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:29.774525    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pcq24"]
Jul 28 13:39:29.801606 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:29.801582    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23faf56e-41c8-4354-92c1-0859ef676070" path="/var/lib/kubelet/pods/23faf56e-41c8-4354-92c1-0859ef676070/volumes"
Jul 28 13:39:32.774158 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:32.774112    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-nt9sr"]
Jul 28 13:39:32.774570 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:32.774504    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23faf56e-41c8-4354-92c1-0859ef676070" containerName="kube-rbac-proxy"
Jul 28 13:39:32.774570 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:32.774520    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="23faf56e-41c8-4354-92c1-0859ef676070" containerName="kube-rbac-proxy"
Jul 28 13:39:32.774570 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:32.774535    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23faf56e-41c8-4354-92c1-0859ef676070" containerName="storage-initializer"
Jul 28 13:39:32.774570 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:32.774541    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="23faf56e-41c8-4354-92c1-0859ef676070" containerName="storage-initializer"
Jul 28 13:39:32.774570 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:32.774553    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23faf56e-41c8-4354-92c1-0859ef676070" containerName="kserve-container"
Jul 28 13:39:32.774570 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:32.774559    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="23faf56e-41c8-4354-92c1-0859ef676070" containerName="kserve-container"
Jul 28 13:39:32.774746 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:32.774617    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="23faf56e-41c8-4354-92c1-0859ef676070" containerName="kube-rbac-proxy"
Jul 28 13:39:32.774746 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:32.774629    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="23faf56e-41c8-4354-92c1-0859ef676070" containerName="kserve-container"
Jul 28 13:39:32.778036 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:32.778008    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-nt9sr"
Jul 28 13:39:32.781616 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:32.781590    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
Jul 28 13:39:32.781754 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:32.781592    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
Jul 28 13:39:32.781754 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:32.781598    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:39:32.781866 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:32.781843    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:39:32.782408 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:32.782390    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:39:32.782526 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:32.782444    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:39:32.782526 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:32.782464    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rxfwd\""
Jul 28 13:39:32.790109 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:32.790088    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-nt9sr"]
Jul 28 13:39:32.825731 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:32.825701    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8b6g\" (UniqueName: \"kubernetes.io/projected/e985bb71-fcba-4a21-98a1-630086bf9559-kube-api-access-l8b6g\") pod \"isvc-sklearn-mcp-predictor-5996957568-nt9sr\" (UID: \"e985bb71-fcba-4a21-98a1-630086bf9559\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-nt9sr"
Jul 28 13:39:32.825862 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:32.825760    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e985bb71-fcba-4a21-98a1-630086bf9559-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-5996957568-nt9sr\" (UID: \"e985bb71-fcba-4a21-98a1-630086bf9559\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-nt9sr"
Jul 28 13:39:32.825862 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:32.825790    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e985bb71-fcba-4a21-98a1-630086bf9559-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-5996957568-nt9sr\" (UID: \"e985bb71-fcba-4a21-98a1-630086bf9559\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-nt9sr"
Jul 28 13:39:32.825862 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:32.825816    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e985bb71-fcba-4a21-98a1-630086bf9559-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-5996957568-nt9sr\" (UID: \"e985bb71-fcba-4a21-98a1-630086bf9559\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-nt9sr"
Jul 28 13:39:32.825862 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:32.825846    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e985bb71-fcba-4a21-98a1-630086bf9559-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-5996957568-nt9sr\" (UID: \"e985bb71-fcba-4a21-98a1-630086bf9559\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-nt9sr"
Jul 28 13:39:32.926553 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:32.926524    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8b6g\" (UniqueName: \"kubernetes.io/projected/e985bb71-fcba-4a21-98a1-630086bf9559-kube-api-access-l8b6g\") pod \"isvc-sklearn-mcp-predictor-5996957568-nt9sr\" (UID: \"e985bb71-fcba-4a21-98a1-630086bf9559\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-nt9sr"
Jul 28 13:39:32.926722 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:32.926582    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e985bb71-fcba-4a21-98a1-630086bf9559-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-5996957568-nt9sr\" (UID: \"e985bb71-fcba-4a21-98a1-630086bf9559\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-nt9sr"
Jul 28 13:39:32.926722 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:32.926607    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e985bb71-fcba-4a21-98a1-630086bf9559-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-5996957568-nt9sr\" (UID: \"e985bb71-fcba-4a21-98a1-630086bf9559\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-nt9sr"
Jul 28 13:39:32.926722 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:32.926634    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e985bb71-fcba-4a21-98a1-630086bf9559-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-5996957568-nt9sr\" (UID: \"e985bb71-fcba-4a21-98a1-630086bf9559\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-nt9sr"
Jul 28 13:39:32.926722 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:32.926659    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e985bb71-fcba-4a21-98a1-630086bf9559-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-5996957568-nt9sr\" (UID: \"e985bb71-fcba-4a21-98a1-630086bf9559\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-nt9sr"
Jul 28 13:39:32.927334 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:32.927307    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e985bb71-fcba-4a21-98a1-630086bf9559-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-5996957568-nt9sr\" (UID: \"e985bb71-fcba-4a21-98a1-630086bf9559\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-nt9sr"
Jul 28 13:39:32.927415 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:32.927331    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e985bb71-fcba-4a21-98a1-630086bf9559-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-5996957568-nt9sr\" (UID: \"e985bb71-fcba-4a21-98a1-630086bf9559\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-nt9sr"
Jul 28 13:39:32.927453 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:32.927417    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e985bb71-fcba-4a21-98a1-630086bf9559-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-5996957568-nt9sr\" (UID: \"e985bb71-fcba-4a21-98a1-630086bf9559\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-nt9sr"
Jul 28 13:39:32.929124 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:32.929104    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e985bb71-fcba-4a21-98a1-630086bf9559-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-5996957568-nt9sr\" (UID: \"e985bb71-fcba-4a21-98a1-630086bf9559\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-nt9sr"
Jul 28 13:39:32.936061 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:32.936044    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8b6g\" (UniqueName: \"kubernetes.io/projected/e985bb71-fcba-4a21-98a1-630086bf9559-kube-api-access-l8b6g\") pod \"isvc-sklearn-mcp-predictor-5996957568-nt9sr\" (UID: \"e985bb71-fcba-4a21-98a1-630086bf9559\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-nt9sr"
Jul 28 13:39:33.089227 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:33.089145    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-nt9sr"
Jul 28 13:39:33.226129 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:33.226100    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-nt9sr"]
Jul 28 13:39:33.228175 ip-10-0-129-177 kubenswrapper[2554]: W0728 13:39:33.228149    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode985bb71_fcba_4a21_98a1_630086bf9559.slice/crio-f8e1fe75d8242202055a98f1ecd871af85087a7d099bd5bbac20ccd1c7dfae8b WatchSource:0}: Error finding container f8e1fe75d8242202055a98f1ecd871af85087a7d099bd5bbac20ccd1c7dfae8b: Status 404 returned error can't find the container with id f8e1fe75d8242202055a98f1ecd871af85087a7d099bd5bbac20ccd1c7dfae8b
Jul 28 13:39:33.765772 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:33.765736    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-nt9sr" event={"ID":"e985bb71-fcba-4a21-98a1-630086bf9559","Type":"ContainerStarted","Data":"98026b83eb68a092ba0986bc8f895209aca9bb695f494ef8979c5a00b6631e35"}
Jul 28 13:39:33.765935 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:33.765832    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-nt9sr" event={"ID":"e985bb71-fcba-4a21-98a1-630086bf9559","Type":"ContainerStarted","Data":"f8e1fe75d8242202055a98f1ecd871af85087a7d099bd5bbac20ccd1c7dfae8b"}
Jul 28 13:39:37.784185 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:37.784146    2554 generic.go:356] "Generic (PLEG): container finished" podID="e985bb71-fcba-4a21-98a1-630086bf9559" containerID="98026b83eb68a092ba0986bc8f895209aca9bb695f494ef8979c5a00b6631e35" exitCode=0
Jul 28 13:39:37.784579 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:37.784220    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-nt9sr" event={"ID":"e985bb71-fcba-4a21-98a1-630086bf9559","Type":"ContainerDied","Data":"98026b83eb68a092ba0986bc8f895209aca9bb695f494ef8979c5a00b6631e35"}
Jul 28 13:39:37.846046 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:37.846027    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 13:39:38.793134 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:38.793101    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-nt9sr" event={"ID":"e985bb71-fcba-4a21-98a1-630086bf9559","Type":"ContainerStarted","Data":"d2947a1012b023f107ea2853e53060baf70e67e20811792fd613a276a7461ab1"}
Jul 28 13:39:40.805589 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:40.805556    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-nt9sr" event={"ID":"e985bb71-fcba-4a21-98a1-630086bf9559","Type":"ContainerStarted","Data":"c42b6b49a70470641bf68ea4d8a20e7eb296cf8011e681e08a7eae9df471643e"}
Jul 28 13:39:40.806062 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:40.805642    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-nt9sr" event={"ID":"e985bb71-fcba-4a21-98a1-630086bf9559","Type":"ContainerStarted","Data":"b2c2f9304a1eac70ba97b7250ff5126831313a4b29b8fa1740cb6a009186c27b"}
Jul 28 13:39:40.806062 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:40.805746    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-nt9sr"
Jul 28 13:39:40.806062 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:40.805827    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-nt9sr"
Jul 28 13:39:40.831918 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:40.831872    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-nt9sr" podStartSLOduration=2.821523841 podStartE2EDuration="8.831854753s" totalImagesPullingTime="2.180355682s" totalInitContainerRuntime="3.82997523s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:39:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:39:40.829862224 +0000 UTC m=+2777.615737362" watchObservedRunningTime="2026-07-28 13:39:40.831854753 +0000 UTC m=+2777.617729864"
Jul 28 13:39:41.809920 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:41.809882    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-nt9sr"
Jul 28 13:39:47.819032 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:39:47.818957    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-nt9sr"
Jul 28 13:40:07.819969 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:40:07.819927    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-nt9sr" podUID="e985bb71-fcba-4a21-98a1-630086bf9559" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.37:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 13:40:17.820959 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:40:17.820929    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-nt9sr"
Jul 28 13:40:47.822234 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:40:47.822209    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-nt9sr"
Jul 28 13:40:52.900705 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:40:52.900670    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-nt9sr"]
Jul 28 13:40:52.901101 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:40:52.900958    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-nt9sr" podUID="e985bb71-fcba-4a21-98a1-630086bf9559" containerName="kserve-container" containerID="cri-o://d2947a1012b023f107ea2853e53060baf70e67e20811792fd613a276a7461ab1" gracePeriod=30
Jul 28 13:40:52.901101 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:40:52.901011    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-nt9sr" podUID="e985bb71-fcba-4a21-98a1-630086bf9559" containerName="kube-rbac-proxy" containerID="cri-o://c42b6b49a70470641bf68ea4d8a20e7eb296cf8011e681e08a7eae9df471643e" gracePeriod=30
Jul 28 13:40:52.901203 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:40:52.901023    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-nt9sr" podUID="e985bb71-fcba-4a21-98a1-630086bf9559" containerName="kserve-agent" containerID="cri-o://b2c2f9304a1eac70ba97b7250ff5126831313a4b29b8fa1740cb6a009186c27b" gracePeriod=30
Jul 28 13:40:53.121604 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:40:53.121566    2554 generic.go:356] "Generic (PLEG): container finished" podID="e985bb71-fcba-4a21-98a1-630086bf9559" containerID="c42b6b49a70470641bf68ea4d8a20e7eb296cf8011e681e08a7eae9df471643e" exitCode=2
Jul 28 13:40:53.121757 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:40:53.121631    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-nt9sr" event={"ID":"e985bb71-fcba-4a21-98a1-630086bf9559","Type":"ContainerDied","Data":"c42b6b49a70470641bf68ea4d8a20e7eb296cf8011e681e08a7eae9df471643e"}
Jul 28 13:40:55.132658 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:40:55.132618    2554 generic.go:356] "Generic (PLEG): container finished" podID="e985bb71-fcba-4a21-98a1-630086bf9559" containerID="d2947a1012b023f107ea2853e53060baf70e67e20811792fd613a276a7461ab1" exitCode=0
Jul 28 13:40:55.132995 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:40:55.132693    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-nt9sr" event={"ID":"e985bb71-fcba-4a21-98a1-630086bf9559","Type":"ContainerDied","Data":"d2947a1012b023f107ea2853e53060baf70e67e20811792fd613a276a7461ab1"}
Jul 28 13:40:57.814385 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:40:57.814347    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-nt9sr" podUID="e985bb71-fcba-4a21-98a1-630086bf9559" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Jul 28 13:40:57.819575 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:40:57.819551    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-nt9sr" podUID="e985bb71-fcba-4a21-98a1-630086bf9559" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.37:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 13:41:02.814950 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:41:02.814909    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-nt9sr" podUID="e985bb71-fcba-4a21-98a1-630086bf9559" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Jul 28 13:41:07.814485 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:41:07.814441    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-nt9sr" podUID="e985bb71-fcba-4a21-98a1-630086bf9559" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Jul 28 13:41:07.814937 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:41:07.814566    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-nt9sr"
Jul 28 13:41:07.819950 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:41:07.819926    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-nt9sr" podUID="e985bb71-fcba-4a21-98a1-630086bf9559" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.37:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 13:41:12.814237 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:41:12.814187    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-nt9sr" podUID="e985bb71-fcba-4a21-98a1-630086bf9559" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Jul 28 13:41:17.814780 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:41:17.814701    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-nt9sr" podUID="e985bb71-fcba-4a21-98a1-630086bf9559" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Jul 28 13:41:17.820151 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:41:17.820126    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-nt9sr" podUID="e985bb71-fcba-4a21-98a1-630086bf9559" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.37:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 13:41:17.820246 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:41:17.820236    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-nt9sr"
Jul 28 13:41:22.814333 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:41:22.814291    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-nt9sr" podUID="e985bb71-fcba-4a21-98a1-630086bf9559" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Jul 28 13:41:23.090465 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:41:23.090441    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-nt9sr"
Jul 28 13:41:23.140122 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:41:23.140088    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e985bb71-fcba-4a21-98a1-630086bf9559-kserve-provision-location\") pod \"e985bb71-fcba-4a21-98a1-630086bf9559\" (UID: \"e985bb71-fcba-4a21-98a1-630086bf9559\") "
Jul 28 13:41:23.140312 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:41:23.140142    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e985bb71-fcba-4a21-98a1-630086bf9559-proxy-tls\") pod \"e985bb71-fcba-4a21-98a1-630086bf9559\" (UID: \"e985bb71-fcba-4a21-98a1-630086bf9559\") "
Jul 28 13:41:23.140312 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:41:23.140161    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e985bb71-fcba-4a21-98a1-630086bf9559-cabundle-cert\") pod \"e985bb71-fcba-4a21-98a1-630086bf9559\" (UID: \"e985bb71-fcba-4a21-98a1-630086bf9559\") "
Jul 28 13:41:23.140312 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:41:23.140176    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e985bb71-fcba-4a21-98a1-630086bf9559-kube-rbac-proxy-sar-config\") pod \"e985bb71-fcba-4a21-98a1-630086bf9559\" (UID: \"e985bb71-fcba-4a21-98a1-630086bf9559\") "
Jul 28 13:41:23.140312 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:41:23.140211    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8b6g\" (UniqueName: \"kubernetes.io/projected/e985bb71-fcba-4a21-98a1-630086bf9559-kube-api-access-l8b6g\") pod \"e985bb71-fcba-4a21-98a1-630086bf9559\" (UID: \"e985bb71-fcba-4a21-98a1-630086bf9559\") "
Jul 28 13:41:23.140533 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:41:23.140473    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e985bb71-fcba-4a21-98a1-630086bf9559-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e985bb71-fcba-4a21-98a1-630086bf9559" (UID: "e985bb71-fcba-4a21-98a1-630086bf9559"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:41:23.140618 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:41:23.140592    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e985bb71-fcba-4a21-98a1-630086bf9559-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e985bb71-fcba-4a21-98a1-630086bf9559" (UID: "e985bb71-fcba-4a21-98a1-630086bf9559"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:41:23.140725 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:41:23.140590    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e985bb71-fcba-4a21-98a1-630086bf9559-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e985bb71-fcba-4a21-98a1-630086bf9559" (UID: "e985bb71-fcba-4a21-98a1-630086bf9559"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:41:23.142444 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:41:23.142426    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e985bb71-fcba-4a21-98a1-630086bf9559-kube-api-access-l8b6g" (OuterVolumeSpecName: "kube-api-access-l8b6g") pod "e985bb71-fcba-4a21-98a1-630086bf9559" (UID: "e985bb71-fcba-4a21-98a1-630086bf9559"). InnerVolumeSpecName "kube-api-access-l8b6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:41:23.142551 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:41:23.142533    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e985bb71-fcba-4a21-98a1-630086bf9559-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e985bb71-fcba-4a21-98a1-630086bf9559" (UID: "e985bb71-fcba-4a21-98a1-630086bf9559"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:41:23.240921 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:41:23.240883    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e985bb71-fcba-4a21-98a1-630086bf9559-cabundle-cert\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:41:23.240921 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:41:23.240915    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e985bb71-fcba-4a21-98a1-630086bf9559-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:41:23.240921 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:41:23.240927    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8b6g\" (UniqueName: \"kubernetes.io/projected/e985bb71-fcba-4a21-98a1-630086bf9559-kube-api-access-l8b6g\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:41:23.241146 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:41:23.240940    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e985bb71-fcba-4a21-98a1-630086bf9559-kserve-provision-location\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:41:23.241146 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:41:23.240952    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e985bb71-fcba-4a21-98a1-630086bf9559-proxy-tls\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:41:23.250606 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:41:23.250574    2554 generic.go:356] "Generic (PLEG): container finished" podID="e985bb71-fcba-4a21-98a1-630086bf9559" containerID="b2c2f9304a1eac70ba97b7250ff5126831313a4b29b8fa1740cb6a009186c27b" exitCode=137
Jul 28 13:41:23.250741 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:41:23.250614    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-nt9sr" event={"ID":"e985bb71-fcba-4a21-98a1-630086bf9559","Type":"ContainerDied","Data":"b2c2f9304a1eac70ba97b7250ff5126831313a4b29b8fa1740cb6a009186c27b"}
Jul 28 13:41:23.250741 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:41:23.250631    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-nt9sr"
Jul 28 13:41:23.250741 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:41:23.250684    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-nt9sr" event={"ID":"e985bb71-fcba-4a21-98a1-630086bf9559","Type":"ContainerDied","Data":"f8e1fe75d8242202055a98f1ecd871af85087a7d099bd5bbac20ccd1c7dfae8b"}
Jul 28 13:41:23.250741 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:41:23.250733    2554 scope.go:117] "RemoveContainer" containerID="c42b6b49a70470641bf68ea4d8a20e7eb296cf8011e681e08a7eae9df471643e"
Jul 28 13:41:23.259180 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:41:23.259163    2554 scope.go:117] "RemoveContainer" containerID="b2c2f9304a1eac70ba97b7250ff5126831313a4b29b8fa1740cb6a009186c27b"
Jul 28 13:41:23.266142 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:41:23.266123    2554 scope.go:117] "RemoveContainer" containerID="d2947a1012b023f107ea2853e53060baf70e67e20811792fd613a276a7461ab1"
Jul 28 13:41:23.273150 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:41:23.273135    2554 scope.go:117] "RemoveContainer" containerID="98026b83eb68a092ba0986bc8f895209aca9bb695f494ef8979c5a00b6631e35"
Jul 28 13:41:23.276984 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:41:23.276963    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-nt9sr"]
Jul 28 13:41:23.279792 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:41:23.279767    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-nt9sr"]
Jul 28 13:41:23.280944 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:41:23.280929    2554 scope.go:117] "RemoveContainer" containerID="c42b6b49a70470641bf68ea4d8a20e7eb296cf8011e681e08a7eae9df471643e"
Jul 28 13:41:23.281333 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:41:23.281314    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c42b6b49a70470641bf68ea4d8a20e7eb296cf8011e681e08a7eae9df471643e\": container with ID starting with c42b6b49a70470641bf68ea4d8a20e7eb296cf8011e681e08a7eae9df471643e not found: ID does not exist" containerID="c42b6b49a70470641bf68ea4d8a20e7eb296cf8011e681e08a7eae9df471643e"
Jul 28 13:41:23.281502 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:41:23.281341    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c42b6b49a70470641bf68ea4d8a20e7eb296cf8011e681e08a7eae9df471643e"} err="failed to get container status \"c42b6b49a70470641bf68ea4d8a20e7eb296cf8011e681e08a7eae9df471643e\": rpc error: code = NotFound desc = could not find container \"c42b6b49a70470641bf68ea4d8a20e7eb296cf8011e681e08a7eae9df471643e\": container with ID starting with c42b6b49a70470641bf68ea4d8a20e7eb296cf8011e681e08a7eae9df471643e not found: ID does not exist"
Jul 28 13:41:23.281502 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:41:23.281359    2554 scope.go:117] "RemoveContainer" containerID="b2c2f9304a1eac70ba97b7250ff5126831313a4b29b8fa1740cb6a009186c27b"
Jul 28 13:41:23.281606 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:41:23.281565    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2c2f9304a1eac70ba97b7250ff5126831313a4b29b8fa1740cb6a009186c27b\": container with ID starting with b2c2f9304a1eac70ba97b7250ff5126831313a4b29b8fa1740cb6a009186c27b not found: ID does not exist" containerID="b2c2f9304a1eac70ba97b7250ff5126831313a4b29b8fa1740cb6a009186c27b"
Jul 28 13:41:23.281606 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:41:23.281587    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2c2f9304a1eac70ba97b7250ff5126831313a4b29b8fa1740cb6a009186c27b"} err="failed to get container status \"b2c2f9304a1eac70ba97b7250ff5126831313a4b29b8fa1740cb6a009186c27b\": rpc error: code = NotFound desc = could not find container \"b2c2f9304a1eac70ba97b7250ff5126831313a4b29b8fa1740cb6a009186c27b\": container with ID starting with b2c2f9304a1eac70ba97b7250ff5126831313a4b29b8fa1740cb6a009186c27b not found: ID does not exist"
Jul 28 13:41:23.281606 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:41:23.281602    2554 scope.go:117] "RemoveContainer" containerID="d2947a1012b023f107ea2853e53060baf70e67e20811792fd613a276a7461ab1"
Jul 28 13:41:23.281860 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:41:23.281841    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2947a1012b023f107ea2853e53060baf70e67e20811792fd613a276a7461ab1\": container with ID starting with d2947a1012b023f107ea2853e53060baf70e67e20811792fd613a276a7461ab1 not found: ID does not exist" containerID="d2947a1012b023f107ea2853e53060baf70e67e20811792fd613a276a7461ab1"
Jul 28 13:41:23.281908 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:41:23.281867    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2947a1012b023f107ea2853e53060baf70e67e20811792fd613a276a7461ab1"} err="failed to get container status \"d2947a1012b023f107ea2853e53060baf70e67e20811792fd613a276a7461ab1\": rpc error: code = NotFound desc = could not find container \"d2947a1012b023f107ea2853e53060baf70e67e20811792fd613a276a7461ab1\": container with ID starting with d2947a1012b023f107ea2853e53060baf70e67e20811792fd613a276a7461ab1 not found: ID does not exist"
Jul 28 13:41:23.281908 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:41:23.281882    2554 scope.go:117] "RemoveContainer" containerID="98026b83eb68a092ba0986bc8f895209aca9bb695f494ef8979c5a00b6631e35"
Jul 28 13:41:23.282117 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:41:23.282101    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98026b83eb68a092ba0986bc8f895209aca9bb695f494ef8979c5a00b6631e35\": container with ID starting with 98026b83eb68a092ba0986bc8f895209aca9bb695f494ef8979c5a00b6631e35 not found: ID does not exist" containerID="98026b83eb68a092ba0986bc8f895209aca9bb695f494ef8979c5a00b6631e35"
Jul 28 13:41:23.282156 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:41:23.282122    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98026b83eb68a092ba0986bc8f895209aca9bb695f494ef8979c5a00b6631e35"} err="failed to get container status \"98026b83eb68a092ba0986bc8f895209aca9bb695f494ef8979c5a00b6631e35\": rpc error: code = NotFound desc = could not find container \"98026b83eb68a092ba0986bc8f895209aca9bb695f494ef8979c5a00b6631e35\": container with ID starting with 98026b83eb68a092ba0986bc8f895209aca9bb695f494ef8979c5a00b6631e35 not found: ID does not exist"
Jul 28 13:41:23.802988 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:41:23.802955    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e985bb71-fcba-4a21-98a1-630086bf9559" path="/var/lib/kubelet/pods/e985bb71-fcba-4a21-98a1-630086bf9559/volumes"
Jul 28 13:43:21.606369 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:43:21.606338    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk"]
Jul 28 13:43:21.606810 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:43:21.606701    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e985bb71-fcba-4a21-98a1-630086bf9559" containerName="kserve-agent"
Jul 28 13:43:21.606810 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:43:21.606715    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="e985bb71-fcba-4a21-98a1-630086bf9559" containerName="kserve-agent"
Jul 28 13:43:21.606810 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:43:21.606742    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e985bb71-fcba-4a21-98a1-630086bf9559" containerName="storage-initializer"
Jul 28 13:43:21.606810 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:43:21.606750    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="e985bb71-fcba-4a21-98a1-630086bf9559" containerName="storage-initializer"
Jul 28 13:43:21.606810 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:43:21.606760    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e985bb71-fcba-4a21-98a1-630086bf9559" containerName="kserve-container"
Jul 28 13:43:21.606810 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:43:21.606770    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="e985bb71-fcba-4a21-98a1-630086bf9559" containerName="kserve-container"
Jul 28 13:43:21.606810 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:43:21.606780    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e985bb71-fcba-4a21-98a1-630086bf9559" containerName="kube-rbac-proxy"
Jul 28 13:43:21.606810 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:43:21.606788    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="e985bb71-fcba-4a21-98a1-630086bf9559" containerName="kube-rbac-proxy"
Jul 28 13:43:21.607167 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:43:21.606863    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="e985bb71-fcba-4a21-98a1-630086bf9559" containerName="kserve-agent"
Jul 28 13:43:21.607167 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:43:21.606877    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="e985bb71-fcba-4a21-98a1-630086bf9559" containerName="kserve-container"
Jul 28 13:43:21.607167 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:43:21.606888    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="e985bb71-fcba-4a21-98a1-630086bf9559" containerName="kube-rbac-proxy"
Jul 28 13:43:21.610004 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:43:21.609987    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk"
Jul 28 13:43:21.612767 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:43:21.612746    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:43:21.612897 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:43:21.612783    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:43:21.612897 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:43:21.612748    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:43:21.612897 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:43:21.612866    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rxfwd\""
Jul 28 13:43:21.613150 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:43:21.613134    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
Jul 28 13:43:21.613548 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:43:21.613530    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
Jul 28 13:43:21.613548 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:43:21.613542    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:43:21.621563 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:43:21.621543    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk"]
Jul 28 13:43:21.789007 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:43:21.788977    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/085238f5-7c1e-4cb0-bee8-88d5587c8fa6-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk\" (UID: \"085238f5-7c1e-4cb0-bee8-88d5587c8fa6\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk"
Jul 28 13:43:21.789189 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:43:21.789012    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/085238f5-7c1e-4cb0-bee8-88d5587c8fa6-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk\" (UID: \"085238f5-7c1e-4cb0-bee8-88d5587c8fa6\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk"
Jul 28 13:43:21.789189 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:43:21.789044    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/085238f5-7c1e-4cb0-bee8-88d5587c8fa6-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk\" (UID: \"085238f5-7c1e-4cb0-bee8-88d5587c8fa6\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk"
Jul 28 13:43:21.789189 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:43:21.789148    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/085238f5-7c1e-4cb0-bee8-88d5587c8fa6-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk\" (UID: \"085238f5-7c1e-4cb0-bee8-88d5587c8fa6\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk"
Jul 28 13:43:21.789388 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:43:21.789233    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5k8v\" (UniqueName: \"kubernetes.io/projected/085238f5-7c1e-4cb0-bee8-88d5587c8fa6-kube-api-access-p5k8v\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk\" (UID: \"085238f5-7c1e-4cb0-bee8-88d5587c8fa6\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk"
Jul 28 13:43:21.889790 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:43:21.889704    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/085238f5-7c1e-4cb0-bee8-88d5587c8fa6-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk\" (UID: \"085238f5-7c1e-4cb0-bee8-88d5587c8fa6\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk"
Jul 28 13:43:21.889790 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:43:21.889756    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/085238f5-7c1e-4cb0-bee8-88d5587c8fa6-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk\" (UID: \"085238f5-7c1e-4cb0-bee8-88d5587c8fa6\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk"
Jul 28 13:43:21.890019 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:43:21.889802    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5k8v\" (UniqueName: \"kubernetes.io/projected/085238f5-7c1e-4cb0-bee8-88d5587c8fa6-kube-api-access-p5k8v\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk\" (UID: \"085238f5-7c1e-4cb0-bee8-88d5587c8fa6\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk"
Jul 28 13:43:21.890019 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:43:21.889826    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/085238f5-7c1e-4cb0-bee8-88d5587c8fa6-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk\" (UID: \"085238f5-7c1e-4cb0-bee8-88d5587c8fa6\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk"
Jul 28 13:43:21.890019 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:43:21.889843    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/085238f5-7c1e-4cb0-bee8-88d5587c8fa6-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk\" (UID: \"085238f5-7c1e-4cb0-bee8-88d5587c8fa6\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk"
Jul 28 13:43:21.890249 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:43:21.890228    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/085238f5-7c1e-4cb0-bee8-88d5587c8fa6-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk\" (UID: \"085238f5-7c1e-4cb0-bee8-88d5587c8fa6\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk"
Jul 28 13:43:21.890475 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:43:21.890455    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/085238f5-7c1e-4cb0-bee8-88d5587c8fa6-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk\" (UID: \"085238f5-7c1e-4cb0-bee8-88d5587c8fa6\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk"
Jul 28 13:43:21.890531 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:43:21.890457    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/085238f5-7c1e-4cb0-bee8-88d5587c8fa6-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk\" (UID: \"085238f5-7c1e-4cb0-bee8-88d5587c8fa6\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk"
Jul 28 13:43:21.892455 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:43:21.892433    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/085238f5-7c1e-4cb0-bee8-88d5587c8fa6-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk\" (UID: \"085238f5-7c1e-4cb0-bee8-88d5587c8fa6\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk"
Jul 28 13:43:21.899355 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:43:21.899331    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5k8v\" (UniqueName: \"kubernetes.io/projected/085238f5-7c1e-4cb0-bee8-88d5587c8fa6-kube-api-access-p5k8v\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk\" (UID: \"085238f5-7c1e-4cb0-bee8-88d5587c8fa6\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk"
Jul 28 13:43:21.921612 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:43:21.921594    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk"
Jul 28 13:43:22.049688 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:43:22.049662    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk"]
Jul 28 13:43:22.052318 ip-10-0-129-177 kubenswrapper[2554]: W0728 13:43:22.052289    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod085238f5_7c1e_4cb0_bee8_88d5587c8fa6.slice/crio-ac7791aa12dc7dd1119368b86cbea8afe5a2d5d4791a868d459c6282b85a17b6 WatchSource:0}: Error finding container ac7791aa12dc7dd1119368b86cbea8afe5a2d5d4791a868d459c6282b85a17b6: Status 404 returned error can't find the container with id ac7791aa12dc7dd1119368b86cbea8afe5a2d5d4791a868d459c6282b85a17b6
Jul 28 13:43:22.736928 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:43:22.736892    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk" event={"ID":"085238f5-7c1e-4cb0-bee8-88d5587c8fa6","Type":"ContainerStarted","Data":"335c51c99254442b4b17ea698bb5bc521ed41d51ac9c8972d24ce31216389f47"}
Jul 28 13:43:22.737400 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:43:22.736956    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk" event={"ID":"085238f5-7c1e-4cb0-bee8-88d5587c8fa6","Type":"ContainerStarted","Data":"ac7791aa12dc7dd1119368b86cbea8afe5a2d5d4791a868d459c6282b85a17b6"}
Jul 28 13:43:27.760412 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:43:27.760375    2554 generic.go:356] "Generic (PLEG): container finished" podID="085238f5-7c1e-4cb0-bee8-88d5587c8fa6" containerID="335c51c99254442b4b17ea698bb5bc521ed41d51ac9c8972d24ce31216389f47" exitCode=0
Jul 28 13:43:27.760803 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:43:27.760455    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk" event={"ID":"085238f5-7c1e-4cb0-bee8-88d5587c8fa6","Type":"ContainerDied","Data":"335c51c99254442b4b17ea698bb5bc521ed41d51ac9c8972d24ce31216389f47"}
Jul 28 13:43:38.816737 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:43:38.816711    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk" event={"ID":"085238f5-7c1e-4cb0-bee8-88d5587c8fa6","Type":"ContainerStarted","Data":"7bfdaf385dcbc1a74e2644b308e94b68f95f8f94ef42d9d22ef996042903aa7e"}
Jul 28 13:43:39.822697 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:43:39.822660    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk" event={"ID":"085238f5-7c1e-4cb0-bee8-88d5587c8fa6","Type":"ContainerStarted","Data":"d89ac0b5d5cd203cff8693ffd244f880d90139a4d6c9c8534929b24064977fc3"}
Jul 28 13:43:39.823168 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:43:39.822866    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk"
Jul 28 13:43:39.823168 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:43:39.823013    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk"
Jul 28 13:43:39.824368 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:43:39.824343    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk" podUID="085238f5-7c1e-4cb0-bee8-88d5587c8fa6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 13:43:39.845641 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:43:39.845563    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk" podStartSLOduration=3.123852206 podStartE2EDuration="18.845546889s" totalImagesPullingTime="10.984862992s" totalInitContainerRuntime="4.736831691s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:43:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:43:39.845067162 +0000 UTC m=+3016.630942273" watchObservedRunningTime="2026-07-28 13:43:39.845546889 +0000 UTC m=+3016.631422000"
Jul 28 13:43:40.827545 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:43:40.827508    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk" podUID="085238f5-7c1e-4cb0-bee8-88d5587c8fa6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 13:43:45.832013 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:43:45.831983    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk"
Jul 28 13:43:45.832525 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:43:45.832444    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk" podUID="085238f5-7c1e-4cb0-bee8-88d5587c8fa6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 13:43:55.832805 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:43:55.832758    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk" podUID="085238f5-7c1e-4cb0-bee8-88d5587c8fa6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 13:44:05.832464 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:05.832420    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk" podUID="085238f5-7c1e-4cb0-bee8-88d5587c8fa6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 13:44:15.832442 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:15.832365    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk" podUID="085238f5-7c1e-4cb0-bee8-88d5587c8fa6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 13:44:25.833013 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:25.832982    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk"
Jul 28 13:44:33.489537 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:33.489502    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk"]
Jul 28 13:44:33.489933 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:33.489750    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk" podUID="085238f5-7c1e-4cb0-bee8-88d5587c8fa6" containerName="kserve-container" containerID="cri-o://7bfdaf385dcbc1a74e2644b308e94b68f95f8f94ef42d9d22ef996042903aa7e" gracePeriod=30
Jul 28 13:44:33.489933 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:33.489799    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk" podUID="085238f5-7c1e-4cb0-bee8-88d5587c8fa6" containerName="kube-rbac-proxy" containerID="cri-o://d89ac0b5d5cd203cff8693ffd244f880d90139a4d6c9c8534929b24064977fc3" gracePeriod=30
Jul 28 13:44:34.058088 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:34.058054    2554 generic.go:356] "Generic (PLEG): container finished" podID="085238f5-7c1e-4cb0-bee8-88d5587c8fa6" containerID="d89ac0b5d5cd203cff8693ffd244f880d90139a4d6c9c8534929b24064977fc3" exitCode=2
Jul 28 13:44:34.058301 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:34.058107    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk" event={"ID":"085238f5-7c1e-4cb0-bee8-88d5587c8fa6","Type":"ContainerDied","Data":"d89ac0b5d5cd203cff8693ffd244f880d90139a4d6c9c8534929b24064977fc3"}
Jul 28 13:44:35.828264 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:35.828226    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk" podUID="085238f5-7c1e-4cb0-bee8-88d5587c8fa6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Jul 28 13:44:35.832500 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:35.832479    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk" podUID="085238f5-7c1e-4cb0-bee8-88d5587c8fa6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 13:44:36.070825 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:36.070794    2554 generic.go:356] "Generic (PLEG): container finished" podID="085238f5-7c1e-4cb0-bee8-88d5587c8fa6" containerID="7bfdaf385dcbc1a74e2644b308e94b68f95f8f94ef42d9d22ef996042903aa7e" exitCode=0
Jul 28 13:44:36.070953 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:36.070867    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk" event={"ID":"085238f5-7c1e-4cb0-bee8-88d5587c8fa6","Type":"ContainerDied","Data":"7bfdaf385dcbc1a74e2644b308e94b68f95f8f94ef42d9d22ef996042903aa7e"}
Jul 28 13:44:36.145060 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:36.145036    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk"
Jul 28 13:44:36.214244 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:36.214218    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p5k8v\" (UniqueName: \"kubernetes.io/projected/085238f5-7c1e-4cb0-bee8-88d5587c8fa6-kube-api-access-p5k8v\") pod \"085238f5-7c1e-4cb0-bee8-88d5587c8fa6\" (UID: \"085238f5-7c1e-4cb0-bee8-88d5587c8fa6\") "
Jul 28 13:44:36.214402 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:36.214254    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/085238f5-7c1e-4cb0-bee8-88d5587c8fa6-cabundle-cert\") pod \"085238f5-7c1e-4cb0-bee8-88d5587c8fa6\" (UID: \"085238f5-7c1e-4cb0-bee8-88d5587c8fa6\") "
Jul 28 13:44:36.214402 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:36.214297    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/085238f5-7c1e-4cb0-bee8-88d5587c8fa6-proxy-tls\") pod \"085238f5-7c1e-4cb0-bee8-88d5587c8fa6\" (UID: \"085238f5-7c1e-4cb0-bee8-88d5587c8fa6\") "
Jul 28 13:44:36.214402 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:36.214330    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/085238f5-7c1e-4cb0-bee8-88d5587c8fa6-kserve-provision-location\") pod \"085238f5-7c1e-4cb0-bee8-88d5587c8fa6\" (UID: \"085238f5-7c1e-4cb0-bee8-88d5587c8fa6\") "
Jul 28 13:44:36.214402 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:36.214351    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/085238f5-7c1e-4cb0-bee8-88d5587c8fa6-kube-rbac-proxy-sar-config\") pod \"085238f5-7c1e-4cb0-bee8-88d5587c8fa6\" (UID: \"085238f5-7c1e-4cb0-bee8-88d5587c8fa6\") "
Jul 28 13:44:36.214730 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:36.214701    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/085238f5-7c1e-4cb0-bee8-88d5587c8fa6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "085238f5-7c1e-4cb0-bee8-88d5587c8fa6" (UID: "085238f5-7c1e-4cb0-bee8-88d5587c8fa6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:44:36.214901 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:36.214783    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/085238f5-7c1e-4cb0-bee8-88d5587c8fa6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "085238f5-7c1e-4cb0-bee8-88d5587c8fa6" (UID: "085238f5-7c1e-4cb0-bee8-88d5587c8fa6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:44:36.216445 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:36.216417    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/085238f5-7c1e-4cb0-bee8-88d5587c8fa6-kube-api-access-p5k8v" (OuterVolumeSpecName: "kube-api-access-p5k8v") pod "085238f5-7c1e-4cb0-bee8-88d5587c8fa6" (UID: "085238f5-7c1e-4cb0-bee8-88d5587c8fa6"). InnerVolumeSpecName "kube-api-access-p5k8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:44:36.216538 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:36.216505    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/085238f5-7c1e-4cb0-bee8-88d5587c8fa6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "085238f5-7c1e-4cb0-bee8-88d5587c8fa6" (UID: "085238f5-7c1e-4cb0-bee8-88d5587c8fa6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:44:36.224183 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:36.224163    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/085238f5-7c1e-4cb0-bee8-88d5587c8fa6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "085238f5-7c1e-4cb0-bee8-88d5587c8fa6" (UID: "085238f5-7c1e-4cb0-bee8-88d5587c8fa6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:44:36.315554 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:36.315522    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5k8v\" (UniqueName: \"kubernetes.io/projected/085238f5-7c1e-4cb0-bee8-88d5587c8fa6-kube-api-access-p5k8v\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:44:36.315554 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:36.315551    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/085238f5-7c1e-4cb0-bee8-88d5587c8fa6-cabundle-cert\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:44:36.315740 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:36.315566    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/085238f5-7c1e-4cb0-bee8-88d5587c8fa6-proxy-tls\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:44:36.315740 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:36.315578    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/085238f5-7c1e-4cb0-bee8-88d5587c8fa6-kserve-provision-location\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:44:36.315740 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:36.315589    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/085238f5-7c1e-4cb0-bee8-88d5587c8fa6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:44:37.077157 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:37.077125    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk" event={"ID":"085238f5-7c1e-4cb0-bee8-88d5587c8fa6","Type":"ContainerDied","Data":"ac7791aa12dc7dd1119368b86cbea8afe5a2d5d4791a868d459c6282b85a17b6"}
Jul 28 13:44:37.077648 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:37.077177    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk"
Jul 28 13:44:37.077648 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:37.077203    2554 scope.go:117] "RemoveContainer" containerID="d89ac0b5d5cd203cff8693ffd244f880d90139a4d6c9c8534929b24064977fc3"
Jul 28 13:44:37.086453 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:37.086433    2554 scope.go:117] "RemoveContainer" containerID="7bfdaf385dcbc1a74e2644b308e94b68f95f8f94ef42d9d22ef996042903aa7e"
Jul 28 13:44:37.093674 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:37.093660    2554 scope.go:117] "RemoveContainer" containerID="335c51c99254442b4b17ea698bb5bc521ed41d51ac9c8972d24ce31216389f47"
Jul 28 13:44:37.107635 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:37.107615    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk"]
Jul 28 13:44:37.112242 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:37.112218    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-qnlfk"]
Jul 28 13:44:37.801287 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:37.801246    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="085238f5-7c1e-4cb0-bee8-88d5587c8fa6" path="/var/lib/kubelet/pods/085238f5-7c1e-4cb0-bee8-88d5587c8fa6/volumes"
Jul 28 13:44:38.654253 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:38.654217    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9lzdb"]
Jul 28 13:44:38.654671 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:38.654548    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="085238f5-7c1e-4cb0-bee8-88d5587c8fa6" containerName="storage-initializer"
Jul 28 13:44:38.654671 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:38.654560    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="085238f5-7c1e-4cb0-bee8-88d5587c8fa6" containerName="storage-initializer"
Jul 28 13:44:38.654671 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:38.654567    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="085238f5-7c1e-4cb0-bee8-88d5587c8fa6" containerName="kube-rbac-proxy"
Jul 28 13:44:38.654671 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:38.654572    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="085238f5-7c1e-4cb0-bee8-88d5587c8fa6" containerName="kube-rbac-proxy"
Jul 28 13:44:38.654671 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:38.654594    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="085238f5-7c1e-4cb0-bee8-88d5587c8fa6" containerName="kserve-container"
Jul 28 13:44:38.654671 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:38.654600    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="085238f5-7c1e-4cb0-bee8-88d5587c8fa6" containerName="kserve-container"
Jul 28 13:44:38.654671 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:38.654648    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="085238f5-7c1e-4cb0-bee8-88d5587c8fa6" containerName="kserve-container"
Jul 28 13:44:38.654671 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:38.654655    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="085238f5-7c1e-4cb0-bee8-88d5587c8fa6" containerName="kube-rbac-proxy"
Jul 28 13:44:38.659482 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:38.659459    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9lzdb"
Jul 28 13:44:38.663059 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:38.663035    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rxfwd\""
Jul 28 13:44:38.663059 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:38.663049    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
Jul 28 13:44:38.663261 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:38.663191    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\""
Jul 28 13:44:38.663812 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:38.663793    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:44:38.663934 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:38.663909    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:44:38.664004 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:38.663912    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:44:38.664004 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:38.663940    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:44:38.671428 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:38.671407    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9lzdb"]
Jul 28 13:44:38.737479 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:38.737446    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bfe8711e-6860-4c32-994e-fb6a6d438aab-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-9lzdb\" (UID: \"bfe8711e-6860-4c32-994e-fb6a6d438aab\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9lzdb"
Jul 28 13:44:38.737479 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:38.737480    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ntc7\" (UniqueName: \"kubernetes.io/projected/bfe8711e-6860-4c32-994e-fb6a6d438aab-kube-api-access-9ntc7\") pod \"isvc-pmml-predictor-8674795f59-9lzdb\" (UID: \"bfe8711e-6860-4c32-994e-fb6a6d438aab\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9lzdb"
Jul 28 13:44:38.737694 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:38.737508    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bfe8711e-6860-4c32-994e-fb6a6d438aab-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-9lzdb\" (UID: \"bfe8711e-6860-4c32-994e-fb6a6d438aab\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9lzdb"
Jul 28 13:44:38.737694 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:38.737553    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bfe8711e-6860-4c32-994e-fb6a6d438aab-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-9lzdb\" (UID: \"bfe8711e-6860-4c32-994e-fb6a6d438aab\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9lzdb"
Jul 28 13:44:38.737694 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:38.737602    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bfe8711e-6860-4c32-994e-fb6a6d438aab-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-9lzdb\" (UID: \"bfe8711e-6860-4c32-994e-fb6a6d438aab\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9lzdb"
Jul 28 13:44:38.838161 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:38.838126    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bfe8711e-6860-4c32-994e-fb6a6d438aab-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-9lzdb\" (UID: \"bfe8711e-6860-4c32-994e-fb6a6d438aab\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9lzdb"
Jul 28 13:44:38.838161 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:38.838159    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ntc7\" (UniqueName: \"kubernetes.io/projected/bfe8711e-6860-4c32-994e-fb6a6d438aab-kube-api-access-9ntc7\") pod \"isvc-pmml-predictor-8674795f59-9lzdb\" (UID: \"bfe8711e-6860-4c32-994e-fb6a6d438aab\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9lzdb"
Jul 28 13:44:38.838450 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:38.838179    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bfe8711e-6860-4c32-994e-fb6a6d438aab-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-9lzdb\" (UID: \"bfe8711e-6860-4c32-994e-fb6a6d438aab\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9lzdb"
Jul 28 13:44:38.838450 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:38.838324    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bfe8711e-6860-4c32-994e-fb6a6d438aab-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-9lzdb\" (UID: \"bfe8711e-6860-4c32-994e-fb6a6d438aab\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9lzdb"
Jul 28 13:44:38.838450 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:38.838408    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bfe8711e-6860-4c32-994e-fb6a6d438aab-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-9lzdb\" (UID: \"bfe8711e-6860-4c32-994e-fb6a6d438aab\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9lzdb"
Jul 28 13:44:38.838593 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:44:38.838465    2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-pmml-predictor-serving-cert: secret "isvc-pmml-predictor-serving-cert" not found
Jul 28 13:44:38.838593 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:44:38.838577    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfe8711e-6860-4c32-994e-fb6a6d438aab-proxy-tls podName:bfe8711e-6860-4c32-994e-fb6a6d438aab nodeName:}" failed. No retries permitted until 2026-07-28 13:44:39.338554442 +0000 UTC m=+3076.124429534 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bfe8711e-6860-4c32-994e-fb6a6d438aab-proxy-tls") pod "isvc-pmml-predictor-8674795f59-9lzdb" (UID: "bfe8711e-6860-4c32-994e-fb6a6d438aab") : secret "isvc-pmml-predictor-serving-cert" not found
Jul 28 13:44:38.838833 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:38.838805    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bfe8711e-6860-4c32-994e-fb6a6d438aab-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-9lzdb\" (UID: \"bfe8711e-6860-4c32-994e-fb6a6d438aab\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9lzdb"
Jul 28 13:44:38.838902 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:38.838883    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bfe8711e-6860-4c32-994e-fb6a6d438aab-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-9lzdb\" (UID: \"bfe8711e-6860-4c32-994e-fb6a6d438aab\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9lzdb"
Jul 28 13:44:38.838989 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:38.838974    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bfe8711e-6860-4c32-994e-fb6a6d438aab-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-9lzdb\" (UID: \"bfe8711e-6860-4c32-994e-fb6a6d438aab\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9lzdb"
Jul 28 13:44:38.848503 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:38.848478    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ntc7\" (UniqueName: \"kubernetes.io/projected/bfe8711e-6860-4c32-994e-fb6a6d438aab-kube-api-access-9ntc7\") pod \"isvc-pmml-predictor-8674795f59-9lzdb\" (UID: \"bfe8711e-6860-4c32-994e-fb6a6d438aab\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9lzdb"
Jul 28 13:44:39.343260 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:39.343227    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bfe8711e-6860-4c32-994e-fb6a6d438aab-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-9lzdb\" (UID: \"bfe8711e-6860-4c32-994e-fb6a6d438aab\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9lzdb"
Jul 28 13:44:39.343444 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:44:39.343359    2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-pmml-predictor-serving-cert: secret "isvc-pmml-predictor-serving-cert" not found
Jul 28 13:44:39.343444 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:44:39.343419    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfe8711e-6860-4c32-994e-fb6a6d438aab-proxy-tls podName:bfe8711e-6860-4c32-994e-fb6a6d438aab nodeName:}" failed. No retries permitted until 2026-07-28 13:44:40.343401018 +0000 UTC m=+3077.129276110 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bfe8711e-6860-4c32-994e-fb6a6d438aab-proxy-tls") pod "isvc-pmml-predictor-8674795f59-9lzdb" (UID: "bfe8711e-6860-4c32-994e-fb6a6d438aab") : secret "isvc-pmml-predictor-serving-cert" not found
Jul 28 13:44:40.352657 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:40.352619    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bfe8711e-6860-4c32-994e-fb6a6d438aab-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-9lzdb\" (UID: \"bfe8711e-6860-4c32-994e-fb6a6d438aab\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9lzdb"
Jul 28 13:44:40.355170 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:40.355146    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bfe8711e-6860-4c32-994e-fb6a6d438aab-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-9lzdb\" (UID: \"bfe8711e-6860-4c32-994e-fb6a6d438aab\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9lzdb"
Jul 28 13:44:40.470916 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:40.470883    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9lzdb"
Jul 28 13:44:40.617267 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:40.617211    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9lzdb"]
Jul 28 13:44:40.619905 ip-10-0-129-177 kubenswrapper[2554]: W0728 13:44:40.619877    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfe8711e_6860_4c32_994e_fb6a6d438aab.slice/crio-d9e5e00bb369161ce8d3be2b7d0106a3159ba17319d50287cad7cd6574ddf1a9 WatchSource:0}: Error finding container d9e5e00bb369161ce8d3be2b7d0106a3159ba17319d50287cad7cd6574ddf1a9: Status 404 returned error can't find the container with id d9e5e00bb369161ce8d3be2b7d0106a3159ba17319d50287cad7cd6574ddf1a9
Jul 28 13:44:41.095869 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:41.095831    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9lzdb" event={"ID":"bfe8711e-6860-4c32-994e-fb6a6d438aab","Type":"ContainerStarted","Data":"2c45e266cf68c2ad4b829665e8ad6ea9409e5b1696e09209fc9a0bcab14f2d5d"}
Jul 28 13:44:41.096036 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:41.095909    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9lzdb" event={"ID":"bfe8711e-6860-4c32-994e-fb6a6d438aab","Type":"ContainerStarted","Data":"d9e5e00bb369161ce8d3be2b7d0106a3159ba17319d50287cad7cd6574ddf1a9"}
Jul 28 13:44:45.113153 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:45.113118    2554 generic.go:356] "Generic (PLEG): container finished" podID="bfe8711e-6860-4c32-994e-fb6a6d438aab" containerID="2c45e266cf68c2ad4b829665e8ad6ea9409e5b1696e09209fc9a0bcab14f2d5d" exitCode=0
Jul 28 13:44:45.113574 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:45.113196    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9lzdb" event={"ID":"bfe8711e-6860-4c32-994e-fb6a6d438aab","Type":"ContainerDied","Data":"2c45e266cf68c2ad4b829665e8ad6ea9409e5b1696e09209fc9a0bcab14f2d5d"}
Jul 28 13:44:45.114456 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:45.114441    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 13:44:52.151542 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:52.151513    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9lzdb" event={"ID":"bfe8711e-6860-4c32-994e-fb6a6d438aab","Type":"ContainerStarted","Data":"8f4e554b762fe63f04ad94cf0cd95739f627b4984ed46280e843c310c994921d"}
Jul 28 13:44:53.157474 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:53.157431    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9lzdb" event={"ID":"bfe8711e-6860-4c32-994e-fb6a6d438aab","Type":"ContainerStarted","Data":"21a767b1cf0fc5a9dffcfa5a4dd8524fdfec03daf60b625e6e497fb86d35b396"}
Jul 28 13:44:53.157962 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:53.157622    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9lzdb"
Jul 28 13:44:53.182220 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:53.182176    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9lzdb" podStartSLOduration=4.321039821 podStartE2EDuration="15.182162962s" totalImagesPullingTime="6.93404122s" totalInitContainerRuntime="3.927081921s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:44:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:44:53.180052743 +0000 UTC m=+3089.965927866" watchObservedRunningTime="2026-07-28 13:44:53.182162962 +0000 UTC m=+3089.968038073"
Jul 28 13:44:54.161717 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:54.161682    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9lzdb"
Jul 28 13:44:54.162735 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:54.162711    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9lzdb" podUID="bfe8711e-6860-4c32-994e-fb6a6d438aab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 13:44:55.165517 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:44:55.165471    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9lzdb" podUID="bfe8711e-6860-4c32-994e-fb6a6d438aab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 13:45:00.169579 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:45:00.169551    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9lzdb"
Jul 28 13:45:00.170041 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:45:00.170015    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9lzdb" podUID="bfe8711e-6860-4c32-994e-fb6a6d438aab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 13:45:10.169962 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:45:10.169917    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9lzdb" podUID="bfe8711e-6860-4c32-994e-fb6a6d438aab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 13:45:20.170578 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:45:20.170542    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9lzdb" podUID="bfe8711e-6860-4c32-994e-fb6a6d438aab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 13:45:23.957785 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:45:23.957758    2554 scope.go:117] "RemoveContainer" containerID="729aac710bb600fe065d50d0cd93123528ae92dd0eaa14b6f18ddca3ade6a475"
Jul 28 13:45:23.965462 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:45:23.965443    2554 scope.go:117] "RemoveContainer" containerID="25b7b07308e43f3e409bec91ecaf36414ce74061f96cdf14e1f7dbc98c1c94ac"
Jul 28 13:45:23.972958 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:45:23.972939    2554 scope.go:117] "RemoveContainer" containerID="b0138580048bd8e3816164a3828243c9c1951411bd10db9e4dbb2147c9cca12a"
Jul 28 13:45:30.170217 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:45:30.170179    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9lzdb" podUID="bfe8711e-6860-4c32-994e-fb6a6d438aab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 13:45:40.170509 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:45:40.170465    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9lzdb" podUID="bfe8711e-6860-4c32-994e-fb6a6d438aab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 13:45:50.170845 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:45:50.170756    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9lzdb" podUID="bfe8711e-6860-4c32-994e-fb6a6d438aab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 13:46:00.170185 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:00.170145    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9lzdb" podUID="bfe8711e-6860-4c32-994e-fb6a6d438aab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 13:46:10.170891 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:10.170862    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9lzdb"
Jul 28 13:46:19.711250 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:19.711210    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9lzdb"]
Jul 28 13:46:19.711734 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:19.711575    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9lzdb" podUID="bfe8711e-6860-4c32-994e-fb6a6d438aab" containerName="kserve-container" containerID="cri-o://8f4e554b762fe63f04ad94cf0cd95739f627b4984ed46280e843c310c994921d" gracePeriod=30
Jul 28 13:46:19.711734 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:19.711605    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9lzdb" podUID="bfe8711e-6860-4c32-994e-fb6a6d438aab" containerName="kube-rbac-proxy" containerID="cri-o://21a767b1cf0fc5a9dffcfa5a4dd8524fdfec03daf60b625e6e497fb86d35b396" gracePeriod=30
Jul 28 13:46:20.166041 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:20.165943    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9lzdb" podUID="bfe8711e-6860-4c32-994e-fb6a6d438aab" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Jul 28 13:46:20.170265 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:20.170241    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9lzdb" podUID="bfe8711e-6860-4c32-994e-fb6a6d438aab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 13:46:20.520301 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:20.520242    2554 generic.go:356] "Generic (PLEG): container finished" podID="bfe8711e-6860-4c32-994e-fb6a6d438aab" containerID="21a767b1cf0fc5a9dffcfa5a4dd8524fdfec03daf60b625e6e497fb86d35b396" exitCode=2
Jul 28 13:46:20.520473 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:20.520322    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9lzdb" event={"ID":"bfe8711e-6860-4c32-994e-fb6a6d438aab","Type":"ContainerDied","Data":"21a767b1cf0fc5a9dffcfa5a4dd8524fdfec03daf60b625e6e497fb86d35b396"}
Jul 28 13:46:23.160611 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:23.160584    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9lzdb"
Jul 28 13:46:23.200997 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:23.200971    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ntc7\" (UniqueName: \"kubernetes.io/projected/bfe8711e-6860-4c32-994e-fb6a6d438aab-kube-api-access-9ntc7\") pod \"bfe8711e-6860-4c32-994e-fb6a6d438aab\" (UID: \"bfe8711e-6860-4c32-994e-fb6a6d438aab\") "
Jul 28 13:46:23.201139 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:23.201043    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bfe8711e-6860-4c32-994e-fb6a6d438aab-kserve-provision-location\") pod \"bfe8711e-6860-4c32-994e-fb6a6d438aab\" (UID: \"bfe8711e-6860-4c32-994e-fb6a6d438aab\") "
Jul 28 13:46:23.201216 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:23.201192    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bfe8711e-6860-4c32-994e-fb6a6d438aab-kube-rbac-proxy-sar-config\") pod \"bfe8711e-6860-4c32-994e-fb6a6d438aab\" (UID: \"bfe8711e-6860-4c32-994e-fb6a6d438aab\") "
Jul 28 13:46:23.201292 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:23.201241    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bfe8711e-6860-4c32-994e-fb6a6d438aab-cabundle-cert\") pod \"bfe8711e-6860-4c32-994e-fb6a6d438aab\" (UID: \"bfe8711e-6860-4c32-994e-fb6a6d438aab\") "
Jul 28 13:46:23.201363 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:23.201312    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bfe8711e-6860-4c32-994e-fb6a6d438aab-proxy-tls\") pod \"bfe8711e-6860-4c32-994e-fb6a6d438aab\" (UID: \"bfe8711e-6860-4c32-994e-fb6a6d438aab\") "
Jul 28 13:46:23.201363 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:23.201345    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfe8711e-6860-4c32-994e-fb6a6d438aab-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bfe8711e-6860-4c32-994e-fb6a6d438aab" (UID: "bfe8711e-6860-4c32-994e-fb6a6d438aab"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:46:23.201545 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:23.201529    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bfe8711e-6860-4c32-994e-fb6a6d438aab-kserve-provision-location\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:46:23.201604 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:23.201578    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfe8711e-6860-4c32-994e-fb6a6d438aab-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bfe8711e-6860-4c32-994e-fb6a6d438aab" (UID: "bfe8711e-6860-4c32-994e-fb6a6d438aab"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:46:23.201704 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:23.201670    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfe8711e-6860-4c32-994e-fb6a6d438aab-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bfe8711e-6860-4c32-994e-fb6a6d438aab" (UID: "bfe8711e-6860-4c32-994e-fb6a6d438aab"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:46:23.203236 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:23.203215    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfe8711e-6860-4c32-994e-fb6a6d438aab-kube-api-access-9ntc7" (OuterVolumeSpecName: "kube-api-access-9ntc7") pod "bfe8711e-6860-4c32-994e-fb6a6d438aab" (UID: "bfe8711e-6860-4c32-994e-fb6a6d438aab"). InnerVolumeSpecName "kube-api-access-9ntc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:46:23.203571 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:23.203549    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfe8711e-6860-4c32-994e-fb6a6d438aab-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bfe8711e-6860-4c32-994e-fb6a6d438aab" (UID: "bfe8711e-6860-4c32-994e-fb6a6d438aab"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:46:23.302741 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:23.302673    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bfe8711e-6860-4c32-994e-fb6a6d438aab-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:46:23.302741 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:23.302695    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bfe8711e-6860-4c32-994e-fb6a6d438aab-cabundle-cert\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:46:23.302741 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:23.302706    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bfe8711e-6860-4c32-994e-fb6a6d438aab-proxy-tls\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:46:23.302741 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:23.302714    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ntc7\" (UniqueName: \"kubernetes.io/projected/bfe8711e-6860-4c32-994e-fb6a6d438aab-kube-api-access-9ntc7\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:46:23.534525 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:23.534493    2554 generic.go:356] "Generic (PLEG): container finished" podID="bfe8711e-6860-4c32-994e-fb6a6d438aab" containerID="8f4e554b762fe63f04ad94cf0cd95739f627b4984ed46280e843c310c994921d" exitCode=0
Jul 28 13:46:23.534688 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:23.534536    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9lzdb" event={"ID":"bfe8711e-6860-4c32-994e-fb6a6d438aab","Type":"ContainerDied","Data":"8f4e554b762fe63f04ad94cf0cd95739f627b4984ed46280e843c310c994921d"}
Jul 28 13:46:23.534688 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:23.534548    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9lzdb"
Jul 28 13:46:23.534688 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:23.534584    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9lzdb" event={"ID":"bfe8711e-6860-4c32-994e-fb6a6d438aab","Type":"ContainerDied","Data":"d9e5e00bb369161ce8d3be2b7d0106a3159ba17319d50287cad7cd6574ddf1a9"}
Jul 28 13:46:23.534688 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:23.534611    2554 scope.go:117] "RemoveContainer" containerID="21a767b1cf0fc5a9dffcfa5a4dd8524fdfec03daf60b625e6e497fb86d35b396"
Jul 28 13:46:23.543133 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:23.543116    2554 scope.go:117] "RemoveContainer" containerID="8f4e554b762fe63f04ad94cf0cd95739f627b4984ed46280e843c310c994921d"
Jul 28 13:46:23.550391 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:23.550371    2554 scope.go:117] "RemoveContainer" containerID="2c45e266cf68c2ad4b829665e8ad6ea9409e5b1696e09209fc9a0bcab14f2d5d"
Jul 28 13:46:23.557348 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:23.557333    2554 scope.go:117] "RemoveContainer" containerID="21a767b1cf0fc5a9dffcfa5a4dd8524fdfec03daf60b625e6e497fb86d35b396"
Jul 28 13:46:23.557582 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:46:23.557565    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21a767b1cf0fc5a9dffcfa5a4dd8524fdfec03daf60b625e6e497fb86d35b396\": container with ID starting with 21a767b1cf0fc5a9dffcfa5a4dd8524fdfec03daf60b625e6e497fb86d35b396 not found: ID does not exist" containerID="21a767b1cf0fc5a9dffcfa5a4dd8524fdfec03daf60b625e6e497fb86d35b396"
Jul 28 13:46:23.557635 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:23.557589    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21a767b1cf0fc5a9dffcfa5a4dd8524fdfec03daf60b625e6e497fb86d35b396"} err="failed to get container status \"21a767b1cf0fc5a9dffcfa5a4dd8524fdfec03daf60b625e6e497fb86d35b396\": rpc error: code = NotFound desc = could not find container \"21a767b1cf0fc5a9dffcfa5a4dd8524fdfec03daf60b625e6e497fb86d35b396\": container with ID starting with 21a767b1cf0fc5a9dffcfa5a4dd8524fdfec03daf60b625e6e497fb86d35b396 not found: ID does not exist"
Jul 28 13:46:23.557635 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:23.557604    2554 scope.go:117] "RemoveContainer" containerID="8f4e554b762fe63f04ad94cf0cd95739f627b4984ed46280e843c310c994921d"
Jul 28 13:46:23.557833 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:46:23.557816    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f4e554b762fe63f04ad94cf0cd95739f627b4984ed46280e843c310c994921d\": container with ID starting with 8f4e554b762fe63f04ad94cf0cd95739f627b4984ed46280e843c310c994921d not found: ID does not exist" containerID="8f4e554b762fe63f04ad94cf0cd95739f627b4984ed46280e843c310c994921d"
Jul 28 13:46:23.557869 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:23.557841    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f4e554b762fe63f04ad94cf0cd95739f627b4984ed46280e843c310c994921d"} err="failed to get container status \"8f4e554b762fe63f04ad94cf0cd95739f627b4984ed46280e843c310c994921d\": rpc error: code = NotFound desc = could not find container \"8f4e554b762fe63f04ad94cf0cd95739f627b4984ed46280e843c310c994921d\": container with ID starting with 8f4e554b762fe63f04ad94cf0cd95739f627b4984ed46280e843c310c994921d not found: ID does not exist"
Jul 28 13:46:23.557869 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:23.557858    2554 scope.go:117] "RemoveContainer" containerID="2c45e266cf68c2ad4b829665e8ad6ea9409e5b1696e09209fc9a0bcab14f2d5d"
Jul 28 13:46:23.558055 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:46:23.558042    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c45e266cf68c2ad4b829665e8ad6ea9409e5b1696e09209fc9a0bcab14f2d5d\": container with ID starting with 2c45e266cf68c2ad4b829665e8ad6ea9409e5b1696e09209fc9a0bcab14f2d5d not found: ID does not exist" containerID="2c45e266cf68c2ad4b829665e8ad6ea9409e5b1696e09209fc9a0bcab14f2d5d"
Jul 28 13:46:23.558095 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:23.558060    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c45e266cf68c2ad4b829665e8ad6ea9409e5b1696e09209fc9a0bcab14f2d5d"} err="failed to get container status \"2c45e266cf68c2ad4b829665e8ad6ea9409e5b1696e09209fc9a0bcab14f2d5d\": rpc error: code = NotFound desc = could not find container \"2c45e266cf68c2ad4b829665e8ad6ea9409e5b1696e09209fc9a0bcab14f2d5d\": container with ID starting with 2c45e266cf68c2ad4b829665e8ad6ea9409e5b1696e09209fc9a0bcab14f2d5d not found: ID does not exist"
Jul 28 13:46:23.566228 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:23.566203    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9lzdb"]
Jul 28 13:46:23.573506 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:23.573477    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-9lzdb"]
Jul 28 13:46:23.802625 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:23.802593    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfe8711e-6860-4c32-994e-fb6a6d438aab" path="/var/lib/kubelet/pods/bfe8711e-6860-4c32-994e-fb6a6d438aab/volumes"
Jul 28 13:46:24.956963 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:24.956927    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-86slx"]
Jul 28 13:46:24.957452 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:24.957433    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfe8711e-6860-4c32-994e-fb6a6d438aab" containerName="storage-initializer"
Jul 28 13:46:24.957541 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:24.957455    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfe8711e-6860-4c32-994e-fb6a6d438aab" containerName="storage-initializer"
Jul 28 13:46:24.957541 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:24.957474    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfe8711e-6860-4c32-994e-fb6a6d438aab" containerName="kserve-container"
Jul 28 13:46:24.957541 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:24.957483    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfe8711e-6860-4c32-994e-fb6a6d438aab" containerName="kserve-container"
Jul 28 13:46:24.957541 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:24.957497    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfe8711e-6860-4c32-994e-fb6a6d438aab" containerName="kube-rbac-proxy"
Jul 28 13:46:24.957541 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:24.957507    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfe8711e-6860-4c32-994e-fb6a6d438aab" containerName="kube-rbac-proxy"
Jul 28 13:46:24.957798 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:24.957609    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfe8711e-6860-4c32-994e-fb6a6d438aab" containerName="kserve-container"
Jul 28 13:46:24.957798 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:24.957620    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfe8711e-6860-4c32-994e-fb6a6d438aab" containerName="kube-rbac-proxy"
Jul 28 13:46:24.962509 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:24.962486    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-86slx"
Jul 28 13:46:24.965462 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:24.965442    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:46:24.965622 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:24.965604    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\""
Jul 28 13:46:24.965702 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:24.965642    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
Jul 28 13:46:24.965887 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:24.965868    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:46:24.966457 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:24.966442    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:46:24.966537 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:24.966472    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:46:24.966664 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:24.966650    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rxfwd\""
Jul 28 13:46:24.973453 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:24.973430    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-86slx"]
Jul 28 13:46:25.017265 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:25.017229    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4dc1fb10-eafd-44b8-9dd3-ff986f00aa62-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-86slx\" (UID: \"4dc1fb10-eafd-44b8-9dd3-ff986f00aa62\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-86slx"
Jul 28 13:46:25.017443 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:25.017263    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqqxs\" (UniqueName: \"kubernetes.io/projected/4dc1fb10-eafd-44b8-9dd3-ff986f00aa62-kube-api-access-kqqxs\") pod \"isvc-pmml-runtime-predictor-885fc8777-86slx\" (UID: \"4dc1fb10-eafd-44b8-9dd3-ff986f00aa62\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-86slx"
Jul 28 13:46:25.017443 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:25.017305    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4dc1fb10-eafd-44b8-9dd3-ff986f00aa62-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-86slx\" (UID: \"4dc1fb10-eafd-44b8-9dd3-ff986f00aa62\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-86slx"
Jul 28 13:46:25.017571 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:25.017443    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4dc1fb10-eafd-44b8-9dd3-ff986f00aa62-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-86slx\" (UID: \"4dc1fb10-eafd-44b8-9dd3-ff986f00aa62\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-86slx"
Jul 28 13:46:25.017571 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:25.017501    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4dc1fb10-eafd-44b8-9dd3-ff986f00aa62-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-86slx\" (UID: \"4dc1fb10-eafd-44b8-9dd3-ff986f00aa62\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-86slx"
Jul 28 13:46:25.118031 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:25.117996    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4dc1fb10-eafd-44b8-9dd3-ff986f00aa62-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-86slx\" (UID: \"4dc1fb10-eafd-44b8-9dd3-ff986f00aa62\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-86slx"
Jul 28 13:46:25.118031 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:25.118029    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqqxs\" (UniqueName: \"kubernetes.io/projected/4dc1fb10-eafd-44b8-9dd3-ff986f00aa62-kube-api-access-kqqxs\") pod \"isvc-pmml-runtime-predictor-885fc8777-86slx\" (UID: \"4dc1fb10-eafd-44b8-9dd3-ff986f00aa62\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-86slx"
Jul 28 13:46:25.118322 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:25.118053    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4dc1fb10-eafd-44b8-9dd3-ff986f00aa62-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-86slx\" (UID: \"4dc1fb10-eafd-44b8-9dd3-ff986f00aa62\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-86slx"
Jul 28 13:46:25.118322 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:25.118106    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4dc1fb10-eafd-44b8-9dd3-ff986f00aa62-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-86slx\" (UID: \"4dc1fb10-eafd-44b8-9dd3-ff986f00aa62\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-86slx"
Jul 28 13:46:25.118322 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:25.118137    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4dc1fb10-eafd-44b8-9dd3-ff986f00aa62-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-86slx\" (UID: \"4dc1fb10-eafd-44b8-9dd3-ff986f00aa62\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-86slx"
Jul 28 13:46:25.118612 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:25.118588    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4dc1fb10-eafd-44b8-9dd3-ff986f00aa62-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-86slx\" (UID: \"4dc1fb10-eafd-44b8-9dd3-ff986f00aa62\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-86slx"
Jul 28 13:46:25.118812 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:25.118789    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4dc1fb10-eafd-44b8-9dd3-ff986f00aa62-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-86slx\" (UID: \"4dc1fb10-eafd-44b8-9dd3-ff986f00aa62\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-86slx"
Jul 28 13:46:25.118878 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:25.118799    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4dc1fb10-eafd-44b8-9dd3-ff986f00aa62-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-86slx\" (UID: \"4dc1fb10-eafd-44b8-9dd3-ff986f00aa62\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-86slx"
Jul 28 13:46:25.121039 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:25.121021    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4dc1fb10-eafd-44b8-9dd3-ff986f00aa62-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-86slx\" (UID: \"4dc1fb10-eafd-44b8-9dd3-ff986f00aa62\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-86slx"
Jul 28 13:46:25.127993 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:25.127969    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqqxs\" (UniqueName: \"kubernetes.io/projected/4dc1fb10-eafd-44b8-9dd3-ff986f00aa62-kube-api-access-kqqxs\") pod \"isvc-pmml-runtime-predictor-885fc8777-86slx\" (UID: \"4dc1fb10-eafd-44b8-9dd3-ff986f00aa62\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-86slx"
Jul 28 13:46:25.273322 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:25.273220    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-86slx"
Jul 28 13:46:25.407730 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:25.407706    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-86slx"]
Jul 28 13:46:25.409979 ip-10-0-129-177 kubenswrapper[2554]: W0728 13:46:25.409952    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dc1fb10_eafd_44b8_9dd3_ff986f00aa62.slice/crio-d0a1b4f8c03dafea3cd00252e25a5ea8dbfecb7e1f4dc4078c40a35b9b090b8d WatchSource:0}: Error finding container d0a1b4f8c03dafea3cd00252e25a5ea8dbfecb7e1f4dc4078c40a35b9b090b8d: Status 404 returned error can't find the container with id d0a1b4f8c03dafea3cd00252e25a5ea8dbfecb7e1f4dc4078c40a35b9b090b8d
Jul 28 13:46:25.545211 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:25.545124    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-86slx" event={"ID":"4dc1fb10-eafd-44b8-9dd3-ff986f00aa62","Type":"ContainerStarted","Data":"bdd75b39ae7e63fc455e98eca06a7219a34ee2431ed34f7a710af4122672ffcb"}
Jul 28 13:46:25.545380 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:25.545232    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-86slx" event={"ID":"4dc1fb10-eafd-44b8-9dd3-ff986f00aa62","Type":"ContainerStarted","Data":"d0a1b4f8c03dafea3cd00252e25a5ea8dbfecb7e1f4dc4078c40a35b9b090b8d"}
Jul 28 13:46:29.563003 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:29.562963    2554 generic.go:356] "Generic (PLEG): container finished" podID="4dc1fb10-eafd-44b8-9dd3-ff986f00aa62" containerID="bdd75b39ae7e63fc455e98eca06a7219a34ee2431ed34f7a710af4122672ffcb" exitCode=0
Jul 28 13:46:29.563408 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:29.563026    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-86slx" event={"ID":"4dc1fb10-eafd-44b8-9dd3-ff986f00aa62","Type":"ContainerDied","Data":"bdd75b39ae7e63fc455e98eca06a7219a34ee2431ed34f7a710af4122672ffcb"}
Jul 28 13:46:30.568967 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:30.568937    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-86slx" event={"ID":"4dc1fb10-eafd-44b8-9dd3-ff986f00aa62","Type":"ContainerStarted","Data":"373c592a1f2db71950b6262943cbfb49a131c2ac6d4cfbf68f7161da31cb4e24"}
Jul 28 13:46:30.569414 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:30.568997    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-86slx" event={"ID":"4dc1fb10-eafd-44b8-9dd3-ff986f00aa62","Type":"ContainerStarted","Data":"06042e5a5818e63c18bf8e265e9201c0c6f98c85f724ef182a1bf2a57156fae6"}
Jul 28 13:46:30.569414 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:30.569183    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-86slx"
Jul 28 13:46:30.595094 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:30.595046    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-86slx" podStartSLOduration=2.990656087 podStartE2EDuration="6.595031802s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.604375715s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:46:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:46:30.594168232 +0000 UTC m=+3187.380043356" watchObservedRunningTime="2026-07-28 13:46:30.595031802 +0000 UTC m=+3187.380906913"
Jul 28 13:46:31.573334 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:31.573305    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-86slx"
Jul 28 13:46:31.574565 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:31.574535    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-86slx" podUID="4dc1fb10-eafd-44b8-9dd3-ff986f00aa62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 13:46:32.577439 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:32.577396    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-86slx" podUID="4dc1fb10-eafd-44b8-9dd3-ff986f00aa62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 13:46:37.582341 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:37.582314    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-86slx"
Jul 28 13:46:37.582833 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:37.582806    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-86slx" podUID="4dc1fb10-eafd-44b8-9dd3-ff986f00aa62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 13:46:47.582807 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:47.582771    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-86slx" podUID="4dc1fb10-eafd-44b8-9dd3-ff986f00aa62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 13:46:57.583207 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:46:57.583169    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-86slx" podUID="4dc1fb10-eafd-44b8-9dd3-ff986f00aa62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 13:47:07.583327 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:47:07.583262    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-86slx" podUID="4dc1fb10-eafd-44b8-9dd3-ff986f00aa62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 13:47:17.583195 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:47:17.583100    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-86slx" podUID="4dc1fb10-eafd-44b8-9dd3-ff986f00aa62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 13:47:27.583058 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:47:27.583016    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-86slx" podUID="4dc1fb10-eafd-44b8-9dd3-ff986f00aa62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 13:47:37.582794 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:47:37.582756    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-86slx" podUID="4dc1fb10-eafd-44b8-9dd3-ff986f00aa62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 13:47:47.584079 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:47:47.584044    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-86slx"
Jul 28 13:47:56.263408 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:47:56.263365    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-86slx"]
Jul 28 13:47:56.263852 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:47:56.263640    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-86slx" podUID="4dc1fb10-eafd-44b8-9dd3-ff986f00aa62" containerName="kserve-container" containerID="cri-o://06042e5a5818e63c18bf8e265e9201c0c6f98c85f724ef182a1bf2a57156fae6" gracePeriod=30
Jul 28 13:47:56.263852 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:47:56.263658    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-86slx" podUID="4dc1fb10-eafd-44b8-9dd3-ff986f00aa62" containerName="kube-rbac-proxy" containerID="cri-o://373c592a1f2db71950b6262943cbfb49a131c2ac6d4cfbf68f7161da31cb4e24" gracePeriod=30
Jul 28 13:47:56.905901 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:47:56.905861    2554 generic.go:356] "Generic (PLEG): container finished" podID="4dc1fb10-eafd-44b8-9dd3-ff986f00aa62" containerID="373c592a1f2db71950b6262943cbfb49a131c2ac6d4cfbf68f7161da31cb4e24" exitCode=2
Jul 28 13:47:56.906074 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:47:56.905910    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-86slx" event={"ID":"4dc1fb10-eafd-44b8-9dd3-ff986f00aa62","Type":"ContainerDied","Data":"373c592a1f2db71950b6262943cbfb49a131c2ac6d4cfbf68f7161da31cb4e24"}
Jul 28 13:47:57.578118 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:47:57.578077    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-86slx" podUID="4dc1fb10-eafd-44b8-9dd3-ff986f00aa62" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
Jul 28 13:47:57.582848 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:47:57.582825    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-86slx" podUID="4dc1fb10-eafd-44b8-9dd3-ff986f00aa62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 28 13:47:59.704994 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:47:59.704968    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-86slx"
Jul 28 13:47:59.782365 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:47:59.782267    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqqxs\" (UniqueName: \"kubernetes.io/projected/4dc1fb10-eafd-44b8-9dd3-ff986f00aa62-kube-api-access-kqqxs\") pod \"4dc1fb10-eafd-44b8-9dd3-ff986f00aa62\" (UID: \"4dc1fb10-eafd-44b8-9dd3-ff986f00aa62\") "
Jul 28 13:47:59.782365 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:47:59.782328    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4dc1fb10-eafd-44b8-9dd3-ff986f00aa62-cabundle-cert\") pod \"4dc1fb10-eafd-44b8-9dd3-ff986f00aa62\" (UID: \"4dc1fb10-eafd-44b8-9dd3-ff986f00aa62\") "
Jul 28 13:47:59.782365 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:47:59.782356    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4dc1fb10-eafd-44b8-9dd3-ff986f00aa62-kube-rbac-proxy-sar-config\") pod \"4dc1fb10-eafd-44b8-9dd3-ff986f00aa62\" (UID: \"4dc1fb10-eafd-44b8-9dd3-ff986f00aa62\") "
Jul 28 13:47:59.782679 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:47:59.782513    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4dc1fb10-eafd-44b8-9dd3-ff986f00aa62-kserve-provision-location\") pod \"4dc1fb10-eafd-44b8-9dd3-ff986f00aa62\" (UID: \"4dc1fb10-eafd-44b8-9dd3-ff986f00aa62\") "
Jul 28 13:47:59.782679 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:47:59.782561    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4dc1fb10-eafd-44b8-9dd3-ff986f00aa62-proxy-tls\") pod \"4dc1fb10-eafd-44b8-9dd3-ff986f00aa62\" (UID: \"4dc1fb10-eafd-44b8-9dd3-ff986f00aa62\") "
Jul 28 13:47:59.782679 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:47:59.782672    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dc1fb10-eafd-44b8-9dd3-ff986f00aa62-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4dc1fb10-eafd-44b8-9dd3-ff986f00aa62" (UID: "4dc1fb10-eafd-44b8-9dd3-ff986f00aa62"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:47:59.782846 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:47:59.782715    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dc1fb10-eafd-44b8-9dd3-ff986f00aa62-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4dc1fb10-eafd-44b8-9dd3-ff986f00aa62" (UID: "4dc1fb10-eafd-44b8-9dd3-ff986f00aa62"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:47:59.782897 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:47:59.782845    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dc1fb10-eafd-44b8-9dd3-ff986f00aa62-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4dc1fb10-eafd-44b8-9dd3-ff986f00aa62" (UID: "4dc1fb10-eafd-44b8-9dd3-ff986f00aa62"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:47:59.782953 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:47:59.782938    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4dc1fb10-eafd-44b8-9dd3-ff986f00aa62-cabundle-cert\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:47:59.782993 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:47:59.782956    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4dc1fb10-eafd-44b8-9dd3-ff986f00aa62-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:47:59.782993 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:47:59.782966    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4dc1fb10-eafd-44b8-9dd3-ff986f00aa62-kserve-provision-location\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:47:59.784593 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:47:59.784571    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dc1fb10-eafd-44b8-9dd3-ff986f00aa62-kube-api-access-kqqxs" (OuterVolumeSpecName: "kube-api-access-kqqxs") pod "4dc1fb10-eafd-44b8-9dd3-ff986f00aa62" (UID: "4dc1fb10-eafd-44b8-9dd3-ff986f00aa62"). InnerVolumeSpecName "kube-api-access-kqqxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:47:59.784704 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:47:59.784643    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dc1fb10-eafd-44b8-9dd3-ff986f00aa62-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4dc1fb10-eafd-44b8-9dd3-ff986f00aa62" (UID: "4dc1fb10-eafd-44b8-9dd3-ff986f00aa62"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:47:59.884106 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:47:59.884069    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4dc1fb10-eafd-44b8-9dd3-ff986f00aa62-proxy-tls\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:47:59.884106 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:47:59.884102    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqqxs\" (UniqueName: \"kubernetes.io/projected/4dc1fb10-eafd-44b8-9dd3-ff986f00aa62-kube-api-access-kqqxs\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:47:59.918862 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:47:59.918831    2554 generic.go:356] "Generic (PLEG): container finished" podID="4dc1fb10-eafd-44b8-9dd3-ff986f00aa62" containerID="06042e5a5818e63c18bf8e265e9201c0c6f98c85f724ef182a1bf2a57156fae6" exitCode=0
Jul 28 13:47:59.918995 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:47:59.918889    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-86slx"
Jul 28 13:47:59.918995 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:47:59.918914    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-86slx" event={"ID":"4dc1fb10-eafd-44b8-9dd3-ff986f00aa62","Type":"ContainerDied","Data":"06042e5a5818e63c18bf8e265e9201c0c6f98c85f724ef182a1bf2a57156fae6"}
Jul 28 13:47:59.918995 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:47:59.918974    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-86slx" event={"ID":"4dc1fb10-eafd-44b8-9dd3-ff986f00aa62","Type":"ContainerDied","Data":"d0a1b4f8c03dafea3cd00252e25a5ea8dbfecb7e1f4dc4078c40a35b9b090b8d"}
Jul 28 13:47:59.919128 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:47:59.919003    2554 scope.go:117] "RemoveContainer" containerID="373c592a1f2db71950b6262943cbfb49a131c2ac6d4cfbf68f7161da31cb4e24"
Jul 28 13:47:59.927463 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:47:59.927447    2554 scope.go:117] "RemoveContainer" containerID="06042e5a5818e63c18bf8e265e9201c0c6f98c85f724ef182a1bf2a57156fae6"
Jul 28 13:47:59.934826 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:47:59.934810    2554 scope.go:117] "RemoveContainer" containerID="bdd75b39ae7e63fc455e98eca06a7219a34ee2431ed34f7a710af4122672ffcb"
Jul 28 13:47:59.939448 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:47:59.939427    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-86slx"]
Jul 28 13:47:59.942550 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:47:59.942536    2554 scope.go:117] "RemoveContainer" containerID="373c592a1f2db71950b6262943cbfb49a131c2ac6d4cfbf68f7161da31cb4e24"
Jul 28 13:47:59.942752 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:47:59.942737    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"373c592a1f2db71950b6262943cbfb49a131c2ac6d4cfbf68f7161da31cb4e24\": container with ID starting with 373c592a1f2db71950b6262943cbfb49a131c2ac6d4cfbf68f7161da31cb4e24 not found: ID does not exist" containerID="373c592a1f2db71950b6262943cbfb49a131c2ac6d4cfbf68f7161da31cb4e24"
Jul 28 13:47:59.942805 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:47:59.942760    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"373c592a1f2db71950b6262943cbfb49a131c2ac6d4cfbf68f7161da31cb4e24"} err="failed to get container status \"373c592a1f2db71950b6262943cbfb49a131c2ac6d4cfbf68f7161da31cb4e24\": rpc error: code = NotFound desc = could not find container \"373c592a1f2db71950b6262943cbfb49a131c2ac6d4cfbf68f7161da31cb4e24\": container with ID starting with 373c592a1f2db71950b6262943cbfb49a131c2ac6d4cfbf68f7161da31cb4e24 not found: ID does not exist"
Jul 28 13:47:59.942805 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:47:59.942778    2554 scope.go:117] "RemoveContainer" containerID="06042e5a5818e63c18bf8e265e9201c0c6f98c85f724ef182a1bf2a57156fae6"
Jul 28 13:47:59.943012 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:47:59.942994    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06042e5a5818e63c18bf8e265e9201c0c6f98c85f724ef182a1bf2a57156fae6\": container with ID starting with 06042e5a5818e63c18bf8e265e9201c0c6f98c85f724ef182a1bf2a57156fae6 not found: ID does not exist" containerID="06042e5a5818e63c18bf8e265e9201c0c6f98c85f724ef182a1bf2a57156fae6"
Jul 28 13:47:59.943078 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:47:59.943021    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06042e5a5818e63c18bf8e265e9201c0c6f98c85f724ef182a1bf2a57156fae6"} err="failed to get container status \"06042e5a5818e63c18bf8e265e9201c0c6f98c85f724ef182a1bf2a57156fae6\": rpc error: code = NotFound desc = could not find container \"06042e5a5818e63c18bf8e265e9201c0c6f98c85f724ef182a1bf2a57156fae6\": container with ID starting with 06042e5a5818e63c18bf8e265e9201c0c6f98c85f724ef182a1bf2a57156fae6 not found: ID does not exist"
Jul 28 13:47:59.943078 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:47:59.943046    2554 scope.go:117] "RemoveContainer" containerID="bdd75b39ae7e63fc455e98eca06a7219a34ee2431ed34f7a710af4122672ffcb"
Jul 28 13:47:59.943296 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:47:59.943262    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdd75b39ae7e63fc455e98eca06a7219a34ee2431ed34f7a710af4122672ffcb\": container with ID starting with bdd75b39ae7e63fc455e98eca06a7219a34ee2431ed34f7a710af4122672ffcb not found: ID does not exist" containerID="bdd75b39ae7e63fc455e98eca06a7219a34ee2431ed34f7a710af4122672ffcb"
Jul 28 13:47:59.943339 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:47:59.943294    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdd75b39ae7e63fc455e98eca06a7219a34ee2431ed34f7a710af4122672ffcb"} err="failed to get container status \"bdd75b39ae7e63fc455e98eca06a7219a34ee2431ed34f7a710af4122672ffcb\": rpc error: code = NotFound desc = could not find container \"bdd75b39ae7e63fc455e98eca06a7219a34ee2431ed34f7a710af4122672ffcb\": container with ID starting with bdd75b39ae7e63fc455e98eca06a7219a34ee2431ed34f7a710af4122672ffcb not found: ID does not exist"
Jul 28 13:47:59.950099 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:47:59.950079    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-86slx"]
Jul 28 13:48:01.392977 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:48:01.392938    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-stz59"]
Jul 28 13:48:01.393462 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:48:01.393294    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dc1fb10-eafd-44b8-9dd3-ff986f00aa62" containerName="storage-initializer"
Jul 28 13:48:01.393462 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:48:01.393306    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dc1fb10-eafd-44b8-9dd3-ff986f00aa62" containerName="storage-initializer"
Jul 28 13:48:01.393462 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:48:01.393322    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dc1fb10-eafd-44b8-9dd3-ff986f00aa62" containerName="kserve-container"
Jul 28 13:48:01.393462 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:48:01.393328    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dc1fb10-eafd-44b8-9dd3-ff986f00aa62" containerName="kserve-container"
Jul 28 13:48:01.393462 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:48:01.393340    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dc1fb10-eafd-44b8-9dd3-ff986f00aa62" containerName="kube-rbac-proxy"
Jul 28 13:48:01.393462 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:48:01.393346    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dc1fb10-eafd-44b8-9dd3-ff986f00aa62" containerName="kube-rbac-proxy"
Jul 28 13:48:01.393462 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:48:01.393394    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="4dc1fb10-eafd-44b8-9dd3-ff986f00aa62" containerName="kserve-container"
Jul 28 13:48:01.393462 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:48:01.393401    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="4dc1fb10-eafd-44b8-9dd3-ff986f00aa62" containerName="kube-rbac-proxy"
Jul 28 13:48:01.398011 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:48:01.397990    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-stz59"
Jul 28 13:48:01.401140 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:48:01.401117    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
Jul 28 13:48:01.401900 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:48:01.401876    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:48:01.402031 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:48:01.401877    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:48:01.402031 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:48:01.401989    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rxfwd\""
Jul 28 13:48:01.402578 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:48:01.402556    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:48:01.402694 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:48:01.402590    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
Jul 28 13:48:01.402694 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:48:01.402569    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:48:01.411507 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:48:01.411484    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-stz59"]
Jul 28 13:48:01.497216 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:48:01.497184    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67f00e5d-1692-41fc-ae31-1d9d4b60a884-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-stz59\" (UID: \"67f00e5d-1692-41fc-ae31-1d9d4b60a884\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-stz59"
Jul 28 13:48:01.497401 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:48:01.497256    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67f00e5d-1692-41fc-ae31-1d9d4b60a884-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-stz59\" (UID: \"67f00e5d-1692-41fc-ae31-1d9d4b60a884\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-stz59"
Jul 28 13:48:01.497401 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:48:01.497296    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67f00e5d-1692-41fc-ae31-1d9d4b60a884-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-stz59\" (UID: \"67f00e5d-1692-41fc-ae31-1d9d4b60a884\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-stz59"
Jul 28 13:48:01.497401 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:48:01.497330    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67f00e5d-1692-41fc-ae31-1d9d4b60a884-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-stz59\" (UID: \"67f00e5d-1692-41fc-ae31-1d9d4b60a884\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-stz59"
Jul 28 13:48:01.497401 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:48:01.497346    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lndt\" (UniqueName: \"kubernetes.io/projected/67f00e5d-1692-41fc-ae31-1d9d4b60a884-kube-api-access-9lndt\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-stz59\" (UID: \"67f00e5d-1692-41fc-ae31-1d9d4b60a884\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-stz59"
Jul 28 13:48:01.598405 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:48:01.598369    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67f00e5d-1692-41fc-ae31-1d9d4b60a884-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-stz59\" (UID: \"67f00e5d-1692-41fc-ae31-1d9d4b60a884\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-stz59"
Jul 28 13:48:01.598405 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:48:01.598409    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lndt\" (UniqueName: \"kubernetes.io/projected/67f00e5d-1692-41fc-ae31-1d9d4b60a884-kube-api-access-9lndt\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-stz59\" (UID: \"67f00e5d-1692-41fc-ae31-1d9d4b60a884\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-stz59"
Jul 28 13:48:01.598649 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:48:01.598532    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67f00e5d-1692-41fc-ae31-1d9d4b60a884-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-stz59\" (UID: \"67f00e5d-1692-41fc-ae31-1d9d4b60a884\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-stz59"
Jul 28 13:48:01.598649 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:48:01.598639    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67f00e5d-1692-41fc-ae31-1d9d4b60a884-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-stz59\" (UID: \"67f00e5d-1692-41fc-ae31-1d9d4b60a884\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-stz59"
Jul 28 13:48:01.598763 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:48:01.598686    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67f00e5d-1692-41fc-ae31-1d9d4b60a884-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-stz59\" (UID: \"67f00e5d-1692-41fc-ae31-1d9d4b60a884\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-stz59"
Jul 28 13:48:01.599021 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:48:01.599002    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67f00e5d-1692-41fc-ae31-1d9d4b60a884-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-stz59\" (UID: \"67f00e5d-1692-41fc-ae31-1d9d4b60a884\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-stz59"
Jul 28 13:48:01.599096 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:48:01.599076    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67f00e5d-1692-41fc-ae31-1d9d4b60a884-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-stz59\" (UID: \"67f00e5d-1692-41fc-ae31-1d9d4b60a884\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-stz59"
Jul 28 13:48:01.599183 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:48:01.599166    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67f00e5d-1692-41fc-ae31-1d9d4b60a884-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-stz59\" (UID: \"67f00e5d-1692-41fc-ae31-1d9d4b60a884\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-stz59"
Jul 28 13:48:01.601311 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:48:01.601263    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67f00e5d-1692-41fc-ae31-1d9d4b60a884-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-stz59\" (UID: \"67f00e5d-1692-41fc-ae31-1d9d4b60a884\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-stz59"
Jul 28 13:48:01.608625 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:48:01.608603    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lndt\" (UniqueName: \"kubernetes.io/projected/67f00e5d-1692-41fc-ae31-1d9d4b60a884-kube-api-access-9lndt\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-stz59\" (UID: \"67f00e5d-1692-41fc-ae31-1d9d4b60a884\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-stz59"
Jul 28 13:48:01.709154 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:48:01.709129    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-stz59"
Jul 28 13:48:01.803354 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:48:01.803313    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4dc1fb10-eafd-44b8-9dd3-ff986f00aa62" path="/var/lib/kubelet/pods/4dc1fb10-eafd-44b8-9dd3-ff986f00aa62/volumes"
Jul 28 13:48:01.849387 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:48:01.849353    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-stz59"]
Jul 28 13:48:01.852359 ip-10-0-129-177 kubenswrapper[2554]: W0728 13:48:01.852328    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67f00e5d_1692_41fc_ae31_1d9d4b60a884.slice/crio-b5164f684f92e400d2a810fb26c3de3196fc5f250bf3e2bcf6a1f4f693bb57a4 WatchSource:0}: Error finding container b5164f684f92e400d2a810fb26c3de3196fc5f250bf3e2bcf6a1f4f693bb57a4: Status 404 returned error can't find the container with id b5164f684f92e400d2a810fb26c3de3196fc5f250bf3e2bcf6a1f4f693bb57a4
Jul 28 13:48:01.930771 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:48:01.930737    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-stz59" event={"ID":"67f00e5d-1692-41fc-ae31-1d9d4b60a884","Type":"ContainerStarted","Data":"ac89281eaf24420d3a57992d19c9577199b0f1dbc593239525c3d169921c4176"}
Jul 28 13:48:01.930901 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:48:01.930811    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-stz59" event={"ID":"67f00e5d-1692-41fc-ae31-1d9d4b60a884","Type":"ContainerStarted","Data":"b5164f684f92e400d2a810fb26c3de3196fc5f250bf3e2bcf6a1f4f693bb57a4"}
Jul 28 13:48:05.950184 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:48:05.950149    2554 generic.go:356] "Generic (PLEG): container finished" podID="67f00e5d-1692-41fc-ae31-1d9d4b60a884" containerID="ac89281eaf24420d3a57992d19c9577199b0f1dbc593239525c3d169921c4176" exitCode=0
Jul 28 13:48:05.950581 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:48:05.950205    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-stz59" event={"ID":"67f00e5d-1692-41fc-ae31-1d9d4b60a884","Type":"ContainerDied","Data":"ac89281eaf24420d3a57992d19c9577199b0f1dbc593239525c3d169921c4176"}
Jul 28 13:48:06.957662 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:48:06.957629    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-stz59" event={"ID":"67f00e5d-1692-41fc-ae31-1d9d4b60a884","Type":"ContainerStarted","Data":"16174f024e5db40785a9d1e42a6950b31be2a72d4ed1200688b57b7278605934"}
Jul 28 13:48:06.958041 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:48:06.957690    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-stz59" event={"ID":"67f00e5d-1692-41fc-ae31-1d9d4b60a884","Type":"ContainerStarted","Data":"3067dd3c62963c8cd864eb3ac3d707db8e71b9757522f27a90f5a4fbf0aa5a97"}
Jul 28 13:48:06.958041 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:48:06.957872    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-stz59"
Jul 28 13:48:06.980006 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:48:06.979955    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-stz59" podStartSLOduration=2.256449233 podStartE2EDuration="5.979939846s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.723490613s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:48:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:48:06.978085302 +0000 UTC m=+3283.763960427" watchObservedRunningTime="2026-07-28 13:48:06.979939846 +0000 UTC m=+3283.765814957"
Jul 28 13:48:07.962259 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:48:07.962228    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-stz59"
Jul 28 13:48:07.963576 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:48:07.963549    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-stz59" podUID="67f00e5d-1692-41fc-ae31-1d9d4b60a884" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 13:48:08.966762 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:48:08.966724    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-stz59" podUID="67f00e5d-1692-41fc-ae31-1d9d4b60a884" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 13:48:13.972349 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:48:13.972314    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-stz59"
Jul 28 13:48:13.974840 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:48:13.972909    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-stz59" podUID="67f00e5d-1692-41fc-ae31-1d9d4b60a884" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 13:48:23.973118 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:48:23.973077    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-stz59" podUID="67f00e5d-1692-41fc-ae31-1d9d4b60a884" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 13:48:33.972869 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:48:33.972827    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-stz59" podUID="67f00e5d-1692-41fc-ae31-1d9d4b60a884" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 13:48:43.973650 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:48:43.973609    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-stz59" podUID="67f00e5d-1692-41fc-ae31-1d9d4b60a884" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 13:48:53.973787 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:48:53.973698    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-stz59" podUID="67f00e5d-1692-41fc-ae31-1d9d4b60a884" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 13:49:03.973181 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:03.973143    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-stz59" podUID="67f00e5d-1692-41fc-ae31-1d9d4b60a884" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 13:49:13.973512 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:13.973475    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-stz59" podUID="67f00e5d-1692-41fc-ae31-1d9d4b60a884" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 13:49:23.974292 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:23.974254    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-stz59"
Jul 28 13:49:32.415603 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:32.415573    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-stz59"]
Jul 28 13:49:32.416149 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:32.415867    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-stz59" podUID="67f00e5d-1692-41fc-ae31-1d9d4b60a884" containerName="kserve-container" containerID="cri-o://3067dd3c62963c8cd864eb3ac3d707db8e71b9757522f27a90f5a4fbf0aa5a97" gracePeriod=30
Jul 28 13:49:32.416149 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:32.415962    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-stz59" podUID="67f00e5d-1692-41fc-ae31-1d9d4b60a884" containerName="kube-rbac-proxy" containerID="cri-o://16174f024e5db40785a9d1e42a6950b31be2a72d4ed1200688b57b7278605934" gracePeriod=30
Jul 28 13:49:33.308795 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:33.308761    2554 generic.go:356] "Generic (PLEG): container finished" podID="67f00e5d-1692-41fc-ae31-1d9d4b60a884" containerID="16174f024e5db40785a9d1e42a6950b31be2a72d4ed1200688b57b7278605934" exitCode=2
Jul 28 13:49:33.308970 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:33.308836    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-stz59" event={"ID":"67f00e5d-1692-41fc-ae31-1d9d4b60a884","Type":"ContainerDied","Data":"16174f024e5db40785a9d1e42a6950b31be2a72d4ed1200688b57b7278605934"}
Jul 28 13:49:33.966969 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:33.966929    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-stz59" podUID="67f00e5d-1692-41fc-ae31-1d9d4b60a884" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Jul 28 13:49:33.972914 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:33.972886    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-stz59" podUID="67f00e5d-1692-41fc-ae31-1d9d4b60a884" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 28 13:49:35.855965 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:35.855936    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-stz59"
Jul 28 13:49:35.985223 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:35.985137    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9lndt\" (UniqueName: \"kubernetes.io/projected/67f00e5d-1692-41fc-ae31-1d9d4b60a884-kube-api-access-9lndt\") pod \"67f00e5d-1692-41fc-ae31-1d9d4b60a884\" (UID: \"67f00e5d-1692-41fc-ae31-1d9d4b60a884\") "
Jul 28 13:49:35.985223 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:35.985204    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67f00e5d-1692-41fc-ae31-1d9d4b60a884-kube-rbac-proxy-sar-config\") pod \"67f00e5d-1692-41fc-ae31-1d9d4b60a884\" (UID: \"67f00e5d-1692-41fc-ae31-1d9d4b60a884\") "
Jul 28 13:49:35.985223 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:35.985221    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67f00e5d-1692-41fc-ae31-1d9d4b60a884-cabundle-cert\") pod \"67f00e5d-1692-41fc-ae31-1d9d4b60a884\" (UID: \"67f00e5d-1692-41fc-ae31-1d9d4b60a884\") "
Jul 28 13:49:35.985519 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:35.985291    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67f00e5d-1692-41fc-ae31-1d9d4b60a884-proxy-tls\") pod \"67f00e5d-1692-41fc-ae31-1d9d4b60a884\" (UID: \"67f00e5d-1692-41fc-ae31-1d9d4b60a884\") "
Jul 28 13:49:35.985519 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:35.985329    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67f00e5d-1692-41fc-ae31-1d9d4b60a884-kserve-provision-location\") pod \"67f00e5d-1692-41fc-ae31-1d9d4b60a884\" (UID: \"67f00e5d-1692-41fc-ae31-1d9d4b60a884\") "
Jul 28 13:49:35.985632 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:35.985542    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67f00e5d-1692-41fc-ae31-1d9d4b60a884-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "67f00e5d-1692-41fc-ae31-1d9d4b60a884" (UID: "67f00e5d-1692-41fc-ae31-1d9d4b60a884"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:49:35.985632 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:35.985558    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67f00e5d-1692-41fc-ae31-1d9d4b60a884-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "67f00e5d-1692-41fc-ae31-1d9d4b60a884" (UID: "67f00e5d-1692-41fc-ae31-1d9d4b60a884"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:49:35.985732 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:35.985695    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67f00e5d-1692-41fc-ae31-1d9d4b60a884-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "67f00e5d-1692-41fc-ae31-1d9d4b60a884" (UID: "67f00e5d-1692-41fc-ae31-1d9d4b60a884"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:49:35.987506 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:35.987479    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67f00e5d-1692-41fc-ae31-1d9d4b60a884-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "67f00e5d-1692-41fc-ae31-1d9d4b60a884" (UID: "67f00e5d-1692-41fc-ae31-1d9d4b60a884"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:49:35.987506 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:35.987479    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67f00e5d-1692-41fc-ae31-1d9d4b60a884-kube-api-access-9lndt" (OuterVolumeSpecName: "kube-api-access-9lndt") pod "67f00e5d-1692-41fc-ae31-1d9d4b60a884" (UID: "67f00e5d-1692-41fc-ae31-1d9d4b60a884"). InnerVolumeSpecName "kube-api-access-9lndt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:49:36.086774 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:36.086731    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67f00e5d-1692-41fc-ae31-1d9d4b60a884-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:49:36.086774 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:36.086768    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67f00e5d-1692-41fc-ae31-1d9d4b60a884-cabundle-cert\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:49:36.086774 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:36.086779    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67f00e5d-1692-41fc-ae31-1d9d4b60a884-proxy-tls\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:49:36.086774 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:36.086788    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67f00e5d-1692-41fc-ae31-1d9d4b60a884-kserve-provision-location\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:49:36.087023 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:36.086797    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lndt\" (UniqueName: \"kubernetes.io/projected/67f00e5d-1692-41fc-ae31-1d9d4b60a884-kube-api-access-9lndt\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:49:36.323037 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:36.322954    2554 generic.go:356] "Generic (PLEG): container finished" podID="67f00e5d-1692-41fc-ae31-1d9d4b60a884" containerID="3067dd3c62963c8cd864eb3ac3d707db8e71b9757522f27a90f5a4fbf0aa5a97" exitCode=0
Jul 28 13:49:36.323037 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:36.323007    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-stz59" event={"ID":"67f00e5d-1692-41fc-ae31-1d9d4b60a884","Type":"ContainerDied","Data":"3067dd3c62963c8cd864eb3ac3d707db8e71b9757522f27a90f5a4fbf0aa5a97"}
Jul 28 13:49:36.323037 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:36.323014    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-stz59"
Jul 28 13:49:36.323301 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:36.323059    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-stz59" event={"ID":"67f00e5d-1692-41fc-ae31-1d9d4b60a884","Type":"ContainerDied","Data":"b5164f684f92e400d2a810fb26c3de3196fc5f250bf3e2bcf6a1f4f693bb57a4"}
Jul 28 13:49:36.323301 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:36.323087    2554 scope.go:117] "RemoveContainer" containerID="16174f024e5db40785a9d1e42a6950b31be2a72d4ed1200688b57b7278605934"
Jul 28 13:49:36.332142 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:36.332125    2554 scope.go:117] "RemoveContainer" containerID="3067dd3c62963c8cd864eb3ac3d707db8e71b9757522f27a90f5a4fbf0aa5a97"
Jul 28 13:49:36.340009 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:36.339992    2554 scope.go:117] "RemoveContainer" containerID="ac89281eaf24420d3a57992d19c9577199b0f1dbc593239525c3d169921c4176"
Jul 28 13:49:36.348191 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:36.348170    2554 scope.go:117] "RemoveContainer" containerID="16174f024e5db40785a9d1e42a6950b31be2a72d4ed1200688b57b7278605934"
Jul 28 13:49:36.348483 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:49:36.348458    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16174f024e5db40785a9d1e42a6950b31be2a72d4ed1200688b57b7278605934\": container with ID starting with 16174f024e5db40785a9d1e42a6950b31be2a72d4ed1200688b57b7278605934 not found: ID does not exist" containerID="16174f024e5db40785a9d1e42a6950b31be2a72d4ed1200688b57b7278605934"
Jul 28 13:49:36.348582 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:36.348492    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16174f024e5db40785a9d1e42a6950b31be2a72d4ed1200688b57b7278605934"} err="failed to get container status \"16174f024e5db40785a9d1e42a6950b31be2a72d4ed1200688b57b7278605934\": rpc error: code = NotFound desc = could not find container \"16174f024e5db40785a9d1e42a6950b31be2a72d4ed1200688b57b7278605934\": container with ID starting with 16174f024e5db40785a9d1e42a6950b31be2a72d4ed1200688b57b7278605934 not found: ID does not exist"
Jul 28 13:49:36.348582 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:36.348510    2554 scope.go:117] "RemoveContainer" containerID="3067dd3c62963c8cd864eb3ac3d707db8e71b9757522f27a90f5a4fbf0aa5a97"
Jul 28 13:49:36.348774 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:49:36.348752    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3067dd3c62963c8cd864eb3ac3d707db8e71b9757522f27a90f5a4fbf0aa5a97\": container with ID starting with 3067dd3c62963c8cd864eb3ac3d707db8e71b9757522f27a90f5a4fbf0aa5a97 not found: ID does not exist" containerID="3067dd3c62963c8cd864eb3ac3d707db8e71b9757522f27a90f5a4fbf0aa5a97"
Jul 28 13:49:36.348833 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:36.348784    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3067dd3c62963c8cd864eb3ac3d707db8e71b9757522f27a90f5a4fbf0aa5a97"} err="failed to get container status \"3067dd3c62963c8cd864eb3ac3d707db8e71b9757522f27a90f5a4fbf0aa5a97\": rpc error: code = NotFound desc = could not find container \"3067dd3c62963c8cd864eb3ac3d707db8e71b9757522f27a90f5a4fbf0aa5a97\": container with ID starting with 3067dd3c62963c8cd864eb3ac3d707db8e71b9757522f27a90f5a4fbf0aa5a97 not found: ID does not exist"
Jul 28 13:49:36.348833 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:36.348805    2554 scope.go:117] "RemoveContainer" containerID="ac89281eaf24420d3a57992d19c9577199b0f1dbc593239525c3d169921c4176"
Jul 28 13:49:36.349075 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:49:36.349057    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac89281eaf24420d3a57992d19c9577199b0f1dbc593239525c3d169921c4176\": container with ID starting with ac89281eaf24420d3a57992d19c9577199b0f1dbc593239525c3d169921c4176 not found: ID does not exist" containerID="ac89281eaf24420d3a57992d19c9577199b0f1dbc593239525c3d169921c4176"
Jul 28 13:49:36.349133 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:36.349082    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac89281eaf24420d3a57992d19c9577199b0f1dbc593239525c3d169921c4176"} err="failed to get container status \"ac89281eaf24420d3a57992d19c9577199b0f1dbc593239525c3d169921c4176\": rpc error: code = NotFound desc = could not find container \"ac89281eaf24420d3a57992d19c9577199b0f1dbc593239525c3d169921c4176\": container with ID starting with ac89281eaf24420d3a57992d19c9577199b0f1dbc593239525c3d169921c4176 not found: ID does not exist"
Jul 28 13:49:36.349681 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:36.349662    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-stz59"]
Jul 28 13:49:36.350992 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:36.350976    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-stz59"]
Jul 28 13:49:37.623581 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:37.623548    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d7d905-predictor-6778c84d57-vlqfx"]
Jul 28 13:49:37.623995 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:37.623883    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67f00e5d-1692-41fc-ae31-1d9d4b60a884" containerName="storage-initializer"
Jul 28 13:49:37.623995 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:37.623895    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="67f00e5d-1692-41fc-ae31-1d9d4b60a884" containerName="storage-initializer"
Jul 28 13:49:37.623995 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:37.623905    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67f00e5d-1692-41fc-ae31-1d9d4b60a884" containerName="kserve-container"
Jul 28 13:49:37.623995 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:37.623911    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="67f00e5d-1692-41fc-ae31-1d9d4b60a884" containerName="kserve-container"
Jul 28 13:49:37.623995 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:37.623927    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67f00e5d-1692-41fc-ae31-1d9d4b60a884" containerName="kube-rbac-proxy"
Jul 28 13:49:37.623995 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:37.623935    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="67f00e5d-1692-41fc-ae31-1d9d4b60a884" containerName="kube-rbac-proxy"
Jul 28 13:49:37.623995 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:37.623988    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="67f00e5d-1692-41fc-ae31-1d9d4b60a884" containerName="kserve-container"
Jul 28 13:49:37.623995 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:37.623998    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="67f00e5d-1692-41fc-ae31-1d9d4b60a884" containerName="kube-rbac-proxy"
Jul 28 13:49:37.628688 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:37.628669    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-d7d905-predictor-6778c84d57-vlqfx"
Jul 28 13:49:37.632442 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:37.632421    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:49:37.633136 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:37.633113    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:49:37.633241 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:37.633169    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:49:37.633241 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:37.633195    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:49:37.633421 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:37.633407    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-d7d905-predictor-serving-cert\""
Jul 28 13:49:37.633501 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:37.633484    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rxfwd\""
Jul 28 13:49:37.633559 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:37.633507    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-d7d905-kube-rbac-proxy-sar-config\""
Jul 28 13:49:37.640854 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:37.640832    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d7d905-predictor-6778c84d57-vlqfx"]
Jul 28 13:49:37.700410 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:37.700379    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0f562a0-028b-4704-bc25-2a7f95a87d76-proxy-tls\") pod \"isvc-primary-d7d905-predictor-6778c84d57-vlqfx\" (UID: \"e0f562a0-028b-4704-bc25-2a7f95a87d76\") " pod="kserve-ci-e2e-test/isvc-primary-d7d905-predictor-6778c84d57-vlqfx"
Jul 28 13:49:37.700543 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:37.700416    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0f562a0-028b-4704-bc25-2a7f95a87d76-kserve-provision-location\") pod \"isvc-primary-d7d905-predictor-6778c84d57-vlqfx\" (UID: \"e0f562a0-028b-4704-bc25-2a7f95a87d76\") " pod="kserve-ci-e2e-test/isvc-primary-d7d905-predictor-6778c84d57-vlqfx"
Jul 28 13:49:37.700543 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:37.700435    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0f562a0-028b-4704-bc25-2a7f95a87d76-kube-rbac-proxy-sar-config\") pod \"isvc-primary-d7d905-predictor-6778c84d57-vlqfx\" (UID: \"e0f562a0-028b-4704-bc25-2a7f95a87d76\") " pod="kserve-ci-e2e-test/isvc-primary-d7d905-predictor-6778c84d57-vlqfx"
Jul 28 13:49:37.700543 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:37.700536    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0f562a0-028b-4704-bc25-2a7f95a87d76-cabundle-cert\") pod \"isvc-primary-d7d905-predictor-6778c84d57-vlqfx\" (UID: \"e0f562a0-028b-4704-bc25-2a7f95a87d76\") " pod="kserve-ci-e2e-test/isvc-primary-d7d905-predictor-6778c84d57-vlqfx"
Jul 28 13:49:37.700656 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:37.700612    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkxmx\" (UniqueName: \"kubernetes.io/projected/e0f562a0-028b-4704-bc25-2a7f95a87d76-kube-api-access-rkxmx\") pod \"isvc-primary-d7d905-predictor-6778c84d57-vlqfx\" (UID: \"e0f562a0-028b-4704-bc25-2a7f95a87d76\") " pod="kserve-ci-e2e-test/isvc-primary-d7d905-predictor-6778c84d57-vlqfx"
Jul 28 13:49:37.801216 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:37.801183    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0f562a0-028b-4704-bc25-2a7f95a87d76-cabundle-cert\") pod \"isvc-primary-d7d905-predictor-6778c84d57-vlqfx\" (UID: \"e0f562a0-028b-4704-bc25-2a7f95a87d76\") " pod="kserve-ci-e2e-test/isvc-primary-d7d905-predictor-6778c84d57-vlqfx"
Jul 28 13:49:37.801397 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:37.801249    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkxmx\" (UniqueName: \"kubernetes.io/projected/e0f562a0-028b-4704-bc25-2a7f95a87d76-kube-api-access-rkxmx\") pod \"isvc-primary-d7d905-predictor-6778c84d57-vlqfx\" (UID: \"e0f562a0-028b-4704-bc25-2a7f95a87d76\") " pod="kserve-ci-e2e-test/isvc-primary-d7d905-predictor-6778c84d57-vlqfx"
Jul 28 13:49:37.801397 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:37.801311    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0f562a0-028b-4704-bc25-2a7f95a87d76-proxy-tls\") pod \"isvc-primary-d7d905-predictor-6778c84d57-vlqfx\" (UID: \"e0f562a0-028b-4704-bc25-2a7f95a87d76\") " pod="kserve-ci-e2e-test/isvc-primary-d7d905-predictor-6778c84d57-vlqfx"
Jul 28 13:49:37.801397 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:37.801358    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0f562a0-028b-4704-bc25-2a7f95a87d76-kserve-provision-location\") pod \"isvc-primary-d7d905-predictor-6778c84d57-vlqfx\" (UID: \"e0f562a0-028b-4704-bc25-2a7f95a87d76\") " pod="kserve-ci-e2e-test/isvc-primary-d7d905-predictor-6778c84d57-vlqfx"
Jul 28 13:49:37.801397 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:37.801384    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0f562a0-028b-4704-bc25-2a7f95a87d76-kube-rbac-proxy-sar-config\") pod \"isvc-primary-d7d905-predictor-6778c84d57-vlqfx\" (UID: \"e0f562a0-028b-4704-bc25-2a7f95a87d76\") " pod="kserve-ci-e2e-test/isvc-primary-d7d905-predictor-6778c84d57-vlqfx"
Jul 28 13:49:37.801793 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:37.801767    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0f562a0-028b-4704-bc25-2a7f95a87d76-kserve-provision-location\") pod \"isvc-primary-d7d905-predictor-6778c84d57-vlqfx\" (UID: \"e0f562a0-028b-4704-bc25-2a7f95a87d76\") " pod="kserve-ci-e2e-test/isvc-primary-d7d905-predictor-6778c84d57-vlqfx"
Jul 28 13:49:37.801942 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:37.801926    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0f562a0-028b-4704-bc25-2a7f95a87d76-cabundle-cert\") pod \"isvc-primary-d7d905-predictor-6778c84d57-vlqfx\" (UID: \"e0f562a0-028b-4704-bc25-2a7f95a87d76\") " pod="kserve-ci-e2e-test/isvc-primary-d7d905-predictor-6778c84d57-vlqfx"
Jul 28 13:49:37.802147 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:37.802125    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0f562a0-028b-4704-bc25-2a7f95a87d76-kube-rbac-proxy-sar-config\") pod \"isvc-primary-d7d905-predictor-6778c84d57-vlqfx\" (UID: \"e0f562a0-028b-4704-bc25-2a7f95a87d76\") " pod="kserve-ci-e2e-test/isvc-primary-d7d905-predictor-6778c84d57-vlqfx"
Jul 28 13:49:37.802988 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:37.802969    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67f00e5d-1692-41fc-ae31-1d9d4b60a884" path="/var/lib/kubelet/pods/67f00e5d-1692-41fc-ae31-1d9d4b60a884/volumes"
Jul 28 13:49:37.803973 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:37.803953    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0f562a0-028b-4704-bc25-2a7f95a87d76-proxy-tls\") pod \"isvc-primary-d7d905-predictor-6778c84d57-vlqfx\" (UID: \"e0f562a0-028b-4704-bc25-2a7f95a87d76\") " pod="kserve-ci-e2e-test/isvc-primary-d7d905-predictor-6778c84d57-vlqfx"
Jul 28 13:49:37.811349 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:37.811327    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkxmx\" (UniqueName: \"kubernetes.io/projected/e0f562a0-028b-4704-bc25-2a7f95a87d76-kube-api-access-rkxmx\") pod \"isvc-primary-d7d905-predictor-6778c84d57-vlqfx\" (UID: \"e0f562a0-028b-4704-bc25-2a7f95a87d76\") " pod="kserve-ci-e2e-test/isvc-primary-d7d905-predictor-6778c84d57-vlqfx"
Jul 28 13:49:37.940113 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:37.940037    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-d7d905-predictor-6778c84d57-vlqfx"
Jul 28 13:49:38.076578 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:38.076555    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d7d905-predictor-6778c84d57-vlqfx"]
Jul 28 13:49:38.078501 ip-10-0-129-177 kubenswrapper[2554]: W0728 13:49:38.078475    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0f562a0_028b_4704_bc25_2a7f95a87d76.slice/crio-1ce9e3c21f34bd700c0cf190187905145bfc6e36900663c05cca70f32768f7f7 WatchSource:0}: Error finding container 1ce9e3c21f34bd700c0cf190187905145bfc6e36900663c05cca70f32768f7f7: Status 404 returned error can't find the container with id 1ce9e3c21f34bd700c0cf190187905145bfc6e36900663c05cca70f32768f7f7
Jul 28 13:49:38.335198 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:38.335155    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d7d905-predictor-6778c84d57-vlqfx" event={"ID":"e0f562a0-028b-4704-bc25-2a7f95a87d76","Type":"ContainerStarted","Data":"bf7d0b2504b676d792e8187c6618a5d19f02f59a9a1fca4b0c374d5e6eab5566"}
Jul 28 13:49:38.335430 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:38.335235    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d7d905-predictor-6778c84d57-vlqfx" event={"ID":"e0f562a0-028b-4704-bc25-2a7f95a87d76","Type":"ContainerStarted","Data":"1ce9e3c21f34bd700c0cf190187905145bfc6e36900663c05cca70f32768f7f7"}
Jul 28 13:49:42.352131 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:42.352045    2554 generic.go:356] "Generic (PLEG): container finished" podID="e0f562a0-028b-4704-bc25-2a7f95a87d76" containerID="bf7d0b2504b676d792e8187c6618a5d19f02f59a9a1fca4b0c374d5e6eab5566" exitCode=0
Jul 28 13:49:42.352131 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:42.352090    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d7d905-predictor-6778c84d57-vlqfx" event={"ID":"e0f562a0-028b-4704-bc25-2a7f95a87d76","Type":"ContainerDied","Data":"bf7d0b2504b676d792e8187c6618a5d19f02f59a9a1fca4b0c374d5e6eab5566"}
Jul 28 13:49:43.358725 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:43.358693    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d7d905-predictor-6778c84d57-vlqfx" event={"ID":"e0f562a0-028b-4704-bc25-2a7f95a87d76","Type":"ContainerStarted","Data":"60ed0125fae585accf1e8d943f78f4a08eb03654205596c635a3fa2487ad9e71"}
Jul 28 13:49:43.359109 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:43.358755    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d7d905-predictor-6778c84d57-vlqfx" event={"ID":"e0f562a0-028b-4704-bc25-2a7f95a87d76","Type":"ContainerStarted","Data":"517f0c02891ef394624d0857e51916821d5742c7d29d8f02b4ea1ce3c389277c"}
Jul 28 13:49:43.359109 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:43.359004    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-d7d905-predictor-6778c84d57-vlqfx"
Jul 28 13:49:43.383101 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:43.383059    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-d7d905-predictor-6778c84d57-vlqfx" podStartSLOduration=2.5739666100000003 podStartE2EDuration="6.383043194s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.809076584s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:49:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:49:43.381045248 +0000 UTC m=+3380.166920360" watchObservedRunningTime="2026-07-28 13:49:43.383043194 +0000 UTC m=+3380.168918304"
Jul 28 13:49:44.362638 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:44.362607    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-d7d905-predictor-6778c84d57-vlqfx"
Jul 28 13:49:44.363929 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:44.363903    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d7d905-predictor-6778c84d57-vlqfx" podUID="e0f562a0-028b-4704-bc25-2a7f95a87d76" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 13:49:45.366759 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:45.366711    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d7d905-predictor-6778c84d57-vlqfx" podUID="e0f562a0-028b-4704-bc25-2a7f95a87d76" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 13:49:50.371771 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:50.371737    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-d7d905-predictor-6778c84d57-vlqfx"
Jul 28 13:49:50.372405 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:49:50.372378    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d7d905-predictor-6778c84d57-vlqfx" podUID="e0f562a0-028b-4704-bc25-2a7f95a87d76" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 13:50:00.372774 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:50:00.372730    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d7d905-predictor-6778c84d57-vlqfx" podUID="e0f562a0-028b-4704-bc25-2a7f95a87d76" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 13:50:10.373289 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:50:10.373240    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d7d905-predictor-6778c84d57-vlqfx" podUID="e0f562a0-028b-4704-bc25-2a7f95a87d76" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 13:50:20.373197 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:50:20.373115    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d7d905-predictor-6778c84d57-vlqfx" podUID="e0f562a0-028b-4704-bc25-2a7f95a87d76" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 13:50:30.372453 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:50:30.372416    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d7d905-predictor-6778c84d57-vlqfx" podUID="e0f562a0-028b-4704-bc25-2a7f95a87d76" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 13:50:40.373100 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:50:40.373062    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d7d905-predictor-6778c84d57-vlqfx" podUID="e0f562a0-028b-4704-bc25-2a7f95a87d76" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 13:50:50.372498 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:50:50.372461    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d7d905-predictor-6778c84d57-vlqfx" podUID="e0f562a0-028b-4704-bc25-2a7f95a87d76" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 13:51:00.373437 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:00.373402    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-d7d905-predictor-6778c84d57-vlqfx"
Jul 28 13:51:22.992431 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:22.992396    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-d7d905-predictor-8696dc745d-tppcx"]
Jul 28 13:51:22.995832 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:22.995816    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-d7d905-predictor-8696dc745d-tppcx"
Jul 28 13:51:22.998611 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:22.998575    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-d7d905\""
Jul 28 13:51:22.998762 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:22.998745    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-d7d905-kube-rbac-proxy-sar-config\""
Jul 28 13:51:22.998851 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:22.998833    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-d7d905-predictor-serving-cert\""
Jul 28 13:51:22.999637 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:22.999620    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-d7d905-dockercfg-4l2sd\""
Jul 28 13:51:23.011606 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:23.011580    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-d7d905-predictor-8696dc745d-tppcx"]
Jul 28 13:51:23.076761 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:23.076731    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b2390ee-a032-4af3-81fe-b85b74bd6bc8-proxy-tls\") pod \"isvc-secondary-d7d905-predictor-8696dc745d-tppcx\" (UID: \"3b2390ee-a032-4af3-81fe-b85b74bd6bc8\") " pod="kserve-ci-e2e-test/isvc-secondary-d7d905-predictor-8696dc745d-tppcx"
Jul 28 13:51:23.076917 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:23.076776    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwwgn\" (UniqueName: \"kubernetes.io/projected/3b2390ee-a032-4af3-81fe-b85b74bd6bc8-kube-api-access-wwwgn\") pod \"isvc-secondary-d7d905-predictor-8696dc745d-tppcx\" (UID: \"3b2390ee-a032-4af3-81fe-b85b74bd6bc8\") " pod="kserve-ci-e2e-test/isvc-secondary-d7d905-predictor-8696dc745d-tppcx"
Jul 28 13:51:23.076917 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:23.076801    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b2390ee-a032-4af3-81fe-b85b74bd6bc8-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-d7d905-predictor-8696dc745d-tppcx\" (UID: \"3b2390ee-a032-4af3-81fe-b85b74bd6bc8\") " pod="kserve-ci-e2e-test/isvc-secondary-d7d905-predictor-8696dc745d-tppcx"
Jul 28 13:51:23.076917 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:23.076843    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b2390ee-a032-4af3-81fe-b85b74bd6bc8-kserve-provision-location\") pod \"isvc-secondary-d7d905-predictor-8696dc745d-tppcx\" (UID: \"3b2390ee-a032-4af3-81fe-b85b74bd6bc8\") " pod="kserve-ci-e2e-test/isvc-secondary-d7d905-predictor-8696dc745d-tppcx"
Jul 28 13:51:23.076917 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:23.076874    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b2390ee-a032-4af3-81fe-b85b74bd6bc8-cabundle-cert\") pod \"isvc-secondary-d7d905-predictor-8696dc745d-tppcx\" (UID: \"3b2390ee-a032-4af3-81fe-b85b74bd6bc8\") " pod="kserve-ci-e2e-test/isvc-secondary-d7d905-predictor-8696dc745d-tppcx"
Jul 28 13:51:23.177287 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:23.177251    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwwgn\" (UniqueName: \"kubernetes.io/projected/3b2390ee-a032-4af3-81fe-b85b74bd6bc8-kube-api-access-wwwgn\") pod \"isvc-secondary-d7d905-predictor-8696dc745d-tppcx\" (UID: \"3b2390ee-a032-4af3-81fe-b85b74bd6bc8\") " pod="kserve-ci-e2e-test/isvc-secondary-d7d905-predictor-8696dc745d-tppcx"
Jul 28 13:51:23.177488 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:23.177330    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b2390ee-a032-4af3-81fe-b85b74bd6bc8-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-d7d905-predictor-8696dc745d-tppcx\" (UID: \"3b2390ee-a032-4af3-81fe-b85b74bd6bc8\") " pod="kserve-ci-e2e-test/isvc-secondary-d7d905-predictor-8696dc745d-tppcx"
Jul 28 13:51:23.177488 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:23.177359    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b2390ee-a032-4af3-81fe-b85b74bd6bc8-kserve-provision-location\") pod \"isvc-secondary-d7d905-predictor-8696dc745d-tppcx\" (UID: \"3b2390ee-a032-4af3-81fe-b85b74bd6bc8\") " pod="kserve-ci-e2e-test/isvc-secondary-d7d905-predictor-8696dc745d-tppcx"
Jul 28 13:51:23.177488 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:23.177409    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b2390ee-a032-4af3-81fe-b85b74bd6bc8-cabundle-cert\") pod \"isvc-secondary-d7d905-predictor-8696dc745d-tppcx\" (UID: \"3b2390ee-a032-4af3-81fe-b85b74bd6bc8\") " pod="kserve-ci-e2e-test/isvc-secondary-d7d905-predictor-8696dc745d-tppcx"
Jul 28 13:51:23.177488 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:23.177459    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b2390ee-a032-4af3-81fe-b85b74bd6bc8-proxy-tls\") pod \"isvc-secondary-d7d905-predictor-8696dc745d-tppcx\" (UID: \"3b2390ee-a032-4af3-81fe-b85b74bd6bc8\") " pod="kserve-ci-e2e-test/isvc-secondary-d7d905-predictor-8696dc745d-tppcx"
Jul 28 13:51:23.177828 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:23.177806    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b2390ee-a032-4af3-81fe-b85b74bd6bc8-kserve-provision-location\") pod \"isvc-secondary-d7d905-predictor-8696dc745d-tppcx\" (UID: \"3b2390ee-a032-4af3-81fe-b85b74bd6bc8\") " pod="kserve-ci-e2e-test/isvc-secondary-d7d905-predictor-8696dc745d-tppcx"
Jul 28 13:51:23.177984 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:23.177963    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b2390ee-a032-4af3-81fe-b85b74bd6bc8-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-d7d905-predictor-8696dc745d-tppcx\" (UID: \"3b2390ee-a032-4af3-81fe-b85b74bd6bc8\") " pod="kserve-ci-e2e-test/isvc-secondary-d7d905-predictor-8696dc745d-tppcx"
Jul 28 13:51:23.178031 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:23.177987    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b2390ee-a032-4af3-81fe-b85b74bd6bc8-cabundle-cert\") pod \"isvc-secondary-d7d905-predictor-8696dc745d-tppcx\" (UID: \"3b2390ee-a032-4af3-81fe-b85b74bd6bc8\") " pod="kserve-ci-e2e-test/isvc-secondary-d7d905-predictor-8696dc745d-tppcx"
Jul 28 13:51:23.180018 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:23.179995    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b2390ee-a032-4af3-81fe-b85b74bd6bc8-proxy-tls\") pod \"isvc-secondary-d7d905-predictor-8696dc745d-tppcx\" (UID: \"3b2390ee-a032-4af3-81fe-b85b74bd6bc8\") " pod="kserve-ci-e2e-test/isvc-secondary-d7d905-predictor-8696dc745d-tppcx"
Jul 28 13:51:23.187583 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:23.187561    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwwgn\" (UniqueName: \"kubernetes.io/projected/3b2390ee-a032-4af3-81fe-b85b74bd6bc8-kube-api-access-wwwgn\") pod \"isvc-secondary-d7d905-predictor-8696dc745d-tppcx\" (UID: \"3b2390ee-a032-4af3-81fe-b85b74bd6bc8\") " pod="kserve-ci-e2e-test/isvc-secondary-d7d905-predictor-8696dc745d-tppcx"
Jul 28 13:51:23.306819 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:23.306721    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-d7d905-predictor-8696dc745d-tppcx"
Jul 28 13:51:23.444940 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:23.444909    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-d7d905-predictor-8696dc745d-tppcx"]
Jul 28 13:51:23.446338 ip-10-0-129-177 kubenswrapper[2554]: W0728 13:51:23.446309    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b2390ee_a032_4af3_81fe_b85b74bd6bc8.slice/crio-4023fbb719e34ab768aefb674c636edb94fc71f3b930ffc4a9e6389a902aee7b WatchSource:0}: Error finding container 4023fbb719e34ab768aefb674c636edb94fc71f3b930ffc4a9e6389a902aee7b: Status 404 returned error can't find the container with id 4023fbb719e34ab768aefb674c636edb94fc71f3b930ffc4a9e6389a902aee7b
Jul 28 13:51:23.786696 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:23.786666    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-d7d905-predictor-8696dc745d-tppcx" event={"ID":"3b2390ee-a032-4af3-81fe-b85b74bd6bc8","Type":"ContainerStarted","Data":"3a325dc0f3540d61ee658f454d4a349d5e40bd52fe702041064a5df9d8e21832"}
Jul 28 13:51:23.786864 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:23.786724    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-d7d905-predictor-8696dc745d-tppcx" event={"ID":"3b2390ee-a032-4af3-81fe-b85b74bd6bc8","Type":"ContainerStarted","Data":"4023fbb719e34ab768aefb674c636edb94fc71f3b930ffc4a9e6389a902aee7b"}
Jul 28 13:51:26.802479 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:26.802450    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d7d905-predictor-8696dc745d-tppcx_3b2390ee-a032-4af3-81fe-b85b74bd6bc8/storage-initializer/0.log"
Jul 28 13:51:26.802849 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:26.802489    2554 generic.go:356] "Generic (PLEG): container finished" podID="3b2390ee-a032-4af3-81fe-b85b74bd6bc8" containerID="3a325dc0f3540d61ee658f454d4a349d5e40bd52fe702041064a5df9d8e21832" exitCode=1
Jul 28 13:51:26.802849 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:26.802518    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-d7d905-predictor-8696dc745d-tppcx" event={"ID":"3b2390ee-a032-4af3-81fe-b85b74bd6bc8","Type":"ContainerDied","Data":"3a325dc0f3540d61ee658f454d4a349d5e40bd52fe702041064a5df9d8e21832"}
Jul 28 13:51:27.808393 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:27.808368    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d7d905-predictor-8696dc745d-tppcx_3b2390ee-a032-4af3-81fe-b85b74bd6bc8/storage-initializer/0.log"
Jul 28 13:51:27.808777 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:27.808454    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-d7d905-predictor-8696dc745d-tppcx" event={"ID":"3b2390ee-a032-4af3-81fe-b85b74bd6bc8","Type":"ContainerStarted","Data":"6b81d8ff9646c53915675c7e56cbfb7aa8525f8eb66ae07ba051d04d688650c1"}
Jul 28 13:51:30.822091 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:30.822065    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d7d905-predictor-8696dc745d-tppcx_3b2390ee-a032-4af3-81fe-b85b74bd6bc8/storage-initializer/1.log"
Jul 28 13:51:30.822542 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:30.822434    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d7d905-predictor-8696dc745d-tppcx_3b2390ee-a032-4af3-81fe-b85b74bd6bc8/storage-initializer/0.log"
Jul 28 13:51:30.822542 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:30.822468    2554 generic.go:356] "Generic (PLEG): container finished" podID="3b2390ee-a032-4af3-81fe-b85b74bd6bc8" containerID="6b81d8ff9646c53915675c7e56cbfb7aa8525f8eb66ae07ba051d04d688650c1" exitCode=1
Jul 28 13:51:30.822542 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:30.822533    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-d7d905-predictor-8696dc745d-tppcx" event={"ID":"3b2390ee-a032-4af3-81fe-b85b74bd6bc8","Type":"ContainerDied","Data":"6b81d8ff9646c53915675c7e56cbfb7aa8525f8eb66ae07ba051d04d688650c1"}
Jul 28 13:51:30.822656 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:30.822593    2554 scope.go:117] "RemoveContainer" containerID="3a325dc0f3540d61ee658f454d4a349d5e40bd52fe702041064a5df9d8e21832"
Jul 28 13:51:30.822951 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:30.822931    2554 scope.go:117] "RemoveContainer" containerID="3a325dc0f3540d61ee658f454d4a349d5e40bd52fe702041064a5df9d8e21832"
Jul 28 13:51:30.833529 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:51:30.833503    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-d7d905-predictor-8696dc745d-tppcx_kserve-ci-e2e-test(3b2390ee-a032-4af3-81fe-b85b74bd6bc8)\"" pod="kserve-ci-e2e-test/isvc-secondary-d7d905-predictor-8696dc745d-tppcx" podUID="3b2390ee-a032-4af3-81fe-b85b74bd6bc8"
Jul 28 13:51:30.833783 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:51:30.833758    2554 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-d7d905-predictor-8696dc745d-tppcx_kserve-ci-e2e-test_3b2390ee-a032-4af3-81fe-b85b74bd6bc8_0 in pod sandbox 4023fbb719e34ab768aefb674c636edb94fc71f3b930ffc4a9e6389a902aee7b from index: no such id: '3a325dc0f3540d61ee658f454d4a349d5e40bd52fe702041064a5df9d8e21832'" containerID="3a325dc0f3540d61ee658f454d4a349d5e40bd52fe702041064a5df9d8e21832"
Jul 28 13:51:30.833836 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:30.833792    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a325dc0f3540d61ee658f454d4a349d5e40bd52fe702041064a5df9d8e21832"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-d7d905-predictor-8696dc745d-tppcx_kserve-ci-e2e-test_3b2390ee-a032-4af3-81fe-b85b74bd6bc8_0 in pod sandbox 4023fbb719e34ab768aefb674c636edb94fc71f3b930ffc4a9e6389a902aee7b from index: no such id: '3a325dc0f3540d61ee658f454d4a349d5e40bd52fe702041064a5df9d8e21832'"
Jul 28 13:51:31.834886 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:31.834855    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d7d905-predictor-8696dc745d-tppcx_3b2390ee-a032-4af3-81fe-b85b74bd6bc8/storage-initializer/1.log"
Jul 28 13:51:38.877286 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:38.877241    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d7d905-predictor-6778c84d57-vlqfx"]
Jul 28 13:51:38.877670 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:38.877545    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-d7d905-predictor-6778c84d57-vlqfx" podUID="e0f562a0-028b-4704-bc25-2a7f95a87d76" containerName="kserve-container" containerID="cri-o://517f0c02891ef394624d0857e51916821d5742c7d29d8f02b4ea1ce3c389277c" gracePeriod=30
Jul 28 13:51:38.877670 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:38.877554    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-d7d905-predictor-6778c84d57-vlqfx" podUID="e0f562a0-028b-4704-bc25-2a7f95a87d76" containerName="kube-rbac-proxy" containerID="cri-o://60ed0125fae585accf1e8d943f78f4a08eb03654205596c635a3fa2487ad9e71" gracePeriod=30
Jul 28 13:51:38.954162 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:38.954124    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-d7d905-predictor-8696dc745d-tppcx"]
Jul 28 13:51:39.098882 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:39.098857    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d7d905-predictor-8696dc745d-tppcx_3b2390ee-a032-4af3-81fe-b85b74bd6bc8/storage-initializer/1.log"
Jul 28 13:51:39.101097 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:39.101083    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-d7d905-predictor-8696dc745d-tppcx"
Jul 28 13:51:39.192635 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:39.192605    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b2390ee-a032-4af3-81fe-b85b74bd6bc8-kube-rbac-proxy-sar-config\") pod \"3b2390ee-a032-4af3-81fe-b85b74bd6bc8\" (UID: \"3b2390ee-a032-4af3-81fe-b85b74bd6bc8\") "
Jul 28 13:51:39.192819 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:39.192660    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b2390ee-a032-4af3-81fe-b85b74bd6bc8-cabundle-cert\") pod \"3b2390ee-a032-4af3-81fe-b85b74bd6bc8\" (UID: \"3b2390ee-a032-4af3-81fe-b85b74bd6bc8\") "
Jul 28 13:51:39.192819 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:39.192697    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b2390ee-a032-4af3-81fe-b85b74bd6bc8-kserve-provision-location\") pod \"3b2390ee-a032-4af3-81fe-b85b74bd6bc8\" (UID: \"3b2390ee-a032-4af3-81fe-b85b74bd6bc8\") "
Jul 28 13:51:39.192819 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:39.192722    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b2390ee-a032-4af3-81fe-b85b74bd6bc8-proxy-tls\") pod \"3b2390ee-a032-4af3-81fe-b85b74bd6bc8\" (UID: \"3b2390ee-a032-4af3-81fe-b85b74bd6bc8\") "
Jul 28 13:51:39.192819 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:39.192790    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwwgn\" (UniqueName: \"kubernetes.io/projected/3b2390ee-a032-4af3-81fe-b85b74bd6bc8-kube-api-access-wwwgn\") pod \"3b2390ee-a032-4af3-81fe-b85b74bd6bc8\" (UID: \"3b2390ee-a032-4af3-81fe-b85b74bd6bc8\") "
Jul 28 13:51:39.193024 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:39.192955    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b2390ee-a032-4af3-81fe-b85b74bd6bc8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3b2390ee-a032-4af3-81fe-b85b74bd6bc8" (UID: "3b2390ee-a032-4af3-81fe-b85b74bd6bc8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:51:39.193024 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:39.192974    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b2390ee-a032-4af3-81fe-b85b74bd6bc8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3b2390ee-a032-4af3-81fe-b85b74bd6bc8" (UID: "3b2390ee-a032-4af3-81fe-b85b74bd6bc8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:51:39.193128 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:39.193055    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b2390ee-a032-4af3-81fe-b85b74bd6bc8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3b2390ee-a032-4af3-81fe-b85b74bd6bc8" (UID: "3b2390ee-a032-4af3-81fe-b85b74bd6bc8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:51:39.193182 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:39.193123    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b2390ee-a032-4af3-81fe-b85b74bd6bc8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:51:39.193182 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:39.193144    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b2390ee-a032-4af3-81fe-b85b74bd6bc8-kserve-provision-location\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:51:39.195046 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:39.195023    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b2390ee-a032-4af3-81fe-b85b74bd6bc8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3b2390ee-a032-4af3-81fe-b85b74bd6bc8" (UID: "3b2390ee-a032-4af3-81fe-b85b74bd6bc8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:51:39.195203 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:39.195044    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b2390ee-a032-4af3-81fe-b85b74bd6bc8-kube-api-access-wwwgn" (OuterVolumeSpecName: "kube-api-access-wwwgn") pod "3b2390ee-a032-4af3-81fe-b85b74bd6bc8" (UID: "3b2390ee-a032-4af3-81fe-b85b74bd6bc8"). InnerVolumeSpecName "kube-api-access-wwwgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:51:39.293867 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:39.293835    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b2390ee-a032-4af3-81fe-b85b74bd6bc8-cabundle-cert\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:51:39.293867 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:39.293859    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b2390ee-a032-4af3-81fe-b85b74bd6bc8-proxy-tls\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:51:39.293867 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:39.293868    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwwgn\" (UniqueName: \"kubernetes.io/projected/3b2390ee-a032-4af3-81fe-b85b74bd6bc8-kube-api-access-wwwgn\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:51:39.870545 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:39.870514    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d7d905-predictor-8696dc745d-tppcx_3b2390ee-a032-4af3-81fe-b85b74bd6bc8/storage-initializer/1.log"
Jul 28 13:51:39.870712 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:39.870617    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-d7d905-predictor-8696dc745d-tppcx"
Jul 28 13:51:39.870712 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:39.870641    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-d7d905-predictor-8696dc745d-tppcx" event={"ID":"3b2390ee-a032-4af3-81fe-b85b74bd6bc8","Type":"ContainerDied","Data":"4023fbb719e34ab768aefb674c636edb94fc71f3b930ffc4a9e6389a902aee7b"}
Jul 28 13:51:39.870812 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:39.870742    2554 scope.go:117] "RemoveContainer" containerID="6b81d8ff9646c53915675c7e56cbfb7aa8525f8eb66ae07ba051d04d688650c1"
Jul 28 13:51:39.873143 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:39.873105    2554 generic.go:356] "Generic (PLEG): container finished" podID="e0f562a0-028b-4704-bc25-2a7f95a87d76" containerID="60ed0125fae585accf1e8d943f78f4a08eb03654205596c635a3fa2487ad9e71" exitCode=2
Jul 28 13:51:39.873143 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:39.873140    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d7d905-predictor-6778c84d57-vlqfx" event={"ID":"e0f562a0-028b-4704-bc25-2a7f95a87d76","Type":"ContainerDied","Data":"60ed0125fae585accf1e8d943f78f4a08eb03654205596c635a3fa2487ad9e71"}
Jul 28 13:51:39.926240 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:39.926213    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-d7d905-predictor-8696dc745d-tppcx"]
Jul 28 13:51:39.932353 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:39.932332    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-d7d905-predictor-8696dc745d-tppcx"]
Jul 28 13:51:40.367258 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:40.367215    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d7d905-predictor-6778c84d57-vlqfx" podUID="e0f562a0-028b-4704-bc25-2a7f95a87d76" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Jul 28 13:51:40.372614 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:40.372594    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d7d905-predictor-6778c84d57-vlqfx" podUID="e0f562a0-028b-4704-bc25-2a7f95a87d76" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 28 13:51:41.802214 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:41.802179    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b2390ee-a032-4af3-81fe-b85b74bd6bc8" path="/var/lib/kubelet/pods/3b2390ee-a032-4af3-81fe-b85b74bd6bc8/volumes"
Jul 28 13:51:43.321745 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:43.321723    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-d7d905-predictor-6778c84d57-vlqfx"
Jul 28 13:51:43.427622 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:43.427552    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rkxmx\" (UniqueName: \"kubernetes.io/projected/e0f562a0-028b-4704-bc25-2a7f95a87d76-kube-api-access-rkxmx\") pod \"e0f562a0-028b-4704-bc25-2a7f95a87d76\" (UID: \"e0f562a0-028b-4704-bc25-2a7f95a87d76\") "
Jul 28 13:51:43.427622 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:43.427587    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0f562a0-028b-4704-bc25-2a7f95a87d76-kserve-provision-location\") pod \"e0f562a0-028b-4704-bc25-2a7f95a87d76\" (UID: \"e0f562a0-028b-4704-bc25-2a7f95a87d76\") "
Jul 28 13:51:43.427795 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:43.427636    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0f562a0-028b-4704-bc25-2a7f95a87d76-cabundle-cert\") pod \"e0f562a0-028b-4704-bc25-2a7f95a87d76\" (UID: \"e0f562a0-028b-4704-bc25-2a7f95a87d76\") "
Jul 28 13:51:43.427795 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:43.427691    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0f562a0-028b-4704-bc25-2a7f95a87d76-kube-rbac-proxy-sar-config\") pod \"e0f562a0-028b-4704-bc25-2a7f95a87d76\" (UID: \"e0f562a0-028b-4704-bc25-2a7f95a87d76\") "
Jul 28 13:51:43.427795 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:43.427734    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0f562a0-028b-4704-bc25-2a7f95a87d76-proxy-tls\") pod \"e0f562a0-028b-4704-bc25-2a7f95a87d76\" (UID: \"e0f562a0-028b-4704-bc25-2a7f95a87d76\") "
Jul 28 13:51:43.427934 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:43.427908    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0f562a0-028b-4704-bc25-2a7f95a87d76-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e0f562a0-028b-4704-bc25-2a7f95a87d76" (UID: "e0f562a0-028b-4704-bc25-2a7f95a87d76"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:51:43.428060 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:43.428034    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0f562a0-028b-4704-bc25-2a7f95a87d76-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e0f562a0-028b-4704-bc25-2a7f95a87d76" (UID: "e0f562a0-028b-4704-bc25-2a7f95a87d76"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:51:43.428125 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:43.428038    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0f562a0-028b-4704-bc25-2a7f95a87d76-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e0f562a0-028b-4704-bc25-2a7f95a87d76" (UID: "e0f562a0-028b-4704-bc25-2a7f95a87d76"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:51:43.429831 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:43.429810    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0f562a0-028b-4704-bc25-2a7f95a87d76-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e0f562a0-028b-4704-bc25-2a7f95a87d76" (UID: "e0f562a0-028b-4704-bc25-2a7f95a87d76"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:51:43.429934 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:43.429917    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0f562a0-028b-4704-bc25-2a7f95a87d76-kube-api-access-rkxmx" (OuterVolumeSpecName: "kube-api-access-rkxmx") pod "e0f562a0-028b-4704-bc25-2a7f95a87d76" (UID: "e0f562a0-028b-4704-bc25-2a7f95a87d76"). InnerVolumeSpecName "kube-api-access-rkxmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:51:43.528477 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:43.528443    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0f562a0-028b-4704-bc25-2a7f95a87d76-kserve-provision-location\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:51:43.528477 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:43.528476    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0f562a0-028b-4704-bc25-2a7f95a87d76-cabundle-cert\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:51:43.528477 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:43.528486    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0f562a0-028b-4704-bc25-2a7f95a87d76-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:51:43.528648 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:43.528494    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0f562a0-028b-4704-bc25-2a7f95a87d76-proxy-tls\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:51:43.528648 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:43.528502    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkxmx\" (UniqueName: \"kubernetes.io/projected/e0f562a0-028b-4704-bc25-2a7f95a87d76-kube-api-access-rkxmx\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:51:43.891141 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:43.891107    2554 generic.go:356] "Generic (PLEG): container finished" podID="e0f562a0-028b-4704-bc25-2a7f95a87d76" containerID="517f0c02891ef394624d0857e51916821d5742c7d29d8f02b4ea1ce3c389277c" exitCode=0
Jul 28 13:51:43.891370 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:43.891168    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-d7d905-predictor-6778c84d57-vlqfx"
Jul 28 13:51:43.891370 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:43.891192    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d7d905-predictor-6778c84d57-vlqfx" event={"ID":"e0f562a0-028b-4704-bc25-2a7f95a87d76","Type":"ContainerDied","Data":"517f0c02891ef394624d0857e51916821d5742c7d29d8f02b4ea1ce3c389277c"}
Jul 28 13:51:43.891370 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:43.891257    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d7d905-predictor-6778c84d57-vlqfx" event={"ID":"e0f562a0-028b-4704-bc25-2a7f95a87d76","Type":"ContainerDied","Data":"1ce9e3c21f34bd700c0cf190187905145bfc6e36900663c05cca70f32768f7f7"}
Jul 28 13:51:43.891370 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:43.891295    2554 scope.go:117] "RemoveContainer" containerID="60ed0125fae585accf1e8d943f78f4a08eb03654205596c635a3fa2487ad9e71"
Jul 28 13:51:43.899623 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:43.899605    2554 scope.go:117] "RemoveContainer" containerID="517f0c02891ef394624d0857e51916821d5742c7d29d8f02b4ea1ce3c389277c"
Jul 28 13:51:43.909796 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:43.908668    2554 scope.go:117] "RemoveContainer" containerID="bf7d0b2504b676d792e8187c6618a5d19f02f59a9a1fca4b0c374d5e6eab5566"
Jul 28 13:51:43.911350 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:43.911333    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d7d905-predictor-6778c84d57-vlqfx"]
Jul 28 13:51:43.914800 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:43.914781    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d7d905-predictor-6778c84d57-vlqfx"]
Jul 28 13:51:43.917869 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:43.917854    2554 scope.go:117] "RemoveContainer" containerID="60ed0125fae585accf1e8d943f78f4a08eb03654205596c635a3fa2487ad9e71"
Jul 28 13:51:43.918113 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:51:43.918094    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60ed0125fae585accf1e8d943f78f4a08eb03654205596c635a3fa2487ad9e71\": container with ID starting with 60ed0125fae585accf1e8d943f78f4a08eb03654205596c635a3fa2487ad9e71 not found: ID does not exist" containerID="60ed0125fae585accf1e8d943f78f4a08eb03654205596c635a3fa2487ad9e71"
Jul 28 13:51:43.918180 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:43.918121    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60ed0125fae585accf1e8d943f78f4a08eb03654205596c635a3fa2487ad9e71"} err="failed to get container status \"60ed0125fae585accf1e8d943f78f4a08eb03654205596c635a3fa2487ad9e71\": rpc error: code = NotFound desc = could not find container \"60ed0125fae585accf1e8d943f78f4a08eb03654205596c635a3fa2487ad9e71\": container with ID starting with 60ed0125fae585accf1e8d943f78f4a08eb03654205596c635a3fa2487ad9e71 not found: ID does not exist"
Jul 28 13:51:43.918180 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:43.918140    2554 scope.go:117] "RemoveContainer" containerID="517f0c02891ef394624d0857e51916821d5742c7d29d8f02b4ea1ce3c389277c"
Jul 28 13:51:43.918410 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:51:43.918393    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"517f0c02891ef394624d0857e51916821d5742c7d29d8f02b4ea1ce3c389277c\": container with ID starting with 517f0c02891ef394624d0857e51916821d5742c7d29d8f02b4ea1ce3c389277c not found: ID does not exist" containerID="517f0c02891ef394624d0857e51916821d5742c7d29d8f02b4ea1ce3c389277c"
Jul 28 13:51:43.918463 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:43.918417    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"517f0c02891ef394624d0857e51916821d5742c7d29d8f02b4ea1ce3c389277c"} err="failed to get container status \"517f0c02891ef394624d0857e51916821d5742c7d29d8f02b4ea1ce3c389277c\": rpc error: code = NotFound desc = could not find container \"517f0c02891ef394624d0857e51916821d5742c7d29d8f02b4ea1ce3c389277c\": container with ID starting with 517f0c02891ef394624d0857e51916821d5742c7d29d8f02b4ea1ce3c389277c not found: ID does not exist"
Jul 28 13:51:43.918463 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:43.918432    2554 scope.go:117] "RemoveContainer" containerID="bf7d0b2504b676d792e8187c6618a5d19f02f59a9a1fca4b0c374d5e6eab5566"
Jul 28 13:51:43.918657 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:51:43.918643    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf7d0b2504b676d792e8187c6618a5d19f02f59a9a1fca4b0c374d5e6eab5566\": container with ID starting with bf7d0b2504b676d792e8187c6618a5d19f02f59a9a1fca4b0c374d5e6eab5566 not found: ID does not exist" containerID="bf7d0b2504b676d792e8187c6618a5d19f02f59a9a1fca4b0c374d5e6eab5566"
Jul 28 13:51:43.918698 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:43.918659    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf7d0b2504b676d792e8187c6618a5d19f02f59a9a1fca4b0c374d5e6eab5566"} err="failed to get container status \"bf7d0b2504b676d792e8187c6618a5d19f02f59a9a1fca4b0c374d5e6eab5566\": rpc error: code = NotFound desc = could not find container \"bf7d0b2504b676d792e8187c6618a5d19f02f59a9a1fca4b0c374d5e6eab5566\": container with ID starting with bf7d0b2504b676d792e8187c6618a5d19f02f59a9a1fca4b0c374d5e6eab5566 not found: ID does not exist"
Jul 28 13:51:44.130153 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:44.130072    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-618844-predictor-687f87cb4b-9vzgb"]
Jul 28 13:51:44.130406 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:44.130393    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0f562a0-028b-4704-bc25-2a7f95a87d76" containerName="kube-rbac-proxy"
Jul 28 13:51:44.130454 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:44.130408    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0f562a0-028b-4704-bc25-2a7f95a87d76" containerName="kube-rbac-proxy"
Jul 28 13:51:44.130454 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:44.130426    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0f562a0-028b-4704-bc25-2a7f95a87d76" containerName="kserve-container"
Jul 28 13:51:44.130454 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:44.130431    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0f562a0-028b-4704-bc25-2a7f95a87d76" containerName="kserve-container"
Jul 28 13:51:44.130454 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:44.130438    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b2390ee-a032-4af3-81fe-b85b74bd6bc8" containerName="storage-initializer"
Jul 28 13:51:44.130454 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:44.130444    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b2390ee-a032-4af3-81fe-b85b74bd6bc8" containerName="storage-initializer"
Jul 28 13:51:44.130454 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:44.130450    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0f562a0-028b-4704-bc25-2a7f95a87d76" containerName="storage-initializer"
Jul 28 13:51:44.130454 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:44.130455    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0f562a0-028b-4704-bc25-2a7f95a87d76" containerName="storage-initializer"
Jul 28 13:51:44.130666 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:44.130461    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b2390ee-a032-4af3-81fe-b85b74bd6bc8" containerName="storage-initializer"
Jul 28 13:51:44.130666 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:44.130466    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b2390ee-a032-4af3-81fe-b85b74bd6bc8" containerName="storage-initializer"
Jul 28 13:51:44.130666 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:44.130515    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0f562a0-028b-4704-bc25-2a7f95a87d76" containerName="kserve-container"
Jul 28 13:51:44.130666 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:44.130525    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b2390ee-a032-4af3-81fe-b85b74bd6bc8" containerName="storage-initializer"
Jul 28 13:51:44.130666 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:44.130533    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b2390ee-a032-4af3-81fe-b85b74bd6bc8" containerName="storage-initializer"
Jul 28 13:51:44.130666 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:44.130538    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0f562a0-028b-4704-bc25-2a7f95a87d76" containerName="kube-rbac-proxy"
Jul 28 13:51:44.134799 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:44.134783    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-618844-predictor-687f87cb4b-9vzgb"
Jul 28 13:51:44.139422 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:44.139387    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-618844\""
Jul 28 13:51:44.139732 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:44.139706    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-618844-predictor-serving-cert\""
Jul 28 13:51:44.140236 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:44.140143    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:51:44.140413 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:44.140392    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-618844-kube-rbac-proxy-sar-config\""
Jul 28 13:51:44.140602 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:44.140588    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:51:44.140788 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:44.140776    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:51:44.141055 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:44.141036    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-618844-dockercfg-9nxts\""
Jul 28 13:51:44.153719 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:44.153689    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-618844-predictor-687f87cb4b-9vzgb"]
Jul 28 13:51:44.234498 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:44.234448    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6688a862-c9f4-43a1-ba6e-bfe9248f8f96-kserve-provision-location\") pod \"isvc-init-fail-618844-predictor-687f87cb4b-9vzgb\" (UID: \"6688a862-c9f4-43a1-ba6e-bfe9248f8f96\") " pod="kserve-ci-e2e-test/isvc-init-fail-618844-predictor-687f87cb4b-9vzgb"
Jul 28 13:51:44.234661 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:44.234505    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6688a862-c9f4-43a1-ba6e-bfe9248f8f96-cabundle-cert\") pod \"isvc-init-fail-618844-predictor-687f87cb4b-9vzgb\" (UID: \"6688a862-c9f4-43a1-ba6e-bfe9248f8f96\") " pod="kserve-ci-e2e-test/isvc-init-fail-618844-predictor-687f87cb4b-9vzgb"
Jul 28 13:51:44.234661 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:44.234569    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6688a862-c9f4-43a1-ba6e-bfe9248f8f96-proxy-tls\") pod \"isvc-init-fail-618844-predictor-687f87cb4b-9vzgb\" (UID: \"6688a862-c9f4-43a1-ba6e-bfe9248f8f96\") " pod="kserve-ci-e2e-test/isvc-init-fail-618844-predictor-687f87cb4b-9vzgb"
Jul 28 13:51:44.234785 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:44.234669    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6688a862-c9f4-43a1-ba6e-bfe9248f8f96-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-618844-predictor-687f87cb4b-9vzgb\" (UID: \"6688a862-c9f4-43a1-ba6e-bfe9248f8f96\") " pod="kserve-ci-e2e-test/isvc-init-fail-618844-predictor-687f87cb4b-9vzgb"
Jul 28 13:51:44.234785 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:44.234755    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc8qw\" (UniqueName: \"kubernetes.io/projected/6688a862-c9f4-43a1-ba6e-bfe9248f8f96-kube-api-access-jc8qw\") pod \"isvc-init-fail-618844-predictor-687f87cb4b-9vzgb\" (UID: \"6688a862-c9f4-43a1-ba6e-bfe9248f8f96\") " pod="kserve-ci-e2e-test/isvc-init-fail-618844-predictor-687f87cb4b-9vzgb"
Jul 28 13:51:44.335840 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:44.335802    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc8qw\" (UniqueName: \"kubernetes.io/projected/6688a862-c9f4-43a1-ba6e-bfe9248f8f96-kube-api-access-jc8qw\") pod \"isvc-init-fail-618844-predictor-687f87cb4b-9vzgb\" (UID: \"6688a862-c9f4-43a1-ba6e-bfe9248f8f96\") " pod="kserve-ci-e2e-test/isvc-init-fail-618844-predictor-687f87cb4b-9vzgb"
Jul 28 13:51:44.336343 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:44.335862    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6688a862-c9f4-43a1-ba6e-bfe9248f8f96-kserve-provision-location\") pod \"isvc-init-fail-618844-predictor-687f87cb4b-9vzgb\" (UID: \"6688a862-c9f4-43a1-ba6e-bfe9248f8f96\") " pod="kserve-ci-e2e-test/isvc-init-fail-618844-predictor-687f87cb4b-9vzgb"
Jul 28 13:51:44.336343 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:44.335887    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6688a862-c9f4-43a1-ba6e-bfe9248f8f96-cabundle-cert\") pod \"isvc-init-fail-618844-predictor-687f87cb4b-9vzgb\" (UID: \"6688a862-c9f4-43a1-ba6e-bfe9248f8f96\") " pod="kserve-ci-e2e-test/isvc-init-fail-618844-predictor-687f87cb4b-9vzgb"
Jul 28 13:51:44.336343 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:44.335904    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6688a862-c9f4-43a1-ba6e-bfe9248f8f96-proxy-tls\") pod \"isvc-init-fail-618844-predictor-687f87cb4b-9vzgb\" (UID: \"6688a862-c9f4-43a1-ba6e-bfe9248f8f96\") " pod="kserve-ci-e2e-test/isvc-init-fail-618844-predictor-687f87cb4b-9vzgb"
Jul 28 13:51:44.336343 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:44.335940    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6688a862-c9f4-43a1-ba6e-bfe9248f8f96-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-618844-predictor-687f87cb4b-9vzgb\" (UID: \"6688a862-c9f4-43a1-ba6e-bfe9248f8f96\") " pod="kserve-ci-e2e-test/isvc-init-fail-618844-predictor-687f87cb4b-9vzgb"
Jul 28 13:51:44.336343 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:51:44.336090    2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-618844-predictor-serving-cert: secret "isvc-init-fail-618844-predictor-serving-cert" not found
Jul 28 13:51:44.336343 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:51:44.336173    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6688a862-c9f4-43a1-ba6e-bfe9248f8f96-proxy-tls podName:6688a862-c9f4-43a1-ba6e-bfe9248f8f96 nodeName:}" failed. No retries permitted until 2026-07-28 13:51:44.836150562 +0000 UTC m=+3501.622025650 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6688a862-c9f4-43a1-ba6e-bfe9248f8f96-proxy-tls") pod "isvc-init-fail-618844-predictor-687f87cb4b-9vzgb" (UID: "6688a862-c9f4-43a1-ba6e-bfe9248f8f96") : secret "isvc-init-fail-618844-predictor-serving-cert" not found
Jul 28 13:51:44.336598 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:44.336353    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6688a862-c9f4-43a1-ba6e-bfe9248f8f96-kserve-provision-location\") pod \"isvc-init-fail-618844-predictor-687f87cb4b-9vzgb\" (UID: \"6688a862-c9f4-43a1-ba6e-bfe9248f8f96\") " pod="kserve-ci-e2e-test/isvc-init-fail-618844-predictor-687f87cb4b-9vzgb"
Jul 28 13:51:44.336683 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:44.336666    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6688a862-c9f4-43a1-ba6e-bfe9248f8f96-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-618844-predictor-687f87cb4b-9vzgb\" (UID: \"6688a862-c9f4-43a1-ba6e-bfe9248f8f96\") " pod="kserve-ci-e2e-test/isvc-init-fail-618844-predictor-687f87cb4b-9vzgb"
Jul 28 13:51:44.336733 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:44.336712    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6688a862-c9f4-43a1-ba6e-bfe9248f8f96-cabundle-cert\") pod \"isvc-init-fail-618844-predictor-687f87cb4b-9vzgb\" (UID: \"6688a862-c9f4-43a1-ba6e-bfe9248f8f96\") " pod="kserve-ci-e2e-test/isvc-init-fail-618844-predictor-687f87cb4b-9vzgb"
Jul 28 13:51:44.347040 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:44.347015    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc8qw\" (UniqueName: \"kubernetes.io/projected/6688a862-c9f4-43a1-ba6e-bfe9248f8f96-kube-api-access-jc8qw\") pod \"isvc-init-fail-618844-predictor-687f87cb4b-9vzgb\" (UID: \"6688a862-c9f4-43a1-ba6e-bfe9248f8f96\") " pod="kserve-ci-e2e-test/isvc-init-fail-618844-predictor-687f87cb4b-9vzgb"
Jul 28 13:51:44.840575 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:44.840477    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6688a862-c9f4-43a1-ba6e-bfe9248f8f96-proxy-tls\") pod \"isvc-init-fail-618844-predictor-687f87cb4b-9vzgb\" (UID: \"6688a862-c9f4-43a1-ba6e-bfe9248f8f96\") " pod="kserve-ci-e2e-test/isvc-init-fail-618844-predictor-687f87cb4b-9vzgb"
Jul 28 13:51:44.843103 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:44.843066    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6688a862-c9f4-43a1-ba6e-bfe9248f8f96-proxy-tls\") pod \"isvc-init-fail-618844-predictor-687f87cb4b-9vzgb\" (UID: \"6688a862-c9f4-43a1-ba6e-bfe9248f8f96\") " pod="kserve-ci-e2e-test/isvc-init-fail-618844-predictor-687f87cb4b-9vzgb"
Jul 28 13:51:45.045086 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:45.045048    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-618844-predictor-687f87cb4b-9vzgb"
Jul 28 13:51:45.180077 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:45.180055    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-618844-predictor-687f87cb4b-9vzgb"]
Jul 28 13:51:45.182316 ip-10-0-129-177 kubenswrapper[2554]: W0728 13:51:45.182245    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6688a862_c9f4_43a1_ba6e_bfe9248f8f96.slice/crio-a5e12da71a9e762390a9baf0e7865305e9856ecc52b312161ca4014074836bc7 WatchSource:0}: Error finding container a5e12da71a9e762390a9baf0e7865305e9856ecc52b312161ca4014074836bc7: Status 404 returned error can't find the container with id a5e12da71a9e762390a9baf0e7865305e9856ecc52b312161ca4014074836bc7
Jul 28 13:51:45.802325 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:45.802263    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0f562a0-028b-4704-bc25-2a7f95a87d76" path="/var/lib/kubelet/pods/e0f562a0-028b-4704-bc25-2a7f95a87d76/volumes"
Jul 28 13:51:45.902934 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:45.902896    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-618844-predictor-687f87cb4b-9vzgb" event={"ID":"6688a862-c9f4-43a1-ba6e-bfe9248f8f96","Type":"ContainerStarted","Data":"6ac10c5ea4a720900214dee91a63499f0ec3f1824a7c1ddcb7c5ad34b72c6f8e"}
Jul 28 13:51:45.903190 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:45.902962    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-618844-predictor-687f87cb4b-9vzgb" event={"ID":"6688a862-c9f4-43a1-ba6e-bfe9248f8f96","Type":"ContainerStarted","Data":"a5e12da71a9e762390a9baf0e7865305e9856ecc52b312161ca4014074836bc7"}
Jul 28 13:51:48.916506 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:48.916478    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-618844-predictor-687f87cb4b-9vzgb_6688a862-c9f4-43a1-ba6e-bfe9248f8f96/storage-initializer/0.log"
Jul 28 13:51:48.916852 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:48.916518    2554 generic.go:356] "Generic (PLEG): container finished" podID="6688a862-c9f4-43a1-ba6e-bfe9248f8f96" containerID="6ac10c5ea4a720900214dee91a63499f0ec3f1824a7c1ddcb7c5ad34b72c6f8e" exitCode=1
Jul 28 13:51:48.916852 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:48.916579    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-618844-predictor-687f87cb4b-9vzgb" event={"ID":"6688a862-c9f4-43a1-ba6e-bfe9248f8f96","Type":"ContainerDied","Data":"6ac10c5ea4a720900214dee91a63499f0ec3f1824a7c1ddcb7c5ad34b72c6f8e"}
Jul 28 13:51:49.922357 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:49.922329    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-618844-predictor-687f87cb4b-9vzgb_6688a862-c9f4-43a1-ba6e-bfe9248f8f96/storage-initializer/0.log"
Jul 28 13:51:49.922731 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:49.922434    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-618844-predictor-687f87cb4b-9vzgb" event={"ID":"6688a862-c9f4-43a1-ba6e-bfe9248f8f96","Type":"ContainerStarted","Data":"b419786ca1f1a75816d0b4bd8fcab23dea85024a9ddf29484cf2dc98b73469fb"}
Jul 28 13:51:51.903438 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:51:51.903303    2554 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6688a862_c9f4_43a1_ba6e_bfe9248f8f96.slice/crio-b419786ca1f1a75816d0b4bd8fcab23dea85024a9ddf29484cf2dc98b73469fb.scope\": RecentStats: unable to find data in memory cache]"
Jul 28 13:51:51.903846 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:51:51.903263    2554 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6688a862_c9f4_43a1_ba6e_bfe9248f8f96.slice/crio-b419786ca1f1a75816d0b4bd8fcab23dea85024a9ddf29484cf2dc98b73469fb.scope\": RecentStats: unable to find data in memory cache]"
Jul 28 13:51:51.931953 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:51.931928    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-618844-predictor-687f87cb4b-9vzgb_6688a862-c9f4-43a1-ba6e-bfe9248f8f96/storage-initializer/1.log"
Jul 28 13:51:51.932331 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:51.932317    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-618844-predictor-687f87cb4b-9vzgb_6688a862-c9f4-43a1-ba6e-bfe9248f8f96/storage-initializer/0.log"
Jul 28 13:51:51.932389 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:51.932352    2554 generic.go:356] "Generic (PLEG): container finished" podID="6688a862-c9f4-43a1-ba6e-bfe9248f8f96" containerID="b419786ca1f1a75816d0b4bd8fcab23dea85024a9ddf29484cf2dc98b73469fb" exitCode=1
Jul 28 13:51:51.932430 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:51.932413    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-618844-predictor-687f87cb4b-9vzgb" event={"ID":"6688a862-c9f4-43a1-ba6e-bfe9248f8f96","Type":"ContainerDied","Data":"b419786ca1f1a75816d0b4bd8fcab23dea85024a9ddf29484cf2dc98b73469fb"}
Jul 28 13:51:51.932490 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:51.932477    2554 scope.go:117] "RemoveContainer" containerID="6ac10c5ea4a720900214dee91a63499f0ec3f1824a7c1ddcb7c5ad34b72c6f8e"
Jul 28 13:51:51.932831 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:51.932810    2554 scope.go:117] "RemoveContainer" containerID="6ac10c5ea4a720900214dee91a63499f0ec3f1824a7c1ddcb7c5ad34b72c6f8e"
Jul 28 13:51:51.942746 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:51:51.942720    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-init-fail-618844-predictor-687f87cb4b-9vzgb_kserve-ci-e2e-test(6688a862-c9f4-43a1-ba6e-bfe9248f8f96)\"" pod="kserve-ci-e2e-test/isvc-init-fail-618844-predictor-687f87cb4b-9vzgb" podUID="6688a862-c9f4-43a1-ba6e-bfe9248f8f96"
Jul 28 13:51:51.942946 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:51:51.942906    2554 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-init-fail-618844-predictor-687f87cb4b-9vzgb_kserve-ci-e2e-test_6688a862-c9f4-43a1-ba6e-bfe9248f8f96_0 in pod sandbox a5e12da71a9e762390a9baf0e7865305e9856ecc52b312161ca4014074836bc7 from index: no such id: '6ac10c5ea4a720900214dee91a63499f0ec3f1824a7c1ddcb7c5ad34b72c6f8e'" containerID="6ac10c5ea4a720900214dee91a63499f0ec3f1824a7c1ddcb7c5ad34b72c6f8e"
Jul 28 13:51:51.943029 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:51.942956    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ac10c5ea4a720900214dee91a63499f0ec3f1824a7c1ddcb7c5ad34b72c6f8e"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-init-fail-618844-predictor-687f87cb4b-9vzgb_kserve-ci-e2e-test_6688a862-c9f4-43a1-ba6e-bfe9248f8f96_0 in pod sandbox a5e12da71a9e762390a9baf0e7865305e9856ecc52b312161ca4014074836bc7 from index: no such id: '6ac10c5ea4a720900214dee91a63499f0ec3f1824a7c1ddcb7c5ad34b72c6f8e'"
Jul 28 13:51:52.937850 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:52.937818    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-618844-predictor-687f87cb4b-9vzgb_6688a862-c9f4-43a1-ba6e-bfe9248f8f96/storage-initializer/1.log"
Jul 28 13:51:54.132604 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:54.132569    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-618844-predictor-687f87cb4b-9vzgb"]
Jul 28 13:51:54.272346 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:54.272323    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-618844-predictor-687f87cb4b-9vzgb_6688a862-c9f4-43a1-ba6e-bfe9248f8f96/storage-initializer/1.log"
Jul 28 13:51:54.276113 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:54.276094    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-618844-predictor-687f87cb4b-9vzgb"
Jul 28 13:51:54.318632 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:54.318604    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6688a862-c9f4-43a1-ba6e-bfe9248f8f96-proxy-tls\") pod \"6688a862-c9f4-43a1-ba6e-bfe9248f8f96\" (UID: \"6688a862-c9f4-43a1-ba6e-bfe9248f8f96\") "
Jul 28 13:51:54.318766 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:54.318654    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6688a862-c9f4-43a1-ba6e-bfe9248f8f96-cabundle-cert\") pod \"6688a862-c9f4-43a1-ba6e-bfe9248f8f96\" (UID: \"6688a862-c9f4-43a1-ba6e-bfe9248f8f96\") "
Jul 28 13:51:54.318766 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:54.318697    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6688a862-c9f4-43a1-ba6e-bfe9248f8f96-kserve-provision-location\") pod \"6688a862-c9f4-43a1-ba6e-bfe9248f8f96\" (UID: \"6688a862-c9f4-43a1-ba6e-bfe9248f8f96\") "
Jul 28 13:51:54.318766 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:54.318725    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6688a862-c9f4-43a1-ba6e-bfe9248f8f96-kube-rbac-proxy-sar-config\") pod \"6688a862-c9f4-43a1-ba6e-bfe9248f8f96\" (UID: \"6688a862-c9f4-43a1-ba6e-bfe9248f8f96\") "
Jul 28 13:51:54.318766 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:54.318764    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jc8qw\" (UniqueName: \"kubernetes.io/projected/6688a862-c9f4-43a1-ba6e-bfe9248f8f96-kube-api-access-jc8qw\") pod \"6688a862-c9f4-43a1-ba6e-bfe9248f8f96\" (UID: \"6688a862-c9f4-43a1-ba6e-bfe9248f8f96\") "
Jul 28 13:51:54.319005 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:54.318979    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6688a862-c9f4-43a1-ba6e-bfe9248f8f96-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6688a862-c9f4-43a1-ba6e-bfe9248f8f96" (UID: "6688a862-c9f4-43a1-ba6e-bfe9248f8f96"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:51:54.319088 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:54.319068    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6688a862-c9f4-43a1-ba6e-bfe9248f8f96-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6688a862-c9f4-43a1-ba6e-bfe9248f8f96" (UID: "6688a862-c9f4-43a1-ba6e-bfe9248f8f96"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:51:54.319139 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:54.319065    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6688a862-c9f4-43a1-ba6e-bfe9248f8f96-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6688a862-c9f4-43a1-ba6e-bfe9248f8f96" (UID: "6688a862-c9f4-43a1-ba6e-bfe9248f8f96"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:51:54.320958 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:54.320938    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6688a862-c9f4-43a1-ba6e-bfe9248f8f96-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6688a862-c9f4-43a1-ba6e-bfe9248f8f96" (UID: "6688a862-c9f4-43a1-ba6e-bfe9248f8f96"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:51:54.320958 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:54.320944    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6688a862-c9f4-43a1-ba6e-bfe9248f8f96-kube-api-access-jc8qw" (OuterVolumeSpecName: "kube-api-access-jc8qw") pod "6688a862-c9f4-43a1-ba6e-bfe9248f8f96" (UID: "6688a862-c9f4-43a1-ba6e-bfe9248f8f96"). InnerVolumeSpecName "kube-api-access-jc8qw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:51:54.420100 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:54.420014    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6688a862-c9f4-43a1-ba6e-bfe9248f8f96-kserve-provision-location\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:51:54.420100 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:54.420053    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6688a862-c9f4-43a1-ba6e-bfe9248f8f96-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:51:54.420100 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:54.420067    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc8qw\" (UniqueName: \"kubernetes.io/projected/6688a862-c9f4-43a1-ba6e-bfe9248f8f96-kube-api-access-jc8qw\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:51:54.420100 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:54.420080    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6688a862-c9f4-43a1-ba6e-bfe9248f8f96-proxy-tls\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:51:54.420100 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:54.420092    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6688a862-c9f4-43a1-ba6e-bfe9248f8f96-cabundle-cert\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:51:54.947992 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:54.947966    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-618844-predictor-687f87cb4b-9vzgb_6688a862-c9f4-43a1-ba6e-bfe9248f8f96/storage-initializer/1.log"
Jul 28 13:51:54.948170 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:54.948050    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-618844-predictor-687f87cb4b-9vzgb" event={"ID":"6688a862-c9f4-43a1-ba6e-bfe9248f8f96","Type":"ContainerDied","Data":"a5e12da71a9e762390a9baf0e7865305e9856ecc52b312161ca4014074836bc7"}
Jul 28 13:51:54.948170 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:54.948071    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-618844-predictor-687f87cb4b-9vzgb"
Jul 28 13:51:54.948170 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:54.948108    2554 scope.go:117] "RemoveContainer" containerID="b419786ca1f1a75816d0b4bd8fcab23dea85024a9ddf29484cf2dc98b73469fb"
Jul 28 13:51:54.989488 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:54.989456    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-618844-predictor-687f87cb4b-9vzgb"]
Jul 28 13:51:54.995340 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:54.995308    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-618844-predictor-687f87cb4b-9vzgb"]
Jul 28 13:51:55.802126 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:55.802092    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6688a862-c9f4-43a1-ba6e-bfe9248f8f96" path="/var/lib/kubelet/pods/6688a862-c9f4-43a1-ba6e-bfe9248f8f96/volumes"
Jul 28 13:51:59.508366 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:59.508324    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-s2tjv"]
Jul 28 13:51:59.508852 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:59.508795    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6688a862-c9f4-43a1-ba6e-bfe9248f8f96" containerName="storage-initializer"
Jul 28 13:51:59.508852 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:59.508813    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="6688a862-c9f4-43a1-ba6e-bfe9248f8f96" containerName="storage-initializer"
Jul 28 13:51:59.508979 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:59.508912    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="6688a862-c9f4-43a1-ba6e-bfe9248f8f96" containerName="storage-initializer"
Jul 28 13:51:59.509038 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:59.509014    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6688a862-c9f4-43a1-ba6e-bfe9248f8f96" containerName="storage-initializer"
Jul 28 13:51:59.509038 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:59.509025    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="6688a862-c9f4-43a1-ba6e-bfe9248f8f96" containerName="storage-initializer"
Jul 28 13:51:59.509139 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:59.509121    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="6688a862-c9f4-43a1-ba6e-bfe9248f8f96" containerName="storage-initializer"
Jul 28 13:51:59.513751 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:59.513730    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-s2tjv"
Jul 28 13:51:59.522320 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:59.522298    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:51:59.522453 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:59.522380    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:51:59.523341 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:59.523323    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:51:59.523441 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:59.523353    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:51:59.523441 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:59.523385    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rxfwd\""
Jul 28 13:51:59.523525 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:59.523447    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
Jul 28 13:51:59.523616 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:59.523601    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
Jul 28 13:51:59.532660 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:59.532637    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-s2tjv"]
Jul 28 13:51:59.567215 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:59.567189    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff2dfb43-46d1-439b-ba51-2341aaaa4ee0-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-s2tjv\" (UID: \"ff2dfb43-46d1-439b-ba51-2341aaaa4ee0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-s2tjv"
Jul 28 13:51:59.567378 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:59.567242    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9skgv\" (UniqueName: \"kubernetes.io/projected/ff2dfb43-46d1-439b-ba51-2341aaaa4ee0-kube-api-access-9skgv\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-s2tjv\" (UID: \"ff2dfb43-46d1-439b-ba51-2341aaaa4ee0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-s2tjv"
Jul 28 13:51:59.567378 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:59.567306    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff2dfb43-46d1-439b-ba51-2341aaaa4ee0-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-s2tjv\" (UID: \"ff2dfb43-46d1-439b-ba51-2341aaaa4ee0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-s2tjv"
Jul 28 13:51:59.567476 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:59.567381    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff2dfb43-46d1-439b-ba51-2341aaaa4ee0-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-s2tjv\" (UID: \"ff2dfb43-46d1-439b-ba51-2341aaaa4ee0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-s2tjv"
Jul 28 13:51:59.567476 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:59.567405    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff2dfb43-46d1-439b-ba51-2341aaaa4ee0-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-s2tjv\" (UID: \"ff2dfb43-46d1-439b-ba51-2341aaaa4ee0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-s2tjv"
Jul 28 13:51:59.668153 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:59.668112    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9skgv\" (UniqueName: \"kubernetes.io/projected/ff2dfb43-46d1-439b-ba51-2341aaaa4ee0-kube-api-access-9skgv\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-s2tjv\" (UID: \"ff2dfb43-46d1-439b-ba51-2341aaaa4ee0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-s2tjv"
Jul 28 13:51:59.668153 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:59.668152    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff2dfb43-46d1-439b-ba51-2341aaaa4ee0-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-s2tjv\" (UID: \"ff2dfb43-46d1-439b-ba51-2341aaaa4ee0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-s2tjv"
Jul 28 13:51:59.668437 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:59.668209    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff2dfb43-46d1-439b-ba51-2341aaaa4ee0-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-s2tjv\" (UID: \"ff2dfb43-46d1-439b-ba51-2341aaaa4ee0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-s2tjv"
Jul 28 13:51:59.668437 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:59.668237    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff2dfb43-46d1-439b-ba51-2341aaaa4ee0-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-s2tjv\" (UID: \"ff2dfb43-46d1-439b-ba51-2341aaaa4ee0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-s2tjv"
Jul 28 13:51:59.668437 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:59.668301    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff2dfb43-46d1-439b-ba51-2341aaaa4ee0-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-s2tjv\" (UID: \"ff2dfb43-46d1-439b-ba51-2341aaaa4ee0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-s2tjv"
Jul 28 13:51:59.668766 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:59.668740    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff2dfb43-46d1-439b-ba51-2341aaaa4ee0-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-s2tjv\" (UID: \"ff2dfb43-46d1-439b-ba51-2341aaaa4ee0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-s2tjv"
Jul 28 13:51:59.669009 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:59.668985    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff2dfb43-46d1-439b-ba51-2341aaaa4ee0-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-s2tjv\" (UID: \"ff2dfb43-46d1-439b-ba51-2341aaaa4ee0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-s2tjv"
Jul 28 13:51:59.669009 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:59.668999    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff2dfb43-46d1-439b-ba51-2341aaaa4ee0-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-s2tjv\" (UID: \"ff2dfb43-46d1-439b-ba51-2341aaaa4ee0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-s2tjv"
Jul 28 13:51:59.670934 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:59.670918    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff2dfb43-46d1-439b-ba51-2341aaaa4ee0-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-s2tjv\" (UID: \"ff2dfb43-46d1-439b-ba51-2341aaaa4ee0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-s2tjv"
Jul 28 13:51:59.678030 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:59.678007    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9skgv\" (UniqueName: \"kubernetes.io/projected/ff2dfb43-46d1-439b-ba51-2341aaaa4ee0-kube-api-access-9skgv\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-s2tjv\" (UID: \"ff2dfb43-46d1-439b-ba51-2341aaaa4ee0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-s2tjv"
Jul 28 13:51:59.824192 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:59.824099    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-s2tjv"
Jul 28 13:51:59.968714 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:51:59.968692    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-s2tjv"]
Jul 28 13:51:59.971516 ip-10-0-129-177 kubenswrapper[2554]: W0728 13:51:59.971489    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff2dfb43_46d1_439b_ba51_2341aaaa4ee0.slice/crio-aa402c178b35f28bf7479963f29632daeae9764fd65fb4e5274e9b6bf77b4016 WatchSource:0}: Error finding container aa402c178b35f28bf7479963f29632daeae9764fd65fb4e5274e9b6bf77b4016: Status 404 returned error can't find the container with id aa402c178b35f28bf7479963f29632daeae9764fd65fb4e5274e9b6bf77b4016
Jul 28 13:52:00.974577 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:52:00.974538    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-s2tjv" event={"ID":"ff2dfb43-46d1-439b-ba51-2341aaaa4ee0","Type":"ContainerStarted","Data":"021924b121c1e318aea9fe2c35ae071dcf23143c41d94adb1cb98b8998b17e16"}
Jul 28 13:52:00.975052 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:52:00.974603    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-s2tjv" event={"ID":"ff2dfb43-46d1-439b-ba51-2341aaaa4ee0","Type":"ContainerStarted","Data":"aa402c178b35f28bf7479963f29632daeae9764fd65fb4e5274e9b6bf77b4016"}
Jul 28 13:52:03.989038 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:52:03.989002    2554 generic.go:356] "Generic (PLEG): container finished" podID="ff2dfb43-46d1-439b-ba51-2341aaaa4ee0" containerID="021924b121c1e318aea9fe2c35ae071dcf23143c41d94adb1cb98b8998b17e16" exitCode=0
Jul 28 13:52:03.989446 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:52:03.989075    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-s2tjv" event={"ID":"ff2dfb43-46d1-439b-ba51-2341aaaa4ee0","Type":"ContainerDied","Data":"021924b121c1e318aea9fe2c35ae071dcf23143c41d94adb1cb98b8998b17e16"}
Jul 28 13:52:03.990242 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:52:03.990227    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 13:52:26.109441 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:52:26.109409    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-s2tjv" event={"ID":"ff2dfb43-46d1-439b-ba51-2341aaaa4ee0","Type":"ContainerStarted","Data":"5d21322364e7b033f343a69d3516ace296da527f9a09df29e13c7b4a69f3ee14"}
Jul 28 13:52:26.109847 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:52:26.109470    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-s2tjv" event={"ID":"ff2dfb43-46d1-439b-ba51-2341aaaa4ee0","Type":"ContainerStarted","Data":"a30928eaa8fad6e5e6b8216c21bb57ee8037d4af1a4a32837f8b1d3e5088ea06"}
Jul 28 13:52:26.109847 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:52:26.109658    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-s2tjv"
Jul 28 13:52:26.133283 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:52:26.133236    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-s2tjv" podStartSLOduration=2.47076512 podStartE2EDuration="27.133222609s" totalImagesPullingTime="21.073846481s" totalInitContainerRuntime="3.588611008s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:51:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:52:26.131495143 +0000 UTC m=+3542.917370273" watchObservedRunningTime="2026-07-28 13:52:26.133222609 +0000 UTC m=+3542.919097719"
Jul 28 13:52:27.113483 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:52:27.113452    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-s2tjv"
Jul 28 13:52:27.114839 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:52:27.114810    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-s2tjv" podUID="ff2dfb43-46d1-439b-ba51-2341aaaa4ee0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 28 13:52:28.117165 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:52:28.117122    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-s2tjv" podUID="ff2dfb43-46d1-439b-ba51-2341aaaa4ee0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 28 13:52:33.121591 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:52:33.121563    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-s2tjv"
Jul 28 13:52:33.122036 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:52:33.122013    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-s2tjv" podUID="ff2dfb43-46d1-439b-ba51-2341aaaa4ee0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 28 13:52:43.122873 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:52:43.122833    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-s2tjv" podUID="ff2dfb43-46d1-439b-ba51-2341aaaa4ee0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 28 13:52:53.122429 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:52:53.122389    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-s2tjv" podUID="ff2dfb43-46d1-439b-ba51-2341aaaa4ee0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 28 13:53:03.122210 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:03.122172    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-s2tjv" podUID="ff2dfb43-46d1-439b-ba51-2341aaaa4ee0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 28 13:53:13.122774 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:13.122731    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-s2tjv" podUID="ff2dfb43-46d1-439b-ba51-2341aaaa4ee0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 28 13:53:23.122939 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:23.122894    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-s2tjv" podUID="ff2dfb43-46d1-439b-ba51-2341aaaa4ee0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 28 13:53:33.122258 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:33.122216    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-s2tjv" podUID="ff2dfb43-46d1-439b-ba51-2341aaaa4ee0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 28 13:53:43.123247 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:43.123213    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-s2tjv"
Jul 28 13:53:49.542742 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:49.542695    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-s2tjv"]
Jul 28 13:53:49.543210 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:49.542984    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-s2tjv" podUID="ff2dfb43-46d1-439b-ba51-2341aaaa4ee0" containerName="kserve-container" containerID="cri-o://a30928eaa8fad6e5e6b8216c21bb57ee8037d4af1a4a32837f8b1d3e5088ea06" gracePeriod=30
Jul 28 13:53:49.543210 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:49.543081    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-s2tjv" podUID="ff2dfb43-46d1-439b-ba51-2341aaaa4ee0" containerName="kube-rbac-proxy" containerID="cri-o://5d21322364e7b033f343a69d3516ace296da527f9a09df29e13c7b4a69f3ee14" gracePeriod=30
Jul 28 13:53:50.441686 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:50.441643    2554 generic.go:356] "Generic (PLEG): container finished" podID="ff2dfb43-46d1-439b-ba51-2341aaaa4ee0" containerID="5d21322364e7b033f343a69d3516ace296da527f9a09df29e13c7b4a69f3ee14" exitCode=2
Jul 28 13:53:50.441863 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:50.441715    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-s2tjv" event={"ID":"ff2dfb43-46d1-439b-ba51-2341aaaa4ee0","Type":"ContainerDied","Data":"5d21322364e7b033f343a69d3516ace296da527f9a09df29e13c7b4a69f3ee14"}
Jul 28 13:53:53.117677 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:53.117635    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-s2tjv" podUID="ff2dfb43-46d1-439b-ba51-2341aaaa4ee0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused"
Jul 28 13:53:53.123003 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:53.122978    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-s2tjv" podUID="ff2dfb43-46d1-439b-ba51-2341aaaa4ee0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 28 13:53:53.998885 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:53.998864    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-s2tjv"
Jul 28 13:53:54.164242 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.164154    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff2dfb43-46d1-439b-ba51-2341aaaa4ee0-proxy-tls\") pod \"ff2dfb43-46d1-439b-ba51-2341aaaa4ee0\" (UID: \"ff2dfb43-46d1-439b-ba51-2341aaaa4ee0\") "
Jul 28 13:53:54.164242 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.164200    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9skgv\" (UniqueName: \"kubernetes.io/projected/ff2dfb43-46d1-439b-ba51-2341aaaa4ee0-kube-api-access-9skgv\") pod \"ff2dfb43-46d1-439b-ba51-2341aaaa4ee0\" (UID: \"ff2dfb43-46d1-439b-ba51-2341aaaa4ee0\") "
Jul 28 13:53:54.164242 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.164221    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff2dfb43-46d1-439b-ba51-2341aaaa4ee0-kserve-provision-location\") pod \"ff2dfb43-46d1-439b-ba51-2341aaaa4ee0\" (UID: \"ff2dfb43-46d1-439b-ba51-2341aaaa4ee0\") "
Jul 28 13:53:54.164769 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.164341    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff2dfb43-46d1-439b-ba51-2341aaaa4ee0-cabundle-cert\") pod \"ff2dfb43-46d1-439b-ba51-2341aaaa4ee0\" (UID: \"ff2dfb43-46d1-439b-ba51-2341aaaa4ee0\") "
Jul 28 13:53:54.164769 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.164448    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff2dfb43-46d1-439b-ba51-2341aaaa4ee0-kube-rbac-proxy-sar-config\") pod \"ff2dfb43-46d1-439b-ba51-2341aaaa4ee0\" (UID: \"ff2dfb43-46d1-439b-ba51-2341aaaa4ee0\") "
Jul 28 13:53:54.164769 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.164598    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff2dfb43-46d1-439b-ba51-2341aaaa4ee0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ff2dfb43-46d1-439b-ba51-2341aaaa4ee0" (UID: "ff2dfb43-46d1-439b-ba51-2341aaaa4ee0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:53:54.164769 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.164700    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff2dfb43-46d1-439b-ba51-2341aaaa4ee0-kserve-provision-location\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:53:54.164769 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.164698    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff2dfb43-46d1-439b-ba51-2341aaaa4ee0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ff2dfb43-46d1-439b-ba51-2341aaaa4ee0" (UID: "ff2dfb43-46d1-439b-ba51-2341aaaa4ee0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:53:54.164769 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.164732    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff2dfb43-46d1-439b-ba51-2341aaaa4ee0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ff2dfb43-46d1-439b-ba51-2341aaaa4ee0" (UID: "ff2dfb43-46d1-439b-ba51-2341aaaa4ee0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:53:54.166542 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.166517    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff2dfb43-46d1-439b-ba51-2341aaaa4ee0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ff2dfb43-46d1-439b-ba51-2341aaaa4ee0" (UID: "ff2dfb43-46d1-439b-ba51-2341aaaa4ee0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:53:54.166655 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.166540    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff2dfb43-46d1-439b-ba51-2341aaaa4ee0-kube-api-access-9skgv" (OuterVolumeSpecName: "kube-api-access-9skgv") pod "ff2dfb43-46d1-439b-ba51-2341aaaa4ee0" (UID: "ff2dfb43-46d1-439b-ba51-2341aaaa4ee0"). InnerVolumeSpecName "kube-api-access-9skgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:53:54.265579 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.265550    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff2dfb43-46d1-439b-ba51-2341aaaa4ee0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:53:54.265579 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.265575    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff2dfb43-46d1-439b-ba51-2341aaaa4ee0-proxy-tls\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:53:54.265579 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.265585    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9skgv\" (UniqueName: \"kubernetes.io/projected/ff2dfb43-46d1-439b-ba51-2341aaaa4ee0-kube-api-access-9skgv\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:53:54.265791 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.265593    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff2dfb43-46d1-439b-ba51-2341aaaa4ee0-cabundle-cert\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:53:54.460715 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.460682    2554 generic.go:356] "Generic (PLEG): container finished" podID="ff2dfb43-46d1-439b-ba51-2341aaaa4ee0" containerID="a30928eaa8fad6e5e6b8216c21bb57ee8037d4af1a4a32837f8b1d3e5088ea06" exitCode=0
Jul 28 13:53:54.460866 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.460736    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-s2tjv"
Jul 28 13:53:54.460866 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.460750    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-s2tjv" event={"ID":"ff2dfb43-46d1-439b-ba51-2341aaaa4ee0","Type":"ContainerDied","Data":"a30928eaa8fad6e5e6b8216c21bb57ee8037d4af1a4a32837f8b1d3e5088ea06"}
Jul 28 13:53:54.460866 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.460821    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-s2tjv" event={"ID":"ff2dfb43-46d1-439b-ba51-2341aaaa4ee0","Type":"ContainerDied","Data":"aa402c178b35f28bf7479963f29632daeae9764fd65fb4e5274e9b6bf77b4016"}
Jul 28 13:53:54.460866 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.460860    2554 scope.go:117] "RemoveContainer" containerID="5d21322364e7b033f343a69d3516ace296da527f9a09df29e13c7b4a69f3ee14"
Jul 28 13:53:54.469952 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.469932    2554 scope.go:117] "RemoveContainer" containerID="a30928eaa8fad6e5e6b8216c21bb57ee8037d4af1a4a32837f8b1d3e5088ea06"
Jul 28 13:53:54.479109 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.479086    2554 scope.go:117] "RemoveContainer" containerID="021924b121c1e318aea9fe2c35ae071dcf23143c41d94adb1cb98b8998b17e16"
Jul 28 13:53:54.485155 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.485126    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-s2tjv"]
Jul 28 13:53:54.487917 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.487898    2554 scope.go:117] "RemoveContainer" containerID="5d21322364e7b033f343a69d3516ace296da527f9a09df29e13c7b4a69f3ee14"
Jul 28 13:53:54.488189 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:53:54.488170    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d21322364e7b033f343a69d3516ace296da527f9a09df29e13c7b4a69f3ee14\": container with ID starting with 5d21322364e7b033f343a69d3516ace296da527f9a09df29e13c7b4a69f3ee14 not found: ID does not exist" containerID="5d21322364e7b033f343a69d3516ace296da527f9a09df29e13c7b4a69f3ee14"
Jul 28 13:53:54.488323 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.488201    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d21322364e7b033f343a69d3516ace296da527f9a09df29e13c7b4a69f3ee14"} err="failed to get container status \"5d21322364e7b033f343a69d3516ace296da527f9a09df29e13c7b4a69f3ee14\": rpc error: code = NotFound desc = could not find container \"5d21322364e7b033f343a69d3516ace296da527f9a09df29e13c7b4a69f3ee14\": container with ID starting with 5d21322364e7b033f343a69d3516ace296da527f9a09df29e13c7b4a69f3ee14 not found: ID does not exist"
Jul 28 13:53:54.488323 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.488217    2554 scope.go:117] "RemoveContainer" containerID="a30928eaa8fad6e5e6b8216c21bb57ee8037d4af1a4a32837f8b1d3e5088ea06"
Jul 28 13:53:54.488511 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:53:54.488493    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a30928eaa8fad6e5e6b8216c21bb57ee8037d4af1a4a32837f8b1d3e5088ea06\": container with ID starting with a30928eaa8fad6e5e6b8216c21bb57ee8037d4af1a4a32837f8b1d3e5088ea06 not found: ID does not exist" containerID="a30928eaa8fad6e5e6b8216c21bb57ee8037d4af1a4a32837f8b1d3e5088ea06"
Jul 28 13:53:54.488571 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.488517    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a30928eaa8fad6e5e6b8216c21bb57ee8037d4af1a4a32837f8b1d3e5088ea06"} err="failed to get container status \"a30928eaa8fad6e5e6b8216c21bb57ee8037d4af1a4a32837f8b1d3e5088ea06\": rpc error: code = NotFound desc = could not find container \"a30928eaa8fad6e5e6b8216c21bb57ee8037d4af1a4a32837f8b1d3e5088ea06\": container with ID starting with a30928eaa8fad6e5e6b8216c21bb57ee8037d4af1a4a32837f8b1d3e5088ea06 not found: ID does not exist"
Jul 28 13:53:54.488571 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.488532    2554 scope.go:117] "RemoveContainer" containerID="021924b121c1e318aea9fe2c35ae071dcf23143c41d94adb1cb98b8998b17e16"
Jul 28 13:53:54.488760 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:53:54.488740    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"021924b121c1e318aea9fe2c35ae071dcf23143c41d94adb1cb98b8998b17e16\": container with ID starting with 021924b121c1e318aea9fe2c35ae071dcf23143c41d94adb1cb98b8998b17e16 not found: ID does not exist" containerID="021924b121c1e318aea9fe2c35ae071dcf23143c41d94adb1cb98b8998b17e16"
Jul 28 13:53:54.488823 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.488765    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"021924b121c1e318aea9fe2c35ae071dcf23143c41d94adb1cb98b8998b17e16"} err="failed to get container status \"021924b121c1e318aea9fe2c35ae071dcf23143c41d94adb1cb98b8998b17e16\": rpc error: code = NotFound desc = could not find container \"021924b121c1e318aea9fe2c35ae071dcf23143c41d94adb1cb98b8998b17e16\": container with ID starting with 021924b121c1e318aea9fe2c35ae071dcf23143c41d94adb1cb98b8998b17e16 not found: ID does not exist"
Jul 28 13:53:54.488975 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.488957    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-s2tjv"]
Jul 28 13:53:54.718908 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.718830    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-f6ncv"]
Jul 28 13:53:54.719168 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.719155    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff2dfb43-46d1-439b-ba51-2341aaaa4ee0" containerName="kube-rbac-proxy"
Jul 28 13:53:54.719212 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.719169    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff2dfb43-46d1-439b-ba51-2341aaaa4ee0" containerName="kube-rbac-proxy"
Jul 28 13:53:54.719212 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.719182    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff2dfb43-46d1-439b-ba51-2341aaaa4ee0" containerName="kserve-container"
Jul 28 13:53:54.719212 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.719187    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff2dfb43-46d1-439b-ba51-2341aaaa4ee0" containerName="kserve-container"
Jul 28 13:53:54.719212 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.719197    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff2dfb43-46d1-439b-ba51-2341aaaa4ee0" containerName="storage-initializer"
Jul 28 13:53:54.719212 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.719203    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff2dfb43-46d1-439b-ba51-2341aaaa4ee0" containerName="storage-initializer"
Jul 28 13:53:54.719399 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.719301    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff2dfb43-46d1-439b-ba51-2341aaaa4ee0" containerName="kube-rbac-proxy"
Jul 28 13:53:54.719399 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.719330    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff2dfb43-46d1-439b-ba51-2341aaaa4ee0" containerName="kserve-container"
Jul 28 13:53:54.723957 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.723936    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-f6ncv"
Jul 28 13:53:54.726460 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.726437    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
Jul 28 13:53:54.726460 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.726450    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
Jul 28 13:53:54.726640 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.726531    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rxfwd\""
Jul 28 13:53:54.726640 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.726531    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:53:54.726743 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.726711    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:53:54.726831 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.726814    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:53:54.726917 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.726891    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:53:54.733977 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.733954    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-f6ncv"]
Jul 28 13:53:54.870980 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.870946    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx9wf\" (UniqueName: \"kubernetes.io/projected/e508880b-f31b-46e6-b189-d5c9a4872595-kube-api-access-bx9wf\") pod \"isvc-predictive-xgboost-predictor-695f978978-f6ncv\" (UID: \"e508880b-f31b-46e6-b189-d5c9a4872595\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-f6ncv"
Jul 28 13:53:54.871147 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.871006    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e508880b-f31b-46e6-b189-d5c9a4872595-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-f6ncv\" (UID: \"e508880b-f31b-46e6-b189-d5c9a4872595\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-f6ncv"
Jul 28 13:53:54.871147 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.871036    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e508880b-f31b-46e6-b189-d5c9a4872595-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-f6ncv\" (UID: \"e508880b-f31b-46e6-b189-d5c9a4872595\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-f6ncv"
Jul 28 13:53:54.871147 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.871068    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e508880b-f31b-46e6-b189-d5c9a4872595-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-f6ncv\" (UID: \"e508880b-f31b-46e6-b189-d5c9a4872595\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-f6ncv"
Jul 28 13:53:54.871147 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.871099    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e508880b-f31b-46e6-b189-d5c9a4872595-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-f6ncv\" (UID: \"e508880b-f31b-46e6-b189-d5c9a4872595\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-f6ncv"
Jul 28 13:53:54.971959 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.971878    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e508880b-f31b-46e6-b189-d5c9a4872595-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-f6ncv\" (UID: \"e508880b-f31b-46e6-b189-d5c9a4872595\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-f6ncv"
Jul 28 13:53:54.971959 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.971923    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e508880b-f31b-46e6-b189-d5c9a4872595-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-f6ncv\" (UID: \"e508880b-f31b-46e6-b189-d5c9a4872595\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-f6ncv"
Jul 28 13:53:54.971959 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.971956    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e508880b-f31b-46e6-b189-d5c9a4872595-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-f6ncv\" (UID: \"e508880b-f31b-46e6-b189-d5c9a4872595\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-f6ncv"
Jul 28 13:53:54.972213 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.972000    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e508880b-f31b-46e6-b189-d5c9a4872595-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-f6ncv\" (UID: \"e508880b-f31b-46e6-b189-d5c9a4872595\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-f6ncv"
Jul 28 13:53:54.972213 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:53:54.972019    2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-serving-cert: secret "isvc-predictive-xgboost-predictor-serving-cert" not found
Jul 28 13:53:54.972213 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.972061    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx9wf\" (UniqueName: \"kubernetes.io/projected/e508880b-f31b-46e6-b189-d5c9a4872595-kube-api-access-bx9wf\") pod \"isvc-predictive-xgboost-predictor-695f978978-f6ncv\" (UID: \"e508880b-f31b-46e6-b189-d5c9a4872595\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-f6ncv"
Jul 28 13:53:54.972213 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:53:54.972076    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e508880b-f31b-46e6-b189-d5c9a4872595-proxy-tls podName:e508880b-f31b-46e6-b189-d5c9a4872595 nodeName:}" failed. No retries permitted until 2026-07-28 13:53:55.472059551 +0000 UTC m=+3632.257934640 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e508880b-f31b-46e6-b189-d5c9a4872595-proxy-tls") pod "isvc-predictive-xgboost-predictor-695f978978-f6ncv" (UID: "e508880b-f31b-46e6-b189-d5c9a4872595") : secret "isvc-predictive-xgboost-predictor-serving-cert" not found
Jul 28 13:53:54.972481 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.972322    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e508880b-f31b-46e6-b189-d5c9a4872595-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-f6ncv\" (UID: \"e508880b-f31b-46e6-b189-d5c9a4872595\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-f6ncv"
Jul 28 13:53:54.972796 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.972771    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e508880b-f31b-46e6-b189-d5c9a4872595-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-f6ncv\" (UID: \"e508880b-f31b-46e6-b189-d5c9a4872595\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-f6ncv"
Jul 28 13:53:54.972876 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.972860    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e508880b-f31b-46e6-b189-d5c9a4872595-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-f6ncv\" (UID: \"e508880b-f31b-46e6-b189-d5c9a4872595\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-f6ncv"
Jul 28 13:53:54.980689 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:54.980670    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx9wf\" (UniqueName: \"kubernetes.io/projected/e508880b-f31b-46e6-b189-d5c9a4872595-kube-api-access-bx9wf\") pod \"isvc-predictive-xgboost-predictor-695f978978-f6ncv\" (UID: \"e508880b-f31b-46e6-b189-d5c9a4872595\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-f6ncv"
Jul 28 13:53:55.476968 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:55.476936    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e508880b-f31b-46e6-b189-d5c9a4872595-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-f6ncv\" (UID: \"e508880b-f31b-46e6-b189-d5c9a4872595\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-f6ncv"
Jul 28 13:53:55.479438 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:55.479420    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e508880b-f31b-46e6-b189-d5c9a4872595-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-f6ncv\" (UID: \"e508880b-f31b-46e6-b189-d5c9a4872595\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-f6ncv"
Jul 28 13:53:55.636315 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:55.636252    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-f6ncv"
Jul 28 13:53:55.764780 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:55.764754    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-f6ncv"]
Jul 28 13:53:55.766959 ip-10-0-129-177 kubenswrapper[2554]: W0728 13:53:55.766931    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode508880b_f31b_46e6_b189_d5c9a4872595.slice/crio-a457b90b6394ce0e43b48309e32a8cb520926c6c9764c7b6a9fd9623238e73e6 WatchSource:0}: Error finding container a457b90b6394ce0e43b48309e32a8cb520926c6c9764c7b6a9fd9623238e73e6: Status 404 returned error can't find the container with id a457b90b6394ce0e43b48309e32a8cb520926c6c9764c7b6a9fd9623238e73e6
Jul 28 13:53:55.803525 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:55.803498    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff2dfb43-46d1-439b-ba51-2341aaaa4ee0" path="/var/lib/kubelet/pods/ff2dfb43-46d1-439b-ba51-2341aaaa4ee0/volumes"
Jul 28 13:53:56.480386 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:56.480353    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-f6ncv" event={"ID":"e508880b-f31b-46e6-b189-d5c9a4872595","Type":"ContainerStarted","Data":"fe5632de191e1e8443b73afb2ac5c9d0c0163c48b9ca1680def73c0d45b07fdf"}
Jul 28 13:53:56.480777 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:56.480412    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-f6ncv" event={"ID":"e508880b-f31b-46e6-b189-d5c9a4872595","Type":"ContainerStarted","Data":"a457b90b6394ce0e43b48309e32a8cb520926c6c9764c7b6a9fd9623238e73e6"}
Jul 28 13:53:59.494252 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:59.494215    2554 generic.go:356] "Generic (PLEG): container finished" podID="e508880b-f31b-46e6-b189-d5c9a4872595" containerID="fe5632de191e1e8443b73afb2ac5c9d0c0163c48b9ca1680def73c0d45b07fdf" exitCode=0
Jul 28 13:53:59.494665 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:53:59.494312    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-f6ncv" event={"ID":"e508880b-f31b-46e6-b189-d5c9a4872595","Type":"ContainerDied","Data":"fe5632de191e1e8443b73afb2ac5c9d0c0163c48b9ca1680def73c0d45b07fdf"}
Jul 28 13:54:00.501478 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:54:00.501446    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-f6ncv" event={"ID":"e508880b-f31b-46e6-b189-d5c9a4872595","Type":"ContainerStarted","Data":"86eb068acac64847c6ead5fc81f978baf3d9d52b1475db4cd33050c9d69bdb3e"}
Jul 28 13:54:00.501859 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:54:00.501506    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-f6ncv" event={"ID":"e508880b-f31b-46e6-b189-d5c9a4872595","Type":"ContainerStarted","Data":"1445855a099016bbc3d3c9193888a60680f18e2fc5276951a293d74d10d74a82"}
Jul 28 13:54:00.501859 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:54:00.501685    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-f6ncv"
Jul 28 13:54:01.506093 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:54:01.506057    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-f6ncv"
Jul 28 13:54:01.507403 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:54:01.507377    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-f6ncv" podUID="e508880b-f31b-46e6-b189-d5c9a4872595" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 28 13:54:02.510250 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:54:02.510212    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-f6ncv" podUID="e508880b-f31b-46e6-b189-d5c9a4872595" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 28 13:54:07.515331 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:54:07.515304    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-f6ncv"
Jul 28 13:54:07.515887 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:54:07.515861    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-f6ncv" podUID="e508880b-f31b-46e6-b189-d5c9a4872595" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 28 13:54:07.535765 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:54:07.535721    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-f6ncv" podStartSLOduration=10.036523233 podStartE2EDuration="13.535707801s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.499184568s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:53:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:54:00.522023292 +0000 UTC m=+3637.307898402" watchObservedRunningTime="2026-07-28 13:54:07.535707801 +0000 UTC m=+3644.321582912"
Jul 28 13:54:17.516197 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:54:17.516159    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-f6ncv" podUID="e508880b-f31b-46e6-b189-d5c9a4872595" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 28 13:54:27.516049 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:54:27.516004    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-f6ncv" podUID="e508880b-f31b-46e6-b189-d5c9a4872595" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 28 13:54:37.516014 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:54:37.515969    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-f6ncv" podUID="e508880b-f31b-46e6-b189-d5c9a4872595" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 28 13:54:47.515929 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:54:47.515842    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-f6ncv" podUID="e508880b-f31b-46e6-b189-d5c9a4872595" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 28 13:54:57.516254 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:54:57.516209    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-f6ncv" podUID="e508880b-f31b-46e6-b189-d5c9a4872595" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 28 13:55:07.516225 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:07.516179    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-f6ncv" podUID="e508880b-f31b-46e6-b189-d5c9a4872595" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 28 13:55:17.516920 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:17.516888    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-f6ncv"
Jul 28 13:55:24.857501 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:24.857464    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-f6ncv"]
Jul 28 13:55:24.858065 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:24.857719    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-f6ncv" podUID="e508880b-f31b-46e6-b189-d5c9a4872595" containerName="kserve-container" containerID="cri-o://1445855a099016bbc3d3c9193888a60680f18e2fc5276951a293d74d10d74a82" gracePeriod=30
Jul 28 13:55:24.858065 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:24.857795    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-f6ncv" podUID="e508880b-f31b-46e6-b189-d5c9a4872595" containerName="kube-rbac-proxy" containerID="cri-o://86eb068acac64847c6ead5fc81f978baf3d9d52b1475db4cd33050c9d69bdb3e" gracePeriod=30
Jul 28 13:55:25.849127 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:25.849095    2554 generic.go:356] "Generic (PLEG): container finished" podID="e508880b-f31b-46e6-b189-d5c9a4872595" containerID="86eb068acac64847c6ead5fc81f978baf3d9d52b1475db4cd33050c9d69bdb3e" exitCode=2
Jul 28 13:55:25.849313 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:25.849167    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-f6ncv" event={"ID":"e508880b-f31b-46e6-b189-d5c9a4872595","Type":"ContainerDied","Data":"86eb068acac64847c6ead5fc81f978baf3d9d52b1475db4cd33050c9d69bdb3e"}
Jul 28 13:55:27.511292 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:27.511232    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-f6ncv" podUID="e508880b-f31b-46e6-b189-d5c9a4872595" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
Jul 28 13:55:27.516626 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:27.516602    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-f6ncv" podUID="e508880b-f31b-46e6-b189-d5c9a4872595" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 28 13:55:29.410090 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:29.410066    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-f6ncv"
Jul 28 13:55:29.470009 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:29.469974    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e508880b-f31b-46e6-b189-d5c9a4872595-kube-rbac-proxy-sar-config\") pod \"e508880b-f31b-46e6-b189-d5c9a4872595\" (UID: \"e508880b-f31b-46e6-b189-d5c9a4872595\") "
Jul 28 13:55:29.470177 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:29.470043    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e508880b-f31b-46e6-b189-d5c9a4872595-cabundle-cert\") pod \"e508880b-f31b-46e6-b189-d5c9a4872595\" (UID: \"e508880b-f31b-46e6-b189-d5c9a4872595\") "
Jul 28 13:55:29.470177 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:29.470076    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e508880b-f31b-46e6-b189-d5c9a4872595-proxy-tls\") pod \"e508880b-f31b-46e6-b189-d5c9a4872595\" (UID: \"e508880b-f31b-46e6-b189-d5c9a4872595\") "
Jul 28 13:55:29.470177 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:29.470165    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bx9wf\" (UniqueName: \"kubernetes.io/projected/e508880b-f31b-46e6-b189-d5c9a4872595-kube-api-access-bx9wf\") pod \"e508880b-f31b-46e6-b189-d5c9a4872595\" (UID: \"e508880b-f31b-46e6-b189-d5c9a4872595\") "
Jul 28 13:55:29.470386 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:29.470206    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e508880b-f31b-46e6-b189-d5c9a4872595-kserve-provision-location\") pod \"e508880b-f31b-46e6-b189-d5c9a4872595\" (UID: \"e508880b-f31b-46e6-b189-d5c9a4872595\") "
Jul 28 13:55:29.470452 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:29.470419    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e508880b-f31b-46e6-b189-d5c9a4872595-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e508880b-f31b-46e6-b189-d5c9a4872595" (UID: "e508880b-f31b-46e6-b189-d5c9a4872595"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:55:29.470452 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:29.470433    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e508880b-f31b-46e6-b189-d5c9a4872595-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e508880b-f31b-46e6-b189-d5c9a4872595" (UID: "e508880b-f31b-46e6-b189-d5c9a4872595"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:55:29.470635 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:29.470611    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e508880b-f31b-46e6-b189-d5c9a4872595-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e508880b-f31b-46e6-b189-d5c9a4872595" (UID: "e508880b-f31b-46e6-b189-d5c9a4872595"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:55:29.472252 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:29.472223    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e508880b-f31b-46e6-b189-d5c9a4872595-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e508880b-f31b-46e6-b189-d5c9a4872595" (UID: "e508880b-f31b-46e6-b189-d5c9a4872595"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:55:29.472378 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:29.472342    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e508880b-f31b-46e6-b189-d5c9a4872595-kube-api-access-bx9wf" (OuterVolumeSpecName: "kube-api-access-bx9wf") pod "e508880b-f31b-46e6-b189-d5c9a4872595" (UID: "e508880b-f31b-46e6-b189-d5c9a4872595"). InnerVolumeSpecName "kube-api-access-bx9wf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:55:29.571419 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:29.571385    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx9wf\" (UniqueName: \"kubernetes.io/projected/e508880b-f31b-46e6-b189-d5c9a4872595-kube-api-access-bx9wf\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:55:29.571419 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:29.571414    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e508880b-f31b-46e6-b189-d5c9a4872595-kserve-provision-location\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:55:29.571419 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:29.571427    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e508880b-f31b-46e6-b189-d5c9a4872595-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:55:29.571673 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:29.571439    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e508880b-f31b-46e6-b189-d5c9a4872595-cabundle-cert\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:55:29.571673 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:29.571451    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e508880b-f31b-46e6-b189-d5c9a4872595-proxy-tls\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:55:29.868841 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:29.868805    2554 generic.go:356] "Generic (PLEG): container finished" podID="e508880b-f31b-46e6-b189-d5c9a4872595" containerID="1445855a099016bbc3d3c9193888a60680f18e2fc5276951a293d74d10d74a82" exitCode=0
Jul 28 13:55:29.869001 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:29.868869    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-f6ncv"
Jul 28 13:55:29.869001 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:29.868891    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-f6ncv" event={"ID":"e508880b-f31b-46e6-b189-d5c9a4872595","Type":"ContainerDied","Data":"1445855a099016bbc3d3c9193888a60680f18e2fc5276951a293d74d10d74a82"}
Jul 28 13:55:29.869001 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:29.868964    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-f6ncv" event={"ID":"e508880b-f31b-46e6-b189-d5c9a4872595","Type":"ContainerDied","Data":"a457b90b6394ce0e43b48309e32a8cb520926c6c9764c7b6a9fd9623238e73e6"}
Jul 28 13:55:29.869128 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:29.869005    2554 scope.go:117] "RemoveContainer" containerID="86eb068acac64847c6ead5fc81f978baf3d9d52b1475db4cd33050c9d69bdb3e"
Jul 28 13:55:29.878229 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:29.878209    2554 scope.go:117] "RemoveContainer" containerID="1445855a099016bbc3d3c9193888a60680f18e2fc5276951a293d74d10d74a82"
Jul 28 13:55:29.886449 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:29.886401    2554 scope.go:117] "RemoveContainer" containerID="fe5632de191e1e8443b73afb2ac5c9d0c0163c48b9ca1680def73c0d45b07fdf"
Jul 28 13:55:29.894157 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:29.894134    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-f6ncv"]
Jul 28 13:55:29.894769 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:29.894754    2554 scope.go:117] "RemoveContainer" containerID="86eb068acac64847c6ead5fc81f978baf3d9d52b1475db4cd33050c9d69bdb3e"
Jul 28 13:55:29.895032 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:55:29.895011    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86eb068acac64847c6ead5fc81f978baf3d9d52b1475db4cd33050c9d69bdb3e\": container with ID starting with 86eb068acac64847c6ead5fc81f978baf3d9d52b1475db4cd33050c9d69bdb3e not found: ID does not exist" containerID="86eb068acac64847c6ead5fc81f978baf3d9d52b1475db4cd33050c9d69bdb3e"
Jul 28 13:55:29.895098 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:29.895044    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86eb068acac64847c6ead5fc81f978baf3d9d52b1475db4cd33050c9d69bdb3e"} err="failed to get container status \"86eb068acac64847c6ead5fc81f978baf3d9d52b1475db4cd33050c9d69bdb3e\": rpc error: code = NotFound desc = could not find container \"86eb068acac64847c6ead5fc81f978baf3d9d52b1475db4cd33050c9d69bdb3e\": container with ID starting with 86eb068acac64847c6ead5fc81f978baf3d9d52b1475db4cd33050c9d69bdb3e not found: ID does not exist"
Jul 28 13:55:29.895098 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:29.895062    2554 scope.go:117] "RemoveContainer" containerID="1445855a099016bbc3d3c9193888a60680f18e2fc5276951a293d74d10d74a82"
Jul 28 13:55:29.895380 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:55:29.895356    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1445855a099016bbc3d3c9193888a60680f18e2fc5276951a293d74d10d74a82\": container with ID starting with 1445855a099016bbc3d3c9193888a60680f18e2fc5276951a293d74d10d74a82 not found: ID does not exist" containerID="1445855a099016bbc3d3c9193888a60680f18e2fc5276951a293d74d10d74a82"
Jul 28 13:55:29.895469 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:29.895386    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1445855a099016bbc3d3c9193888a60680f18e2fc5276951a293d74d10d74a82"} err="failed to get container status \"1445855a099016bbc3d3c9193888a60680f18e2fc5276951a293d74d10d74a82\": rpc error: code = NotFound desc = could not find container \"1445855a099016bbc3d3c9193888a60680f18e2fc5276951a293d74d10d74a82\": container with ID starting with 1445855a099016bbc3d3c9193888a60680f18e2fc5276951a293d74d10d74a82 not found: ID does not exist"
Jul 28 13:55:29.895469 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:29.895402    2554 scope.go:117] "RemoveContainer" containerID="fe5632de191e1e8443b73afb2ac5c9d0c0163c48b9ca1680def73c0d45b07fdf"
Jul 28 13:55:29.895697 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:55:29.895674    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe5632de191e1e8443b73afb2ac5c9d0c0163c48b9ca1680def73c0d45b07fdf\": container with ID starting with fe5632de191e1e8443b73afb2ac5c9d0c0163c48b9ca1680def73c0d45b07fdf not found: ID does not exist" containerID="fe5632de191e1e8443b73afb2ac5c9d0c0163c48b9ca1680def73c0d45b07fdf"
Jul 28 13:55:29.895748 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:29.895703    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe5632de191e1e8443b73afb2ac5c9d0c0163c48b9ca1680def73c0d45b07fdf"} err="failed to get container status \"fe5632de191e1e8443b73afb2ac5c9d0c0163c48b9ca1680def73c0d45b07fdf\": rpc error: code = NotFound desc = could not find container \"fe5632de191e1e8443b73afb2ac5c9d0c0163c48b9ca1680def73c0d45b07fdf\": container with ID starting with fe5632de191e1e8443b73afb2ac5c9d0c0163c48b9ca1680def73c0d45b07fdf not found: ID does not exist"
Jul 28 13:55:29.903123 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:29.903103    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-f6ncv"]
Jul 28 13:55:30.137319 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:30.137227    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9"]
Jul 28 13:55:30.137641 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:30.137625    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e508880b-f31b-46e6-b189-d5c9a4872595" containerName="storage-initializer"
Jul 28 13:55:30.137641 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:30.137642    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="e508880b-f31b-46e6-b189-d5c9a4872595" containerName="storage-initializer"
Jul 28 13:55:30.137820 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:30.137655    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e508880b-f31b-46e6-b189-d5c9a4872595" containerName="kserve-container"
Jul 28 13:55:30.137820 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:30.137660    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="e508880b-f31b-46e6-b189-d5c9a4872595" containerName="kserve-container"
Jul 28 13:55:30.137820 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:30.137668    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e508880b-f31b-46e6-b189-d5c9a4872595" containerName="kube-rbac-proxy"
Jul 28 13:55:30.137820 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:30.137674    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="e508880b-f31b-46e6-b189-d5c9a4872595" containerName="kube-rbac-proxy"
Jul 28 13:55:30.137820 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:30.137729    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="e508880b-f31b-46e6-b189-d5c9a4872595" containerName="kube-rbac-proxy"
Jul 28 13:55:30.137820 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:30.137736    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="e508880b-f31b-46e6-b189-d5c9a4872595" containerName="kserve-container"
Jul 28 13:55:30.142438 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:30.142420    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9"
Jul 28 13:55:30.145840 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:30.145819    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
Jul 28 13:55:30.146056 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:30.146039    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 13:55:30.146392 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:30.146377    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 13:55:30.146510 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:30.146494    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rxfwd\""
Jul 28 13:55:30.146722 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:30.146704    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
Jul 28 13:55:30.146835 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:30.146821    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 13:55:30.147183 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:30.147169    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 13:55:30.157354 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:30.157332    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9"]
Jul 28 13:55:30.276386 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:30.276353    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c07c02b-8105-4e90-9062-c65549925264-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9\" (UID: \"9c07c02b-8105-4e90-9062-c65549925264\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9"
Jul 28 13:55:30.276386 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:30.276389    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86kzk\" (UniqueName: \"kubernetes.io/projected/9c07c02b-8105-4e90-9062-c65549925264-kube-api-access-86kzk\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9\" (UID: \"9c07c02b-8105-4e90-9062-c65549925264\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9"
Jul 28 13:55:30.276631 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:30.276423    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c07c02b-8105-4e90-9062-c65549925264-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9\" (UID: \"9c07c02b-8105-4e90-9062-c65549925264\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9"
Jul 28 13:55:30.276631 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:30.276469    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c07c02b-8105-4e90-9062-c65549925264-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9\" (UID: \"9c07c02b-8105-4e90-9062-c65549925264\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9"
Jul 28 13:55:30.276631 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:30.276599    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c07c02b-8105-4e90-9062-c65549925264-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9\" (UID: \"9c07c02b-8105-4e90-9062-c65549925264\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9"
Jul 28 13:55:30.377013 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:30.376979    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c07c02b-8105-4e90-9062-c65549925264-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9\" (UID: \"9c07c02b-8105-4e90-9062-c65549925264\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9"
Jul 28 13:55:30.377200 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:30.377043    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c07c02b-8105-4e90-9062-c65549925264-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9\" (UID: \"9c07c02b-8105-4e90-9062-c65549925264\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9"
Jul 28 13:55:30.377200 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:30.377080    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c07c02b-8105-4e90-9062-c65549925264-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9\" (UID: \"9c07c02b-8105-4e90-9062-c65549925264\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9"
Jul 28 13:55:30.377200 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:30.377100    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86kzk\" (UniqueName: \"kubernetes.io/projected/9c07c02b-8105-4e90-9062-c65549925264-kube-api-access-86kzk\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9\" (UID: \"9c07c02b-8105-4e90-9062-c65549925264\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9"
Jul 28 13:55:30.377200 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:30.377120    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c07c02b-8105-4e90-9062-c65549925264-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9\" (UID: \"9c07c02b-8105-4e90-9062-c65549925264\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9"
Jul 28 13:55:30.377491 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:30.377469    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c07c02b-8105-4e90-9062-c65549925264-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9\" (UID: \"9c07c02b-8105-4e90-9062-c65549925264\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9"
Jul 28 13:55:30.377888 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:30.377860    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c07c02b-8105-4e90-9062-c65549925264-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9\" (UID: \"9c07c02b-8105-4e90-9062-c65549925264\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9"
Jul 28 13:55:30.377981 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:30.377860    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c07c02b-8105-4e90-9062-c65549925264-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9\" (UID: \"9c07c02b-8105-4e90-9062-c65549925264\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9"
Jul 28 13:55:30.379713 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:30.379683    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c07c02b-8105-4e90-9062-c65549925264-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9\" (UID: \"9c07c02b-8105-4e90-9062-c65549925264\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9"
Jul 28 13:55:30.387623 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:30.387572    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86kzk\" (UniqueName: \"kubernetes.io/projected/9c07c02b-8105-4e90-9062-c65549925264-kube-api-access-86kzk\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9\" (UID: \"9c07c02b-8105-4e90-9062-c65549925264\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9"
Jul 28 13:55:30.453497 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:30.453467    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9"
Jul 28 13:55:30.587237 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:30.587214    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9"]
Jul 28 13:55:30.589798 ip-10-0-129-177 kubenswrapper[2554]: W0728 13:55:30.589771    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c07c02b_8105_4e90_9062_c65549925264.slice/crio-714d4af7a161735da5133d4debf567a0a51d1a3e5b4f2dc3675a5c84e79eddc1 WatchSource:0}: Error finding container 714d4af7a161735da5133d4debf567a0a51d1a3e5b4f2dc3675a5c84e79eddc1: Status 404 returned error can't find the container with id 714d4af7a161735da5133d4debf567a0a51d1a3e5b4f2dc3675a5c84e79eddc1
Jul 28 13:55:30.874760 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:30.874715    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9" event={"ID":"9c07c02b-8105-4e90-9062-c65549925264","Type":"ContainerStarted","Data":"105bb2e676835a6208b760767ad172b92b92226cbe53dfa759c5f3068ef49ff5"}
Jul 28 13:55:30.874932 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:30.874786    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9" event={"ID":"9c07c02b-8105-4e90-9062-c65549925264","Type":"ContainerStarted","Data":"714d4af7a161735da5133d4debf567a0a51d1a3e5b4f2dc3675a5c84e79eddc1"}
Jul 28 13:55:31.801715 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:31.801685    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e508880b-f31b-46e6-b189-d5c9a4872595" path="/var/lib/kubelet/pods/e508880b-f31b-46e6-b189-d5c9a4872595/volumes"
Jul 28 13:55:34.900459 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:34.900421    2554 generic.go:356] "Generic (PLEG): container finished" podID="9c07c02b-8105-4e90-9062-c65549925264" containerID="105bb2e676835a6208b760767ad172b92b92226cbe53dfa759c5f3068ef49ff5" exitCode=0
Jul 28 13:55:34.900834 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:34.900470    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9" event={"ID":"9c07c02b-8105-4e90-9062-c65549925264","Type":"ContainerDied","Data":"105bb2e676835a6208b760767ad172b92b92226cbe53dfa759c5f3068ef49ff5"}
Jul 28 13:55:35.907259 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:35.907225    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9" event={"ID":"9c07c02b-8105-4e90-9062-c65549925264","Type":"ContainerStarted","Data":"adebf21e13d83ca5e322e3e8bc0f6c0c221c434b78e93efa2a83bc3254d90b3a"}
Jul 28 13:55:35.907640 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:35.907310    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9" event={"ID":"9c07c02b-8105-4e90-9062-c65549925264","Type":"ContainerStarted","Data":"b5981222755f9039d1ed6f1dc48f5612495c8bade1142bd246539ec0d5f98bac"}
Jul 28 13:55:35.907640 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:35.907619    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9"
Jul 28 13:55:35.907748 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:35.907689    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9"
Jul 28 13:55:35.909165 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:35.909140    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9" podUID="9c07c02b-8105-4e90-9062-c65549925264" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 28 13:55:35.939520 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:35.939463    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9" podStartSLOduration=2.171821018 podStartE2EDuration="5.939447402s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.767626384s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:55:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:55:35.935534219 +0000 UTC m=+3732.721409354" watchObservedRunningTime="2026-07-28 13:55:35.939447402 +0000 UTC m=+3732.725322516"
Jul 28 13:55:36.911485 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:36.911439    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9" podUID="9c07c02b-8105-4e90-9062-c65549925264" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 28 13:55:41.916348 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:41.916321    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9"
Jul 28 13:55:41.916811 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:41.916780    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9" podUID="9c07c02b-8105-4e90-9062-c65549925264" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 28 13:55:51.917332 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:55:51.917294    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9" podUID="9c07c02b-8105-4e90-9062-c65549925264" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 28 13:56:01.917694 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:56:01.917656    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9" podUID="9c07c02b-8105-4e90-9062-c65549925264" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 28 13:56:11.917096 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:56:11.917058    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9" podUID="9c07c02b-8105-4e90-9062-c65549925264" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 28 13:56:21.916834 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:56:21.916755    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9" podUID="9c07c02b-8105-4e90-9062-c65549925264" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 28 13:56:31.916907 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:56:31.916870    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9" podUID="9c07c02b-8105-4e90-9062-c65549925264" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 28 13:56:41.917334 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:56:41.917268    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9" podUID="9c07c02b-8105-4e90-9062-c65549925264" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 28 13:56:51.917694 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:56:51.917665    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9"
Jul 28 13:57:00.195434 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:00.195386    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9"]
Jul 28 13:57:00.196117 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:00.195605    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9" podUID="9c07c02b-8105-4e90-9062-c65549925264" containerName="kserve-container" containerID="cri-o://b5981222755f9039d1ed6f1dc48f5612495c8bade1142bd246539ec0d5f98bac" gracePeriod=30
Jul 28 13:57:00.196117 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:00.195679    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9" podUID="9c07c02b-8105-4e90-9062-c65549925264" containerName="kube-rbac-proxy" containerID="cri-o://adebf21e13d83ca5e322e3e8bc0f6c0c221c434b78e93efa2a83bc3254d90b3a" gracePeriod=30
Jul 28 13:57:01.265686 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:01.265649    2554 generic.go:356] "Generic (PLEG): container finished" podID="9c07c02b-8105-4e90-9062-c65549925264" containerID="adebf21e13d83ca5e322e3e8bc0f6c0c221c434b78e93efa2a83bc3254d90b3a" exitCode=2
Jul 28 13:57:01.266076 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:01.265723    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9" event={"ID":"9c07c02b-8105-4e90-9062-c65549925264","Type":"ContainerDied","Data":"adebf21e13d83ca5e322e3e8bc0f6c0c221c434b78e93efa2a83bc3254d90b3a"}
Jul 28 13:57:01.911723 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:01.911684    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9" podUID="9c07c02b-8105-4e90-9062-c65549925264" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.47:8643/healthz\": dial tcp 10.134.0.47:8643: connect: connection refused"
Jul 28 13:57:01.916980 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:01.916957    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9" podUID="9c07c02b-8105-4e90-9062-c65549925264" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 28 13:57:05.345325 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:05.345267    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv"]
Jul 28 13:57:05.349299 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:05.349258    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv"
Jul 28 13:57:05.351851 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:05.351830    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
Jul 28 13:57:05.351997 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:05.351857    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
Jul 28 13:57:05.359471 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:05.359409    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv"]
Jul 28 13:57:05.461874 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:05.461829    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ec35cd8-9a2c-441b-8dc7-2c495d1a250e-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv\" (UID: \"6ec35cd8-9a2c-441b-8dc7-2c495d1a250e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv"
Jul 28 13:57:05.461874 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:05.461876    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ec35cd8-9a2c-441b-8dc7-2c495d1a250e-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv\" (UID: \"6ec35cd8-9a2c-441b-8dc7-2c495d1a250e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv"
Jul 28 13:57:05.462087 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:05.461918    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp2mt\" (UniqueName: \"kubernetes.io/projected/6ec35cd8-9a2c-441b-8dc7-2c495d1a250e-kube-api-access-qp2mt\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv\" (UID: \"6ec35cd8-9a2c-441b-8dc7-2c495d1a250e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv"
Jul 28 13:57:05.462087 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:05.461972    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ec35cd8-9a2c-441b-8dc7-2c495d1a250e-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv\" (UID: \"6ec35cd8-9a2c-441b-8dc7-2c495d1a250e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv"
Jul 28 13:57:05.462087 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:05.461996    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ec35cd8-9a2c-441b-8dc7-2c495d1a250e-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv\" (UID: \"6ec35cd8-9a2c-441b-8dc7-2c495d1a250e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv"
Jul 28 13:57:05.563070 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:05.563024    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ec35cd8-9a2c-441b-8dc7-2c495d1a250e-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv\" (UID: \"6ec35cd8-9a2c-441b-8dc7-2c495d1a250e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv"
Jul 28 13:57:05.563319 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:05.563086    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ec35cd8-9a2c-441b-8dc7-2c495d1a250e-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv\" (UID: \"6ec35cd8-9a2c-441b-8dc7-2c495d1a250e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv"
Jul 28 13:57:05.563319 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:05.563110    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp2mt\" (UniqueName: \"kubernetes.io/projected/6ec35cd8-9a2c-441b-8dc7-2c495d1a250e-kube-api-access-qp2mt\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv\" (UID: \"6ec35cd8-9a2c-441b-8dc7-2c495d1a250e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv"
Jul 28 13:57:05.563319 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:05.563130    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ec35cd8-9a2c-441b-8dc7-2c495d1a250e-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv\" (UID: \"6ec35cd8-9a2c-441b-8dc7-2c495d1a250e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv"
Jul 28 13:57:05.563319 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:05.563153    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ec35cd8-9a2c-441b-8dc7-2c495d1a250e-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv\" (UID: \"6ec35cd8-9a2c-441b-8dc7-2c495d1a250e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv"
Jul 28 13:57:05.563319 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:57:05.563200    2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-serving-cert: secret "isvc-predictive-sklearn-v2-predictor-serving-cert" not found
Jul 28 13:57:05.563319 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:57:05.563306    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ec35cd8-9a2c-441b-8dc7-2c495d1a250e-proxy-tls podName:6ec35cd8-9a2c-441b-8dc7-2c495d1a250e nodeName:}" failed. No retries permitted until 2026-07-28 13:57:06.063252251 +0000 UTC m=+3822.849127341 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6ec35cd8-9a2c-441b-8dc7-2c495d1a250e-proxy-tls") pod "isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv" (UID: "6ec35cd8-9a2c-441b-8dc7-2c495d1a250e") : secret "isvc-predictive-sklearn-v2-predictor-serving-cert" not found
Jul 28 13:57:05.563714 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:05.563692    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ec35cd8-9a2c-441b-8dc7-2c495d1a250e-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv\" (UID: \"6ec35cd8-9a2c-441b-8dc7-2c495d1a250e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv"
Jul 28 13:57:05.563990 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:05.563969    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ec35cd8-9a2c-441b-8dc7-2c495d1a250e-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv\" (UID: \"6ec35cd8-9a2c-441b-8dc7-2c495d1a250e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv"
Jul 28 13:57:05.564032 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:05.563969    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ec35cd8-9a2c-441b-8dc7-2c495d1a250e-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv\" (UID: \"6ec35cd8-9a2c-441b-8dc7-2c495d1a250e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv"
Jul 28 13:57:05.572598 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:05.572567    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp2mt\" (UniqueName: \"kubernetes.io/projected/6ec35cd8-9a2c-441b-8dc7-2c495d1a250e-kube-api-access-qp2mt\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv\" (UID: \"6ec35cd8-9a2c-441b-8dc7-2c495d1a250e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv"
Jul 28 13:57:05.744463 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:05.744437    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9"
Jul 28 13:57:05.764663 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:05.764631    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c07c02b-8105-4e90-9062-c65549925264-kserve-provision-location\") pod \"9c07c02b-8105-4e90-9062-c65549925264\" (UID: \"9c07c02b-8105-4e90-9062-c65549925264\") "
Jul 28 13:57:05.764805 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:05.764669    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c07c02b-8105-4e90-9062-c65549925264-kube-rbac-proxy-sar-config\") pod \"9c07c02b-8105-4e90-9062-c65549925264\" (UID: \"9c07c02b-8105-4e90-9062-c65549925264\") "
Jul 28 13:57:05.764805 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:05.764729    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86kzk\" (UniqueName: \"kubernetes.io/projected/9c07c02b-8105-4e90-9062-c65549925264-kube-api-access-86kzk\") pod \"9c07c02b-8105-4e90-9062-c65549925264\" (UID: \"9c07c02b-8105-4e90-9062-c65549925264\") "
Jul 28 13:57:05.764805 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:05.764747    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c07c02b-8105-4e90-9062-c65549925264-cabundle-cert\") pod \"9c07c02b-8105-4e90-9062-c65549925264\" (UID: \"9c07c02b-8105-4e90-9062-c65549925264\") "
Jul 28 13:57:05.764805 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:05.764774    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c07c02b-8105-4e90-9062-c65549925264-proxy-tls\") pod \"9c07c02b-8105-4e90-9062-c65549925264\" (UID: \"9c07c02b-8105-4e90-9062-c65549925264\") "
Jul 28 13:57:05.765024 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:05.764979    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c07c02b-8105-4e90-9062-c65549925264-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9c07c02b-8105-4e90-9062-c65549925264" (UID: "9c07c02b-8105-4e90-9062-c65549925264"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:57:05.765077 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:05.765023    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c07c02b-8105-4e90-9062-c65549925264-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9c07c02b-8105-4e90-9062-c65549925264" (UID: "9c07c02b-8105-4e90-9062-c65549925264"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:57:05.765127 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:05.765111    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c07c02b-8105-4e90-9062-c65549925264-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9c07c02b-8105-4e90-9062-c65549925264" (UID: "9c07c02b-8105-4e90-9062-c65549925264"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:57:05.767614 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:05.767566    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c07c02b-8105-4e90-9062-c65549925264-kube-api-access-86kzk" (OuterVolumeSpecName: "kube-api-access-86kzk") pod "9c07c02b-8105-4e90-9062-c65549925264" (UID: "9c07c02b-8105-4e90-9062-c65549925264"). InnerVolumeSpecName "kube-api-access-86kzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:57:05.767794 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:05.767768    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c07c02b-8105-4e90-9062-c65549925264-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9c07c02b-8105-4e90-9062-c65549925264" (UID: "9c07c02b-8105-4e90-9062-c65549925264"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:57:05.865654 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:05.865571    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c07c02b-8105-4e90-9062-c65549925264-proxy-tls\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:57:05.865654 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:05.865599    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c07c02b-8105-4e90-9062-c65549925264-kserve-provision-location\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:57:05.865654 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:05.865609    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c07c02b-8105-4e90-9062-c65549925264-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:57:05.865654 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:05.865618    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86kzk\" (UniqueName: \"kubernetes.io/projected/9c07c02b-8105-4e90-9062-c65549925264-kube-api-access-86kzk\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:57:05.865654 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:05.865629    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c07c02b-8105-4e90-9062-c65549925264-cabundle-cert\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:57:06.067663 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:06.067614    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ec35cd8-9a2c-441b-8dc7-2c495d1a250e-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv\" (UID: \"6ec35cd8-9a2c-441b-8dc7-2c495d1a250e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv"
Jul 28 13:57:06.070130 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:06.070111    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ec35cd8-9a2c-441b-8dc7-2c495d1a250e-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv\" (UID: \"6ec35cd8-9a2c-441b-8dc7-2c495d1a250e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv"
Jul 28 13:57:06.264739 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:06.264701    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv"
Jul 28 13:57:06.289964 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:06.289940    2554 generic.go:356] "Generic (PLEG): container finished" podID="9c07c02b-8105-4e90-9062-c65549925264" containerID="b5981222755f9039d1ed6f1dc48f5612495c8bade1142bd246539ec0d5f98bac" exitCode=0
Jul 28 13:57:06.290092 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:06.289973    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9" event={"ID":"9c07c02b-8105-4e90-9062-c65549925264","Type":"ContainerDied","Data":"b5981222755f9039d1ed6f1dc48f5612495c8bade1142bd246539ec0d5f98bac"}
Jul 28 13:57:06.290092 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:06.290002    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9"
Jul 28 13:57:06.290092 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:06.290019    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9" event={"ID":"9c07c02b-8105-4e90-9062-c65549925264","Type":"ContainerDied","Data":"714d4af7a161735da5133d4debf567a0a51d1a3e5b4f2dc3675a5c84e79eddc1"}
Jul 28 13:57:06.290092 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:06.290049    2554 scope.go:117] "RemoveContainer" containerID="adebf21e13d83ca5e322e3e8bc0f6c0c221c434b78e93efa2a83bc3254d90b3a"
Jul 28 13:57:06.298714 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:06.298680    2554 scope.go:117] "RemoveContainer" containerID="b5981222755f9039d1ed6f1dc48f5612495c8bade1142bd246539ec0d5f98bac"
Jul 28 13:57:06.306913 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:06.306889    2554 scope.go:117] "RemoveContainer" containerID="105bb2e676835a6208b760767ad172b92b92226cbe53dfa759c5f3068ef49ff5"
Jul 28 13:57:06.320095 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:06.320025    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9"]
Jul 28 13:57:06.320472 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:06.320438    2554 scope.go:117] "RemoveContainer" containerID="adebf21e13d83ca5e322e3e8bc0f6c0c221c434b78e93efa2a83bc3254d90b3a"
Jul 28 13:57:06.321439 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:57:06.321410    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adebf21e13d83ca5e322e3e8bc0f6c0c221c434b78e93efa2a83bc3254d90b3a\": container with ID starting with adebf21e13d83ca5e322e3e8bc0f6c0c221c434b78e93efa2a83bc3254d90b3a not found: ID does not exist" containerID="adebf21e13d83ca5e322e3e8bc0f6c0c221c434b78e93efa2a83bc3254d90b3a"
Jul 28 13:57:06.321541 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:06.321452    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adebf21e13d83ca5e322e3e8bc0f6c0c221c434b78e93efa2a83bc3254d90b3a"} err="failed to get container status \"adebf21e13d83ca5e322e3e8bc0f6c0c221c434b78e93efa2a83bc3254d90b3a\": rpc error: code = NotFound desc = could not find container \"adebf21e13d83ca5e322e3e8bc0f6c0c221c434b78e93efa2a83bc3254d90b3a\": container with ID starting with adebf21e13d83ca5e322e3e8bc0f6c0c221c434b78e93efa2a83bc3254d90b3a not found: ID does not exist"
Jul 28 13:57:06.321541 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:06.321478    2554 scope.go:117] "RemoveContainer" containerID="b5981222755f9039d1ed6f1dc48f5612495c8bade1142bd246539ec0d5f98bac"
Jul 28 13:57:06.321882 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:57:06.321856    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5981222755f9039d1ed6f1dc48f5612495c8bade1142bd246539ec0d5f98bac\": container with ID starting with b5981222755f9039d1ed6f1dc48f5612495c8bade1142bd246539ec0d5f98bac not found: ID does not exist" containerID="b5981222755f9039d1ed6f1dc48f5612495c8bade1142bd246539ec0d5f98bac"
Jul 28 13:57:06.322078 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:06.322051    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5981222755f9039d1ed6f1dc48f5612495c8bade1142bd246539ec0d5f98bac"} err="failed to get container status \"b5981222755f9039d1ed6f1dc48f5612495c8bade1142bd246539ec0d5f98bac\": rpc error: code = NotFound desc = could not find container \"b5981222755f9039d1ed6f1dc48f5612495c8bade1142bd246539ec0d5f98bac\": container with ID starting with b5981222755f9039d1ed6f1dc48f5612495c8bade1142bd246539ec0d5f98bac not found: ID does not exist"
Jul 28 13:57:06.322185 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:06.322173    2554 scope.go:117] "RemoveContainer" containerID="105bb2e676835a6208b760767ad172b92b92226cbe53dfa759c5f3068ef49ff5"
Jul 28 13:57:06.322918 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:57:06.322894    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"105bb2e676835a6208b760767ad172b92b92226cbe53dfa759c5f3068ef49ff5\": container with ID starting with 105bb2e676835a6208b760767ad172b92b92226cbe53dfa759c5f3068ef49ff5 not found: ID does not exist" containerID="105bb2e676835a6208b760767ad172b92b92226cbe53dfa759c5f3068ef49ff5"
Jul 28 13:57:06.323065 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:06.323042    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"105bb2e676835a6208b760767ad172b92b92226cbe53dfa759c5f3068ef49ff5"} err="failed to get container status \"105bb2e676835a6208b760767ad172b92b92226cbe53dfa759c5f3068ef49ff5\": rpc error: code = NotFound desc = could not find container \"105bb2e676835a6208b760767ad172b92b92226cbe53dfa759c5f3068ef49ff5\": container with ID starting with 105bb2e676835a6208b760767ad172b92b92226cbe53dfa759c5f3068ef49ff5 not found: ID does not exist"
Jul 28 13:57:06.324191 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:06.324162    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-rwws9"]
Jul 28 13:57:06.401513 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:06.401485    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv"]
Jul 28 13:57:06.403622 ip-10-0-129-177 kubenswrapper[2554]: W0728 13:57:06.403593    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ec35cd8_9a2c_441b_8dc7_2c495d1a250e.slice/crio-5457085ca0f4390d429f13c84ca554584269fbacd2320a02cf3549a7f4f66a62 WatchSource:0}: Error finding container 5457085ca0f4390d429f13c84ca554584269fbacd2320a02cf3549a7f4f66a62: Status 404 returned error can't find the container with id 5457085ca0f4390d429f13c84ca554584269fbacd2320a02cf3549a7f4f66a62
Jul 28 13:57:07.295763 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:07.295720    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv" event={"ID":"6ec35cd8-9a2c-441b-8dc7-2c495d1a250e","Type":"ContainerStarted","Data":"4205b31decf1b72ef487bff4afcd8ccd9e1b196cab5de3b1f738444ad60da365"}
Jul 28 13:57:07.295968 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:07.295786    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv" event={"ID":"6ec35cd8-9a2c-441b-8dc7-2c495d1a250e","Type":"ContainerStarted","Data":"5457085ca0f4390d429f13c84ca554584269fbacd2320a02cf3549a7f4f66a62"}
Jul 28 13:57:07.802207 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:07.802176    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c07c02b-8105-4e90-9062-c65549925264" path="/var/lib/kubelet/pods/9c07c02b-8105-4e90-9062-c65549925264/volumes"
Jul 28 13:57:11.315090 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:11.315060    2554 generic.go:356] "Generic (PLEG): container finished" podID="6ec35cd8-9a2c-441b-8dc7-2c495d1a250e" containerID="4205b31decf1b72ef487bff4afcd8ccd9e1b196cab5de3b1f738444ad60da365" exitCode=0
Jul 28 13:57:11.315600 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:11.315139    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv" event={"ID":"6ec35cd8-9a2c-441b-8dc7-2c495d1a250e","Type":"ContainerDied","Data":"4205b31decf1b72ef487bff4afcd8ccd9e1b196cab5de3b1f738444ad60da365"}
Jul 28 13:57:12.321676 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:12.321642    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv" event={"ID":"6ec35cd8-9a2c-441b-8dc7-2c495d1a250e","Type":"ContainerStarted","Data":"efdd216217086641797cde5364de93b9e6dd88479f843998d0e278adec079092"}
Jul 28 13:57:12.322072 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:12.321710    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv" event={"ID":"6ec35cd8-9a2c-441b-8dc7-2c495d1a250e","Type":"ContainerStarted","Data":"d0dba2cb2e78b1c31febfcca31a827adf9426db72447c5ef3fc5f26228a95827"}
Jul 28 13:57:12.322072 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:12.321919    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv"
Jul 28 13:57:12.342515 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:12.342458    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv" podStartSLOduration=3.368980428 podStartE2EDuration="7.342439957s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.973459529s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:57:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:57:12.340768232 +0000 UTC m=+3829.126643341" watchObservedRunningTime="2026-07-28 13:57:12.342439957 +0000 UTC m=+3829.128315068"
Jul 28 13:57:13.326365 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:13.326336    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv"
Jul 28 13:57:19.335600 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:19.335573    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv"
Jul 28 13:57:49.337485 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:49.337395    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv" podUID="6ec35cd8-9a2c-441b-8dc7-2c495d1a250e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.48:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 28 13:57:59.336607 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:57:59.336570    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv" podUID="6ec35cd8-9a2c-441b-8dc7-2c495d1a250e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.48:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 28 13:58:09.336868 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:09.336824    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv" podUID="6ec35cd8-9a2c-441b-8dc7-2c495d1a250e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.48:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 28 13:58:19.336908 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:19.336872    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv" podUID="6ec35cd8-9a2c-441b-8dc7-2c495d1a250e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.48:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 28 13:58:29.394289 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:29.394234    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv"
Jul 28 13:58:36.393221 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:36.393186    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv"]
Jul 28 13:58:36.393694 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:36.393580    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv" podUID="6ec35cd8-9a2c-441b-8dc7-2c495d1a250e" containerName="kserve-container" containerID="cri-o://d0dba2cb2e78b1c31febfcca31a827adf9426db72447c5ef3fc5f26228a95827" gracePeriod=30
Jul 28 13:58:36.393694 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:36.393611    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv" podUID="6ec35cd8-9a2c-441b-8dc7-2c495d1a250e" containerName="kube-rbac-proxy" containerID="cri-o://efdd216217086641797cde5364de93b9e6dd88479f843998d0e278adec079092" gracePeriod=30
Jul 28 13:58:36.669968 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:36.669880    2554 generic.go:356] "Generic (PLEG): container finished" podID="6ec35cd8-9a2c-441b-8dc7-2c495d1a250e" containerID="efdd216217086641797cde5364de93b9e6dd88479f843998d0e278adec079092" exitCode=2
Jul 28 13:58:36.669968 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:36.669953    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv" event={"ID":"6ec35cd8-9a2c-441b-8dc7-2c495d1a250e","Type":"ContainerDied","Data":"efdd216217086641797cde5364de93b9e6dd88479f843998d0e278adec079092"}
Jul 28 13:58:39.331877 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:39.331838    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv" podUID="6ec35cd8-9a2c-441b-8dc7-2c495d1a250e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.48:8643/healthz\": dial tcp 10.134.0.48:8643: connect: connection refused"
Jul 28 13:58:39.336219 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:39.336196    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv" podUID="6ec35cd8-9a2c-441b-8dc7-2c495d1a250e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.48:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 28 13:58:40.944072 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:40.944050    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv"
Jul 28 13:58:41.056111 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.056040    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ec35cd8-9a2c-441b-8dc7-2c495d1a250e-cabundle-cert\") pod \"6ec35cd8-9a2c-441b-8dc7-2c495d1a250e\" (UID: \"6ec35cd8-9a2c-441b-8dc7-2c495d1a250e\") "
Jul 28 13:58:41.056111 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.056097    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ec35cd8-9a2c-441b-8dc7-2c495d1a250e-kube-rbac-proxy-sar-config\") pod \"6ec35cd8-9a2c-441b-8dc7-2c495d1a250e\" (UID: \"6ec35cd8-9a2c-441b-8dc7-2c495d1a250e\") "
Jul 28 13:58:41.056366 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.056134    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ec35cd8-9a2c-441b-8dc7-2c495d1a250e-proxy-tls\") pod \"6ec35cd8-9a2c-441b-8dc7-2c495d1a250e\" (UID: \"6ec35cd8-9a2c-441b-8dc7-2c495d1a250e\") "
Jul 28 13:58:41.056366 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.056150    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qp2mt\" (UniqueName: \"kubernetes.io/projected/6ec35cd8-9a2c-441b-8dc7-2c495d1a250e-kube-api-access-qp2mt\") pod \"6ec35cd8-9a2c-441b-8dc7-2c495d1a250e\" (UID: \"6ec35cd8-9a2c-441b-8dc7-2c495d1a250e\") "
Jul 28 13:58:41.056366 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.056309    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ec35cd8-9a2c-441b-8dc7-2c495d1a250e-kserve-provision-location\") pod \"6ec35cd8-9a2c-441b-8dc7-2c495d1a250e\" (UID: \"6ec35cd8-9a2c-441b-8dc7-2c495d1a250e\") "
Jul 28 13:58:41.056531 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.056473    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ec35cd8-9a2c-441b-8dc7-2c495d1a250e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6ec35cd8-9a2c-441b-8dc7-2c495d1a250e" (UID: "6ec35cd8-9a2c-441b-8dc7-2c495d1a250e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:58:41.056627 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.056606    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ec35cd8-9a2c-441b-8dc7-2c495d1a250e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6ec35cd8-9a2c-441b-8dc7-2c495d1a250e" (UID: "6ec35cd8-9a2c-441b-8dc7-2c495d1a250e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 13:58:41.056679 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.056623    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ec35cd8-9a2c-441b-8dc7-2c495d1a250e-cabundle-cert\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:58:41.056679 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.056615    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ec35cd8-9a2c-441b-8dc7-2c495d1a250e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6ec35cd8-9a2c-441b-8dc7-2c495d1a250e" (UID: "6ec35cd8-9a2c-441b-8dc7-2c495d1a250e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 13:58:41.058924 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.058906    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ec35cd8-9a2c-441b-8dc7-2c495d1a250e-kube-api-access-qp2mt" (OuterVolumeSpecName: "kube-api-access-qp2mt") pod "6ec35cd8-9a2c-441b-8dc7-2c495d1a250e" (UID: "6ec35cd8-9a2c-441b-8dc7-2c495d1a250e"). InnerVolumeSpecName "kube-api-access-qp2mt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 13:58:41.058924 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.058913    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ec35cd8-9a2c-441b-8dc7-2c495d1a250e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6ec35cd8-9a2c-441b-8dc7-2c495d1a250e" (UID: "6ec35cd8-9a2c-441b-8dc7-2c495d1a250e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 13:58:41.157164 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.157132    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ec35cd8-9a2c-441b-8dc7-2c495d1a250e-kserve-provision-location\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:58:41.157164 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.157157    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ec35cd8-9a2c-441b-8dc7-2c495d1a250e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:58:41.157164 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.157168    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ec35cd8-9a2c-441b-8dc7-2c495d1a250e-proxy-tls\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:58:41.157414 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.157177    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp2mt\" (UniqueName: \"kubernetes.io/projected/6ec35cd8-9a2c-441b-8dc7-2c495d1a250e-kube-api-access-qp2mt\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 13:58:41.541059 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.541020    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb"]
Jul 28 13:58:41.541372 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.541358    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ec35cd8-9a2c-441b-8dc7-2c495d1a250e" containerName="storage-initializer"
Jul 28 13:58:41.541430 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.541373    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ec35cd8-9a2c-441b-8dc7-2c495d1a250e" containerName="storage-initializer"
Jul 28 13:58:41.541430 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.541383    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c07c02b-8105-4e90-9062-c65549925264" containerName="storage-initializer"
Jul 28 13:58:41.541430 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.541389    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c07c02b-8105-4e90-9062-c65549925264" containerName="storage-initializer"
Jul 28 13:58:41.541430 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.541399    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ec35cd8-9a2c-441b-8dc7-2c495d1a250e" containerName="kserve-container"
Jul 28 13:58:41.541430 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.541405    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ec35cd8-9a2c-441b-8dc7-2c495d1a250e" containerName="kserve-container"
Jul 28 13:58:41.541430 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.541411    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ec35cd8-9a2c-441b-8dc7-2c495d1a250e" containerName="kube-rbac-proxy"
Jul 28 13:58:41.541430 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.541418    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ec35cd8-9a2c-441b-8dc7-2c495d1a250e" containerName="kube-rbac-proxy"
Jul 28 13:58:41.541430 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.541430    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c07c02b-8105-4e90-9062-c65549925264" containerName="kube-rbac-proxy"
Jul 28 13:58:41.541709 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.541435    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c07c02b-8105-4e90-9062-c65549925264" containerName="kube-rbac-proxy"
Jul 28 13:58:41.541709 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.541442    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c07c02b-8105-4e90-9062-c65549925264" containerName="kserve-container"
Jul 28 13:58:41.541709 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.541448    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c07c02b-8105-4e90-9062-c65549925264" containerName="kserve-container"
Jul 28 13:58:41.541709 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.541498    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c07c02b-8105-4e90-9062-c65549925264" containerName="kube-rbac-proxy"
Jul 28 13:58:41.541709 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.541505    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ec35cd8-9a2c-441b-8dc7-2c495d1a250e" containerName="kserve-container"
Jul 28 13:58:41.541709 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.541511    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ec35cd8-9a2c-441b-8dc7-2c495d1a250e" containerName="kube-rbac-proxy"
Jul 28 13:58:41.541709 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.541517    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c07c02b-8105-4e90-9062-c65549925264" containerName="kserve-container"
Jul 28 13:58:41.544858 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.544842    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb"
Jul 28 13:58:41.547193 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.547165    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
Jul 28 13:58:41.547338 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.547236    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
Jul 28 13:58:41.554160 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.554130    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb"]
Jul 28 13:58:41.660965 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.660928    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b404b4d6-1ff5-4146-808d-b5a2907fcb87-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb\" (UID: \"b404b4d6-1ff5-4146-808d-b5a2907fcb87\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb"
Jul 28 13:58:41.661155 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.660986    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd8vl\" (UniqueName: \"kubernetes.io/projected/b404b4d6-1ff5-4146-808d-b5a2907fcb87-kube-api-access-pd8vl\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb\" (UID: \"b404b4d6-1ff5-4146-808d-b5a2907fcb87\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb"
Jul 28 13:58:41.661155 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.661039    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b404b4d6-1ff5-4146-808d-b5a2907fcb87-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb\" (UID: \"b404b4d6-1ff5-4146-808d-b5a2907fcb87\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb"
Jul 28 13:58:41.661155 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.661059    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b404b4d6-1ff5-4146-808d-b5a2907fcb87-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb\" (UID: \"b404b4d6-1ff5-4146-808d-b5a2907fcb87\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb"
Jul 28 13:58:41.661155 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.661135    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b404b4d6-1ff5-4146-808d-b5a2907fcb87-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb\" (UID: \"b404b4d6-1ff5-4146-808d-b5a2907fcb87\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb"
Jul 28 13:58:41.694682 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.694646    2554 generic.go:356] "Generic (PLEG): container finished" podID="6ec35cd8-9a2c-441b-8dc7-2c495d1a250e" containerID="d0dba2cb2e78b1c31febfcca31a827adf9426db72447c5ef3fc5f26228a95827" exitCode=0
Jul 28 13:58:41.694852 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.694705    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv" event={"ID":"6ec35cd8-9a2c-441b-8dc7-2c495d1a250e","Type":"ContainerDied","Data":"d0dba2cb2e78b1c31febfcca31a827adf9426db72447c5ef3fc5f26228a95827"}
Jul 28 13:58:41.694852 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.694721    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv"
Jul 28 13:58:41.694852 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.694773    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv" event={"ID":"6ec35cd8-9a2c-441b-8dc7-2c495d1a250e","Type":"ContainerDied","Data":"5457085ca0f4390d429f13c84ca554584269fbacd2320a02cf3549a7f4f66a62"}
Jul 28 13:58:41.694852 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.694800    2554 scope.go:117] "RemoveContainer" containerID="efdd216217086641797cde5364de93b9e6dd88479f843998d0e278adec079092"
Jul 28 13:58:41.708170 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.708150    2554 scope.go:117] "RemoveContainer" containerID="d0dba2cb2e78b1c31febfcca31a827adf9426db72447c5ef3fc5f26228a95827"
Jul 28 13:58:41.716367 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.716349    2554 scope.go:117] "RemoveContainer" containerID="4205b31decf1b72ef487bff4afcd8ccd9e1b196cab5de3b1f738444ad60da365"
Jul 28 13:58:41.722725 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.722700    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv"]
Jul 28 13:58:41.725301 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.725259    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-j5nrv"]
Jul 28 13:58:41.725645 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.725620    2554 scope.go:117] "RemoveContainer" containerID="efdd216217086641797cde5364de93b9e6dd88479f843998d0e278adec079092"
Jul 28 13:58:41.725935 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:58:41.725915    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efdd216217086641797cde5364de93b9e6dd88479f843998d0e278adec079092\": container with ID starting with efdd216217086641797cde5364de93b9e6dd88479f843998d0e278adec079092 not found: ID does not exist" containerID="efdd216217086641797cde5364de93b9e6dd88479f843998d0e278adec079092"
Jul 28 13:58:41.725992 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.725945    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efdd216217086641797cde5364de93b9e6dd88479f843998d0e278adec079092"} err="failed to get container status \"efdd216217086641797cde5364de93b9e6dd88479f843998d0e278adec079092\": rpc error: code = NotFound desc = could not find container \"efdd216217086641797cde5364de93b9e6dd88479f843998d0e278adec079092\": container with ID starting with efdd216217086641797cde5364de93b9e6dd88479f843998d0e278adec079092 not found: ID does not exist"
Jul 28 13:58:41.725992 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.725963    2554 scope.go:117] "RemoveContainer" containerID="d0dba2cb2e78b1c31febfcca31a827adf9426db72447c5ef3fc5f26228a95827"
Jul 28 13:58:41.726245 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:58:41.726230    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0dba2cb2e78b1c31febfcca31a827adf9426db72447c5ef3fc5f26228a95827\": container with ID starting with d0dba2cb2e78b1c31febfcca31a827adf9426db72447c5ef3fc5f26228a95827 not found: ID does not exist" containerID="d0dba2cb2e78b1c31febfcca31a827adf9426db72447c5ef3fc5f26228a95827"
Jul 28 13:58:41.726324 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.726257    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0dba2cb2e78b1c31febfcca31a827adf9426db72447c5ef3fc5f26228a95827"} err="failed to get container status \"d0dba2cb2e78b1c31febfcca31a827adf9426db72447c5ef3fc5f26228a95827\": rpc error: code = NotFound desc = could not find container \"d0dba2cb2e78b1c31febfcca31a827adf9426db72447c5ef3fc5f26228a95827\": container with ID starting with d0dba2cb2e78b1c31febfcca31a827adf9426db72447c5ef3fc5f26228a95827 not found: ID does not exist"
Jul 28 13:58:41.726324 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.726287    2554 scope.go:117] "RemoveContainer" containerID="4205b31decf1b72ef487bff4afcd8ccd9e1b196cab5de3b1f738444ad60da365"
Jul 28 13:58:41.726499 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:58:41.726484    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4205b31decf1b72ef487bff4afcd8ccd9e1b196cab5de3b1f738444ad60da365\": container with ID starting with 4205b31decf1b72ef487bff4afcd8ccd9e1b196cab5de3b1f738444ad60da365 not found: ID does not exist" containerID="4205b31decf1b72ef487bff4afcd8ccd9e1b196cab5de3b1f738444ad60da365"
Jul 28 13:58:41.726560 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.726502    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4205b31decf1b72ef487bff4afcd8ccd9e1b196cab5de3b1f738444ad60da365"} err="failed to get container status \"4205b31decf1b72ef487bff4afcd8ccd9e1b196cab5de3b1f738444ad60da365\": rpc error: code = NotFound desc = could not find container \"4205b31decf1b72ef487bff4afcd8ccd9e1b196cab5de3b1f738444ad60da365\": container with ID starting with 4205b31decf1b72ef487bff4afcd8ccd9e1b196cab5de3b1f738444ad60da365 not found: ID does not exist"
Jul 28 13:58:41.762062 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.762028    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b404b4d6-1ff5-4146-808d-b5a2907fcb87-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb\" (UID: \"b404b4d6-1ff5-4146-808d-b5a2907fcb87\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb"
Jul 28 13:58:41.762187 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.762071    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b404b4d6-1ff5-4146-808d-b5a2907fcb87-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb\" (UID: \"b404b4d6-1ff5-4146-808d-b5a2907fcb87\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb"
Jul 28 13:58:41.762187 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.762138    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b404b4d6-1ff5-4146-808d-b5a2907fcb87-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb\" (UID: \"b404b4d6-1ff5-4146-808d-b5a2907fcb87\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb"
Jul 28 13:58:41.762327 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.762187    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd8vl\" (UniqueName: \"kubernetes.io/projected/b404b4d6-1ff5-4146-808d-b5a2907fcb87-kube-api-access-pd8vl\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb\" (UID: \"b404b4d6-1ff5-4146-808d-b5a2907fcb87\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb"
Jul 28 13:58:41.762327 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.762228    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b404b4d6-1ff5-4146-808d-b5a2907fcb87-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb\" (UID: \"b404b4d6-1ff5-4146-808d-b5a2907fcb87\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb"
Jul 28 13:58:41.762445 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:58:41.762336    2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-serving-cert: secret "isvc-predictive-xgboost-v2-predictor-serving-cert" not found
Jul 28 13:58:41.762445 ip-10-0-129-177 kubenswrapper[2554]: E0728 13:58:41.762409    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b404b4d6-1ff5-4146-808d-b5a2907fcb87-proxy-tls podName:b404b4d6-1ff5-4146-808d-b5a2907fcb87 nodeName:}" failed. No retries permitted until 2026-07-28 13:58:42.262391104 +0000 UTC m=+3919.048266193 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b404b4d6-1ff5-4146-808d-b5a2907fcb87-proxy-tls") pod "isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb" (UID: "b404b4d6-1ff5-4146-808d-b5a2907fcb87") : secret "isvc-predictive-xgboost-v2-predictor-serving-cert" not found
Jul 28 13:58:41.762605 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.762585    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b404b4d6-1ff5-4146-808d-b5a2907fcb87-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb\" (UID: \"b404b4d6-1ff5-4146-808d-b5a2907fcb87\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb"
Jul 28 13:58:41.762788 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.762772    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b404b4d6-1ff5-4146-808d-b5a2907fcb87-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb\" (UID: \"b404b4d6-1ff5-4146-808d-b5a2907fcb87\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb"
Jul 28 13:58:41.762850 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.762807    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b404b4d6-1ff5-4146-808d-b5a2907fcb87-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb\" (UID: \"b404b4d6-1ff5-4146-808d-b5a2907fcb87\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb"
Jul 28 13:58:41.773254 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.773228    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd8vl\" (UniqueName: \"kubernetes.io/projected/b404b4d6-1ff5-4146-808d-b5a2907fcb87-kube-api-access-pd8vl\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb\" (UID: \"b404b4d6-1ff5-4146-808d-b5a2907fcb87\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb"
Jul 28 13:58:41.801367 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:41.801304    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ec35cd8-9a2c-441b-8dc7-2c495d1a250e" path="/var/lib/kubelet/pods/6ec35cd8-9a2c-441b-8dc7-2c495d1a250e/volumes"
Jul 28 13:58:42.265480 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:42.265445    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b404b4d6-1ff5-4146-808d-b5a2907fcb87-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb\" (UID: \"b404b4d6-1ff5-4146-808d-b5a2907fcb87\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb"
Jul 28 13:58:42.268137 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:42.268102    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b404b4d6-1ff5-4146-808d-b5a2907fcb87-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb\" (UID: \"b404b4d6-1ff5-4146-808d-b5a2907fcb87\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb"
Jul 28 13:58:42.457070 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:42.457027    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb"
Jul 28 13:58:42.595612 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:42.595505    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb"]
Jul 28 13:58:42.597857 ip-10-0-129-177 kubenswrapper[2554]: W0728 13:58:42.597832    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb404b4d6_1ff5_4146_808d_b5a2907fcb87.slice/crio-ae5800a42414647ce3190d738923983e68a396b803893b8003b644ced1dd6902 WatchSource:0}: Error finding container ae5800a42414647ce3190d738923983e68a396b803893b8003b644ced1dd6902: Status 404 returned error can't find the container with id ae5800a42414647ce3190d738923983e68a396b803893b8003b644ced1dd6902
Jul 28 13:58:42.701356 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:42.701314    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb" event={"ID":"b404b4d6-1ff5-4146-808d-b5a2907fcb87","Type":"ContainerStarted","Data":"e98cb2420b5c5b4338d7bf32eb9233f88e01dd5a21d0bf568970433b64ec1ef1"}
Jul 28 13:58:42.701526 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:42.701403    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb" event={"ID":"b404b4d6-1ff5-4146-808d-b5a2907fcb87","Type":"ContainerStarted","Data":"ae5800a42414647ce3190d738923983e68a396b803893b8003b644ced1dd6902"}
Jul 28 13:58:46.719923 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:46.719837    2554 generic.go:356] "Generic (PLEG): container finished" podID="b404b4d6-1ff5-4146-808d-b5a2907fcb87" containerID="e98cb2420b5c5b4338d7bf32eb9233f88e01dd5a21d0bf568970433b64ec1ef1" exitCode=0
Jul 28 13:58:46.720372 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:46.719915    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb" event={"ID":"b404b4d6-1ff5-4146-808d-b5a2907fcb87","Type":"ContainerDied","Data":"e98cb2420b5c5b4338d7bf32eb9233f88e01dd5a21d0bf568970433b64ec1ef1"}
Jul 28 13:58:47.727185 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:47.727143    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb" event={"ID":"b404b4d6-1ff5-4146-808d-b5a2907fcb87","Type":"ContainerStarted","Data":"26df5913cad2e503a34080f3f935697cc40a6d59f49225b00b2cffc6b68f2185"}
Jul 28 13:58:47.727595 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:47.727231    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb" event={"ID":"b404b4d6-1ff5-4146-808d-b5a2907fcb87","Type":"ContainerStarted","Data":"d4d737be9a2d12f298d1060cdf7d3d801ec8a6121245c47e560feac1fcdd74f5"}
Jul 28 13:58:47.727595 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:47.727426    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb"
Jul 28 13:58:47.727595 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:47.727471    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb"
Jul 28 13:58:47.748319 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:47.748259    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb" podStartSLOduration=2.960107845 podStartE2EDuration="6.7482464s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.788138555s" isStatefulPod=false podCreationTimestamp="2026-07-28 13:58:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 13:58:47.74635039 +0000 UTC m=+3924.532225500" watchObservedRunningTime="2026-07-28 13:58:47.7482464 +0000 UTC m=+3924.534121511"
Jul 28 13:58:53.737791 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:58:53.737757    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb"
Jul 28 13:59:23.739480 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:59:23.739403    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb" podUID="b404b4d6-1ff5-4146-808d-b5a2907fcb87" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.49:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 28 13:59:33.738568 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:59:33.738532    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb" podUID="b404b4d6-1ff5-4146-808d-b5a2907fcb87" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.49:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 28 13:59:43.739039 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:59:43.738998    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb" podUID="b404b4d6-1ff5-4146-808d-b5a2907fcb87" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.49:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 28 13:59:53.739205 ip-10-0-129-177 kubenswrapper[2554]: I0728 13:59:53.739168    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb" podUID="b404b4d6-1ff5-4146-808d-b5a2907fcb87" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.49:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 28 14:00:03.793418 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:03.793382    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb"
Jul 28 14:00:11.682145 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:11.682106    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb"]
Jul 28 14:00:11.682637 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:11.682377    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb" podUID="b404b4d6-1ff5-4146-808d-b5a2907fcb87" containerName="kserve-container" containerID="cri-o://d4d737be9a2d12f298d1060cdf7d3d801ec8a6121245c47e560feac1fcdd74f5" gracePeriod=30
Jul 28 14:00:11.682637 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:11.682420    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb" podUID="b404b4d6-1ff5-4146-808d-b5a2907fcb87" containerName="kube-rbac-proxy" containerID="cri-o://26df5913cad2e503a34080f3f935697cc40a6d59f49225b00b2cffc6b68f2185" gracePeriod=30
Jul 28 14:00:12.073317 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:12.073259    2554 generic.go:356] "Generic (PLEG): container finished" podID="b404b4d6-1ff5-4146-808d-b5a2907fcb87" containerID="26df5913cad2e503a34080f3f935697cc40a6d59f49225b00b2cffc6b68f2185" exitCode=2
Jul 28 14:00:12.073485 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:12.073337    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb" event={"ID":"b404b4d6-1ff5-4146-808d-b5a2907fcb87","Type":"ContainerDied","Data":"26df5913cad2e503a34080f3f935697cc40a6d59f49225b00b2cffc6b68f2185"}
Jul 28 14:00:13.732246 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:13.732202    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb" podUID="b404b4d6-1ff5-4146-808d-b5a2907fcb87" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.49:8643/healthz\": dial tcp 10.134.0.49:8643: connect: connection refused"
Jul 28 14:00:13.738928 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:13.738902    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb" podUID="b404b4d6-1ff5-4146-808d-b5a2907fcb87" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.49:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 28 14:00:16.092261 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:16.092227    2554 generic.go:356] "Generic (PLEG): container finished" podID="b404b4d6-1ff5-4146-808d-b5a2907fcb87" containerID="d4d737be9a2d12f298d1060cdf7d3d801ec8a6121245c47e560feac1fcdd74f5" exitCode=0
Jul 28 14:00:16.092648 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:16.092306    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb" event={"ID":"b404b4d6-1ff5-4146-808d-b5a2907fcb87","Type":"ContainerDied","Data":"d4d737be9a2d12f298d1060cdf7d3d801ec8a6121245c47e560feac1fcdd74f5"}
Jul 28 14:00:16.127747 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:16.127722    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb"
Jul 28 14:00:16.223107 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:16.223079    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b404b4d6-1ff5-4146-808d-b5a2907fcb87-kserve-provision-location\") pod \"b404b4d6-1ff5-4146-808d-b5a2907fcb87\" (UID: \"b404b4d6-1ff5-4146-808d-b5a2907fcb87\") "
Jul 28 14:00:16.223294 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:16.223124    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b404b4d6-1ff5-4146-808d-b5a2907fcb87-kube-rbac-proxy-sar-config\") pod \"b404b4d6-1ff5-4146-808d-b5a2907fcb87\" (UID: \"b404b4d6-1ff5-4146-808d-b5a2907fcb87\") "
Jul 28 14:00:16.223294 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:16.223157    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b404b4d6-1ff5-4146-808d-b5a2907fcb87-proxy-tls\") pod \"b404b4d6-1ff5-4146-808d-b5a2907fcb87\" (UID: \"b404b4d6-1ff5-4146-808d-b5a2907fcb87\") "
Jul 28 14:00:16.223294 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:16.223185    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pd8vl\" (UniqueName: \"kubernetes.io/projected/b404b4d6-1ff5-4146-808d-b5a2907fcb87-kube-api-access-pd8vl\") pod \"b404b4d6-1ff5-4146-808d-b5a2907fcb87\" (UID: \"b404b4d6-1ff5-4146-808d-b5a2907fcb87\") "
Jul 28 14:00:16.223294 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:16.223215    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b404b4d6-1ff5-4146-808d-b5a2907fcb87-cabundle-cert\") pod \"b404b4d6-1ff5-4146-808d-b5a2907fcb87\" (UID: \"b404b4d6-1ff5-4146-808d-b5a2907fcb87\") "
Jul 28 14:00:16.223519 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:16.223403    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b404b4d6-1ff5-4146-808d-b5a2907fcb87-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b404b4d6-1ff5-4146-808d-b5a2907fcb87" (UID: "b404b4d6-1ff5-4146-808d-b5a2907fcb87"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:00:16.223583 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:16.223529    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b404b4d6-1ff5-4146-808d-b5a2907fcb87-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b404b4d6-1ff5-4146-808d-b5a2907fcb87" (UID: "b404b4d6-1ff5-4146-808d-b5a2907fcb87"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:00:16.223583 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:16.223534    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b404b4d6-1ff5-4146-808d-b5a2907fcb87-kserve-provision-location\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:00:16.223746 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:16.223641    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b404b4d6-1ff5-4146-808d-b5a2907fcb87-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b404b4d6-1ff5-4146-808d-b5a2907fcb87" (UID: "b404b4d6-1ff5-4146-808d-b5a2907fcb87"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:00:16.225490 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:16.225467    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b404b4d6-1ff5-4146-808d-b5a2907fcb87-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b404b4d6-1ff5-4146-808d-b5a2907fcb87" (UID: "b404b4d6-1ff5-4146-808d-b5a2907fcb87"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:00:16.225579 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:16.225503    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b404b4d6-1ff5-4146-808d-b5a2907fcb87-kube-api-access-pd8vl" (OuterVolumeSpecName: "kube-api-access-pd8vl") pod "b404b4d6-1ff5-4146-808d-b5a2907fcb87" (UID: "b404b4d6-1ff5-4146-808d-b5a2907fcb87"). InnerVolumeSpecName "kube-api-access-pd8vl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:00:16.324527 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:16.324493    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b404b4d6-1ff5-4146-808d-b5a2907fcb87-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:00:16.324527 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:16.324520    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b404b4d6-1ff5-4146-808d-b5a2907fcb87-proxy-tls\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:00:16.324527 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:16.324529    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd8vl\" (UniqueName: \"kubernetes.io/projected/b404b4d6-1ff5-4146-808d-b5a2907fcb87-kube-api-access-pd8vl\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:00:16.324527 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:16.324538    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b404b4d6-1ff5-4146-808d-b5a2907fcb87-cabundle-cert\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:00:17.099052 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:17.099019    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb" event={"ID":"b404b4d6-1ff5-4146-808d-b5a2907fcb87","Type":"ContainerDied","Data":"ae5800a42414647ce3190d738923983e68a396b803893b8003b644ced1dd6902"}
Jul 28 14:00:17.099052 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:17.099038    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb"
Jul 28 14:00:17.099577 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:17.099085    2554 scope.go:117] "RemoveContainer" containerID="26df5913cad2e503a34080f3f935697cc40a6d59f49225b00b2cffc6b68f2185"
Jul 28 14:00:17.108309 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:17.108290    2554 scope.go:117] "RemoveContainer" containerID="d4d737be9a2d12f298d1060cdf7d3d801ec8a6121245c47e560feac1fcdd74f5"
Jul 28 14:00:17.116195 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:17.116178    2554 scope.go:117] "RemoveContainer" containerID="e98cb2420b5c5b4338d7bf32eb9233f88e01dd5a21d0bf568970433b64ec1ef1"
Jul 28 14:00:17.122253 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:17.122222    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb"]
Jul 28 14:00:17.127350 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:17.127329    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gzrhb"]
Jul 28 14:00:17.802731 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:17.802701    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b404b4d6-1ff5-4146-808d-b5a2907fcb87" path="/var/lib/kubelet/pods/b404b4d6-1ff5-4146-808d-b5a2907fcb87/volumes"
Jul 28 14:00:21.920953 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:21.920920    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg"]
Jul 28 14:00:21.921449 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:21.921409    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b404b4d6-1ff5-4146-808d-b5a2907fcb87" containerName="kube-rbac-proxy"
Jul 28 14:00:21.921449 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:21.921428    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="b404b4d6-1ff5-4146-808d-b5a2907fcb87" containerName="kube-rbac-proxy"
Jul 28 14:00:21.921572 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:21.921456    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b404b4d6-1ff5-4146-808d-b5a2907fcb87" containerName="storage-initializer"
Jul 28 14:00:21.921572 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:21.921465    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="b404b4d6-1ff5-4146-808d-b5a2907fcb87" containerName="storage-initializer"
Jul 28 14:00:21.921572 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:21.921488    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b404b4d6-1ff5-4146-808d-b5a2907fcb87" containerName="kserve-container"
Jul 28 14:00:21.921572 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:21.921497    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="b404b4d6-1ff5-4146-808d-b5a2907fcb87" containerName="kserve-container"
Jul 28 14:00:21.921572 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:21.921568    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="b404b4d6-1ff5-4146-808d-b5a2907fcb87" containerName="kube-rbac-proxy"
Jul 28 14:00:21.921828 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:21.921582    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="b404b4d6-1ff5-4146-808d-b5a2907fcb87" containerName="kserve-container"
Jul 28 14:00:21.926474 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:21.926450    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg"
Jul 28 14:00:21.929466 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:21.929439    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
Jul 28 14:00:21.929602 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:21.929498    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
Jul 28 14:00:21.930011 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:21.929750    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 14:00:21.930011 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:21.929765    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 14:00:21.930011 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:21.929833    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 14:00:21.930538 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:21.930520    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rxfwd\""
Jul 28 14:00:21.930645 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:21.930566    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 14:00:21.937772 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:21.937752    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg"]
Jul 28 14:00:21.968862 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:21.968835    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a481e92-31f8-4610-b635-9bc985a80ec7-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg\" (UID: \"8a481e92-31f8-4610-b635-9bc985a80ec7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg"
Jul 28 14:00:21.968980 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:21.968864    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a481e92-31f8-4610-b635-9bc985a80ec7-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg\" (UID: \"8a481e92-31f8-4610-b635-9bc985a80ec7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg"
Jul 28 14:00:21.968980 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:21.968887    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a481e92-31f8-4610-b635-9bc985a80ec7-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg\" (UID: \"8a481e92-31f8-4610-b635-9bc985a80ec7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg"
Jul 28 14:00:21.968980 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:21.968923    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m4x2\" (UniqueName: \"kubernetes.io/projected/8a481e92-31f8-4610-b635-9bc985a80ec7-kube-api-access-2m4x2\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg\" (UID: \"8a481e92-31f8-4610-b635-9bc985a80ec7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg"
Jul 28 14:00:21.969103 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:21.969018    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a481e92-31f8-4610-b635-9bc985a80ec7-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg\" (UID: \"8a481e92-31f8-4610-b635-9bc985a80ec7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg"
Jul 28 14:00:22.070292 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:22.070252    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a481e92-31f8-4610-b635-9bc985a80ec7-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg\" (UID: \"8a481e92-31f8-4610-b635-9bc985a80ec7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg"
Jul 28 14:00:22.070457 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:22.070337    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a481e92-31f8-4610-b635-9bc985a80ec7-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg\" (UID: \"8a481e92-31f8-4610-b635-9bc985a80ec7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg"
Jul 28 14:00:22.070457 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:22.070368    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a481e92-31f8-4610-b635-9bc985a80ec7-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg\" (UID: \"8a481e92-31f8-4610-b635-9bc985a80ec7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg"
Jul 28 14:00:22.070457 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:22.070402    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a481e92-31f8-4610-b635-9bc985a80ec7-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg\" (UID: \"8a481e92-31f8-4610-b635-9bc985a80ec7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg"
Jul 28 14:00:22.070457 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:22.070446    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m4x2\" (UniqueName: \"kubernetes.io/projected/8a481e92-31f8-4610-b635-9bc985a80ec7-kube-api-access-2m4x2\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg\" (UID: \"8a481e92-31f8-4610-b635-9bc985a80ec7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg"
Jul 28 14:00:22.070694 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:00:22.070484    2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-serving-cert: secret "isvc-predictive-lightgbm-v2-predictor-serving-cert" not found
Jul 28 14:00:22.070694 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:00:22.070558    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a481e92-31f8-4610-b635-9bc985a80ec7-proxy-tls podName:8a481e92-31f8-4610-b635-9bc985a80ec7 nodeName:}" failed. No retries permitted until 2026-07-28 14:00:22.570536312 +0000 UTC m=+4019.356411401 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8a481e92-31f8-4610-b635-9bc985a80ec7-proxy-tls") pod "isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg" (UID: "8a481e92-31f8-4610-b635-9bc985a80ec7") : secret "isvc-predictive-lightgbm-v2-predictor-serving-cert" not found
Jul 28 14:00:22.070825 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:22.070808    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a481e92-31f8-4610-b635-9bc985a80ec7-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg\" (UID: \"8a481e92-31f8-4610-b635-9bc985a80ec7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg"
Jul 28 14:00:22.070938 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:22.070921    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a481e92-31f8-4610-b635-9bc985a80ec7-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg\" (UID: \"8a481e92-31f8-4610-b635-9bc985a80ec7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg"
Jul 28 14:00:22.070980 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:22.070957    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a481e92-31f8-4610-b635-9bc985a80ec7-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg\" (UID: \"8a481e92-31f8-4610-b635-9bc985a80ec7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg"
Jul 28 14:00:22.080888 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:22.080866    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m4x2\" (UniqueName: \"kubernetes.io/projected/8a481e92-31f8-4610-b635-9bc985a80ec7-kube-api-access-2m4x2\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg\" (UID: \"8a481e92-31f8-4610-b635-9bc985a80ec7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg"
Jul 28 14:00:22.574952 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:22.574915    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a481e92-31f8-4610-b635-9bc985a80ec7-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg\" (UID: \"8a481e92-31f8-4610-b635-9bc985a80ec7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg"
Jul 28 14:00:22.577468 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:22.577445    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a481e92-31f8-4610-b635-9bc985a80ec7-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg\" (UID: \"8a481e92-31f8-4610-b635-9bc985a80ec7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg"
Jul 28 14:00:22.837055 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:22.836972    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg"
Jul 28 14:00:22.969361 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:22.969334    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg"]
Jul 28 14:00:22.971790 ip-10-0-129-177 kubenswrapper[2554]: W0728 14:00:22.971761    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a481e92_31f8_4610_b635_9bc985a80ec7.slice/crio-55031f96e201c3aecdee90e600e3e6d51dfb8eb6a320dde69df7604cb09a4f03 WatchSource:0}: Error finding container 55031f96e201c3aecdee90e600e3e6d51dfb8eb6a320dde69df7604cb09a4f03: Status 404 returned error can't find the container with id 55031f96e201c3aecdee90e600e3e6d51dfb8eb6a320dde69df7604cb09a4f03
Jul 28 14:00:23.128094 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:23.128013    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg" event={"ID":"8a481e92-31f8-4610-b635-9bc985a80ec7","Type":"ContainerStarted","Data":"9939c7d2cc9e6c0bb59a4d7d25dd48b724173160ef204c6eb5d0dff59e691b3b"}
Jul 28 14:00:23.128094 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:23.128084    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg" event={"ID":"8a481e92-31f8-4610-b635-9bc985a80ec7","Type":"ContainerStarted","Data":"55031f96e201c3aecdee90e600e3e6d51dfb8eb6a320dde69df7604cb09a4f03"}
Jul 28 14:00:27.148474 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:27.148386    2554 generic.go:356] "Generic (PLEG): container finished" podID="8a481e92-31f8-4610-b635-9bc985a80ec7" containerID="9939c7d2cc9e6c0bb59a4d7d25dd48b724173160ef204c6eb5d0dff59e691b3b" exitCode=0
Jul 28 14:00:27.148830 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:27.148472    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg" event={"ID":"8a481e92-31f8-4610-b635-9bc985a80ec7","Type":"ContainerDied","Data":"9939c7d2cc9e6c0bb59a4d7d25dd48b724173160ef204c6eb5d0dff59e691b3b"}
Jul 28 14:00:28.154830 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:28.154795    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg" event={"ID":"8a481e92-31f8-4610-b635-9bc985a80ec7","Type":"ContainerStarted","Data":"8bc78c0a880ae95a3b68e0fd0ac04a3856f5643375b9f15b9c66a4694f42ee78"}
Jul 28 14:00:28.155260 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:28.154856    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg" event={"ID":"8a481e92-31f8-4610-b635-9bc985a80ec7","Type":"ContainerStarted","Data":"016e5b9e1de7a3f82f680c8ca5ab56d07721d98d0afbd54248917e2f2d1b3094"}
Jul 28 14:00:28.155260 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:28.155068    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg"
Jul 28 14:00:28.155260 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:28.155242    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg"
Jul 28 14:00:28.177315 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:28.177264    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg" podStartSLOduration=3.476867181 podStartE2EDuration="7.177250564s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.700383383s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:00:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:00:28.175183759 +0000 UTC m=+4024.961058871" watchObservedRunningTime="2026-07-28 14:00:28.177250564 +0000 UTC m=+4024.963125675"
Jul 28 14:00:34.164420 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:00:34.164389    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg"
Jul 28 14:01:04.165287 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:01:04.165233    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg" podUID="8a481e92-31f8-4610-b635-9bc985a80ec7" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.50:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 28 14:01:14.165629 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:01:14.165543    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg" podUID="8a481e92-31f8-4610-b635-9bc985a80ec7" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.50:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 28 14:01:24.165903 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:01:24.165858    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg" podUID="8a481e92-31f8-4610-b635-9bc985a80ec7" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.50:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 28 14:01:34.165549 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:01:34.165512    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg" podUID="8a481e92-31f8-4610-b635-9bc985a80ec7" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.50:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 28 14:01:44.192382 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:01:44.192344    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg"
Jul 28 14:01:51.989383 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:01:51.989351    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg"]
Jul 28 14:01:51.989904 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:01:51.989568    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg" podUID="8a481e92-31f8-4610-b635-9bc985a80ec7" containerName="kserve-container" containerID="cri-o://016e5b9e1de7a3f82f680c8ca5ab56d07721d98d0afbd54248917e2f2d1b3094" gracePeriod=30
Jul 28 14:01:51.989904 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:01:51.989629    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg" podUID="8a481e92-31f8-4610-b635-9bc985a80ec7" containerName="kube-rbac-proxy" containerID="cri-o://8bc78c0a880ae95a3b68e0fd0ac04a3856f5643375b9f15b9c66a4694f42ee78" gracePeriod=30
Jul 28 14:01:52.486743 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:01:52.486710    2554 generic.go:356] "Generic (PLEG): container finished" podID="8a481e92-31f8-4610-b635-9bc985a80ec7" containerID="8bc78c0a880ae95a3b68e0fd0ac04a3856f5643375b9f15b9c66a4694f42ee78" exitCode=2
Jul 28 14:01:52.486884 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:01:52.486777    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg" event={"ID":"8a481e92-31f8-4610-b635-9bc985a80ec7","Type":"ContainerDied","Data":"8bc78c0a880ae95a3b68e0fd0ac04a3856f5643375b9f15b9c66a4694f42ee78"}
Jul 28 14:01:54.159775 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:01:54.159728    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg" podUID="8a481e92-31f8-4610-b635-9bc985a80ec7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.50:8643/healthz\": dial tcp 10.134.0.50:8643: connect: connection refused"
Jul 28 14:01:54.165165 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:01:54.165137    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg" podUID="8a481e92-31f8-4610-b635-9bc985a80ec7" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.50:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 28 14:01:57.510184 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:01:57.510142    2554 generic.go:356] "Generic (PLEG): container finished" podID="8a481e92-31f8-4610-b635-9bc985a80ec7" containerID="016e5b9e1de7a3f82f680c8ca5ab56d07721d98d0afbd54248917e2f2d1b3094" exitCode=0
Jul 28 14:01:57.510550 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:01:57.510206    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg" event={"ID":"8a481e92-31f8-4610-b635-9bc985a80ec7","Type":"ContainerDied","Data":"016e5b9e1de7a3f82f680c8ca5ab56d07721d98d0afbd54248917e2f2d1b3094"}
Jul 28 14:01:57.533298 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:01:57.533260    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg"
Jul 28 14:01:57.571881 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:01:57.571855    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2m4x2\" (UniqueName: \"kubernetes.io/projected/8a481e92-31f8-4610-b635-9bc985a80ec7-kube-api-access-2m4x2\") pod \"8a481e92-31f8-4610-b635-9bc985a80ec7\" (UID: \"8a481e92-31f8-4610-b635-9bc985a80ec7\") "
Jul 28 14:01:57.572004 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:01:57.571894    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a481e92-31f8-4610-b635-9bc985a80ec7-proxy-tls\") pod \"8a481e92-31f8-4610-b635-9bc985a80ec7\" (UID: \"8a481e92-31f8-4610-b635-9bc985a80ec7\") "
Jul 28 14:01:57.572004 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:01:57.571942    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a481e92-31f8-4610-b635-9bc985a80ec7-kube-rbac-proxy-sar-config\") pod \"8a481e92-31f8-4610-b635-9bc985a80ec7\" (UID: \"8a481e92-31f8-4610-b635-9bc985a80ec7\") "
Jul 28 14:01:57.572004 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:01:57.571964    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a481e92-31f8-4610-b635-9bc985a80ec7-kserve-provision-location\") pod \"8a481e92-31f8-4610-b635-9bc985a80ec7\" (UID: \"8a481e92-31f8-4610-b635-9bc985a80ec7\") "
Jul 28 14:01:57.572190 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:01:57.572016    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a481e92-31f8-4610-b635-9bc985a80ec7-cabundle-cert\") pod \"8a481e92-31f8-4610-b635-9bc985a80ec7\" (UID: \"8a481e92-31f8-4610-b635-9bc985a80ec7\") "
Jul 28 14:01:57.572379 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:01:57.572342    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a481e92-31f8-4610-b635-9bc985a80ec7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8a481e92-31f8-4610-b635-9bc985a80ec7" (UID: "8a481e92-31f8-4610-b635-9bc985a80ec7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:01:57.572452 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:01:57.572353    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a481e92-31f8-4610-b635-9bc985a80ec7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8a481e92-31f8-4610-b635-9bc985a80ec7" (UID: "8a481e92-31f8-4610-b635-9bc985a80ec7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:01:57.572500 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:01:57.572467    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a481e92-31f8-4610-b635-9bc985a80ec7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8a481e92-31f8-4610-b635-9bc985a80ec7" (UID: "8a481e92-31f8-4610-b635-9bc985a80ec7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:01:57.574101 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:01:57.574079    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a481e92-31f8-4610-b635-9bc985a80ec7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8a481e92-31f8-4610-b635-9bc985a80ec7" (UID: "8a481e92-31f8-4610-b635-9bc985a80ec7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:01:57.574178 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:01:57.574135    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a481e92-31f8-4610-b635-9bc985a80ec7-kube-api-access-2m4x2" (OuterVolumeSpecName: "kube-api-access-2m4x2") pod "8a481e92-31f8-4610-b635-9bc985a80ec7" (UID: "8a481e92-31f8-4610-b635-9bc985a80ec7"). InnerVolumeSpecName "kube-api-access-2m4x2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:01:57.672860 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:01:57.672780    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a481e92-31f8-4610-b635-9bc985a80ec7-cabundle-cert\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:01:57.672860 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:01:57.672809    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m4x2\" (UniqueName: \"kubernetes.io/projected/8a481e92-31f8-4610-b635-9bc985a80ec7-kube-api-access-2m4x2\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:01:57.672860 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:01:57.672820    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a481e92-31f8-4610-b635-9bc985a80ec7-proxy-tls\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:01:57.672860 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:01:57.672828    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a481e92-31f8-4610-b635-9bc985a80ec7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:01:57.672860 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:01:57.672837    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a481e92-31f8-4610-b635-9bc985a80ec7-kserve-provision-location\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:01:58.516082 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:01:58.516051    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg" event={"ID":"8a481e92-31f8-4610-b635-9bc985a80ec7","Type":"ContainerDied","Data":"55031f96e201c3aecdee90e600e3e6d51dfb8eb6a320dde69df7604cb09a4f03"}
Jul 28 14:01:58.516082 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:01:58.516070    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg"
Jul 28 14:01:58.516606 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:01:58.516118    2554 scope.go:117] "RemoveContainer" containerID="8bc78c0a880ae95a3b68e0fd0ac04a3856f5643375b9f15b9c66a4694f42ee78"
Jul 28 14:01:58.524423 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:01:58.524404    2554 scope.go:117] "RemoveContainer" containerID="016e5b9e1de7a3f82f680c8ca5ab56d07721d98d0afbd54248917e2f2d1b3094"
Jul 28 14:01:58.531627 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:01:58.531607    2554 scope.go:117] "RemoveContainer" containerID="9939c7d2cc9e6c0bb59a4d7d25dd48b724173160ef204c6eb5d0dff59e691b3b"
Jul 28 14:01:58.536478 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:01:58.536453    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg"]
Jul 28 14:01:58.541542 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:01:58.541521    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-fwbtg"]
Jul 28 14:01:59.801326 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:01:59.801294    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a481e92-31f8-4610-b635-9bc985a80ec7" path="/var/lib/kubelet/pods/8a481e92-31f8-4610-b635-9bc985a80ec7/volumes"
Jul 28 14:02:04.354164 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:02:04.354120    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-5mb6r"]
Jul 28 14:02:04.354636 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:02:04.354590    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a481e92-31f8-4610-b635-9bc985a80ec7" containerName="storage-initializer"
Jul 28 14:02:04.354636 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:02:04.354605    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a481e92-31f8-4610-b635-9bc985a80ec7" containerName="storage-initializer"
Jul 28 14:02:04.354636 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:02:04.354619    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a481e92-31f8-4610-b635-9bc985a80ec7" containerName="kube-rbac-proxy"
Jul 28 14:02:04.354636 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:02:04.354627    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a481e92-31f8-4610-b635-9bc985a80ec7" containerName="kube-rbac-proxy"
Jul 28 14:02:04.354844 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:02:04.354641    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a481e92-31f8-4610-b635-9bc985a80ec7" containerName="kserve-container"
Jul 28 14:02:04.354844 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:02:04.354651    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a481e92-31f8-4610-b635-9bc985a80ec7" containerName="kserve-container"
Jul 28 14:02:04.354844 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:02:04.354746    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a481e92-31f8-4610-b635-9bc985a80ec7" containerName="kube-rbac-proxy"
Jul 28 14:02:04.354844 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:02:04.354759    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a481e92-31f8-4610-b635-9bc985a80ec7" containerName="kserve-container"
Jul 28 14:02:04.359820 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:02:04.359795    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-5mb6r"
Jul 28 14:02:04.362739 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:02:04.362710    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
Jul 28 14:02:04.362868 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:02:04.362749    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 14:02:04.362868 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:02:04.362775    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 14:02:04.362868 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:02:04.362749    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 14:02:04.362868 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:02:04.362858    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 14:02:04.363068 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:02:04.363015    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
Jul 28 14:02:04.363068 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:02:04.363037    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rxfwd\""
Jul 28 14:02:04.368450 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:02:04.368428    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-5mb6r"]
Jul 28 14:02:04.422840 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:02:04.422801    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91-proxy-tls\") pod \"isvc-sklearn-predictor-5f4d89cc44-5mb6r\" (UID: \"b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-5mb6r"
Jul 28 14:02:04.422840 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:02:04.422845    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91-kserve-provision-location\") pod \"isvc-sklearn-predictor-5f4d89cc44-5mb6r\" (UID: \"b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-5mb6r"
Jul 28 14:02:04.423103 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:02:04.422929    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-5f4d89cc44-5mb6r\" (UID: \"b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-5mb6r"
Jul 28 14:02:04.423103 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:02:04.423027    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljjxd\" (UniqueName: \"kubernetes.io/projected/b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91-kube-api-access-ljjxd\") pod \"isvc-sklearn-predictor-5f4d89cc44-5mb6r\" (UID: \"b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-5mb6r"
Jul 28 14:02:04.423103 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:02:04.423066    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91-cabundle-cert\") pod \"isvc-sklearn-predictor-5f4d89cc44-5mb6r\" (UID: \"b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-5mb6r"
Jul 28 14:02:04.523732 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:02:04.523676    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-5f4d89cc44-5mb6r\" (UID: \"b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-5mb6r"
Jul 28 14:02:04.523927 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:02:04.523763    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljjxd\" (UniqueName: \"kubernetes.io/projected/b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91-kube-api-access-ljjxd\") pod \"isvc-sklearn-predictor-5f4d89cc44-5mb6r\" (UID: \"b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-5mb6r"
Jul 28 14:02:04.523927 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:02:04.523797    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91-cabundle-cert\") pod \"isvc-sklearn-predictor-5f4d89cc44-5mb6r\" (UID: \"b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-5mb6r"
Jul 28 14:02:04.523927 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:02:04.523849    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91-proxy-tls\") pod \"isvc-sklearn-predictor-5f4d89cc44-5mb6r\" (UID: \"b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-5mb6r"
Jul 28 14:02:04.523927 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:02:04.523871    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91-kserve-provision-location\") pod \"isvc-sklearn-predictor-5f4d89cc44-5mb6r\" (UID: \"b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-5mb6r"
Jul 28 14:02:04.524245 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:02:04.524224    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91-kserve-provision-location\") pod \"isvc-sklearn-predictor-5f4d89cc44-5mb6r\" (UID: \"b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-5mb6r"
Jul 28 14:02:04.524475 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:02:04.524453    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-5f4d89cc44-5mb6r\" (UID: \"b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-5mb6r"
Jul 28 14:02:04.524582 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:02:04.524560    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91-cabundle-cert\") pod \"isvc-sklearn-predictor-5f4d89cc44-5mb6r\" (UID: \"b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-5mb6r"
Jul 28 14:02:04.526677 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:02:04.526655    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91-proxy-tls\") pod \"isvc-sklearn-predictor-5f4d89cc44-5mb6r\" (UID: \"b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-5mb6r"
Jul 28 14:02:04.532548 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:02:04.532526    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljjxd\" (UniqueName: \"kubernetes.io/projected/b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91-kube-api-access-ljjxd\") pod \"isvc-sklearn-predictor-5f4d89cc44-5mb6r\" (UID: \"b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-5mb6r"
Jul 28 14:02:04.672859 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:02:04.672769    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-5mb6r"
Jul 28 14:02:04.811906 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:02:04.811881    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-5mb6r"]
Jul 28 14:02:04.814031 ip-10-0-129-177 kubenswrapper[2554]: W0728 14:02:04.814002    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb87af0ff_1f2c_42d9_bb5e_48d32ad0fb91.slice/crio-375c6024dc2e53d1c84183e79f0f52f2320da8a5660a6b82c105c70201794779 WatchSource:0}: Error finding container 375c6024dc2e53d1c84183e79f0f52f2320da8a5660a6b82c105c70201794779: Status 404 returned error can't find the container with id 375c6024dc2e53d1c84183e79f0f52f2320da8a5660a6b82c105c70201794779
Jul 28 14:02:05.545356 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:02:05.545320    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-5mb6r" event={"ID":"b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91","Type":"ContainerStarted","Data":"e0c7dc19ad9242558592b4c2612b0218bf99918a3a369622c6175e2159181879"}
Jul 28 14:02:05.545716 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:02:05.545382    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-5mb6r" event={"ID":"b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91","Type":"ContainerStarted","Data":"375c6024dc2e53d1c84183e79f0f52f2320da8a5660a6b82c105c70201794779"}
Jul 28 14:02:08.559058 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:02:08.559019    2554 generic.go:356] "Generic (PLEG): container finished" podID="b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91" containerID="e0c7dc19ad9242558592b4c2612b0218bf99918a3a369622c6175e2159181879" exitCode=0
Jul 28 14:02:08.559476 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:02:08.559098    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-5mb6r" event={"ID":"b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91","Type":"ContainerDied","Data":"e0c7dc19ad9242558592b4c2612b0218bf99918a3a369622c6175e2159181879"}
Jul 28 14:02:09.565496 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:02:09.565455    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-5mb6r" event={"ID":"b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91","Type":"ContainerStarted","Data":"d92cd083af8cb77b183773f4610f4bcebb49f868c9d59d4f11be309fb3454882"}
Jul 28 14:02:09.565906 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:02:09.565523    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-5mb6r" event={"ID":"b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91","Type":"ContainerStarted","Data":"0869ff81b68ea3e8824dc705660f85e6d3107cb4c29490aa70193d74b0a61307"}
Jul 28 14:02:09.565906 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:02:09.565821    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-5mb6r"
Jul 28 14:02:09.589931 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:02:09.589868    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-5mb6r" podStartSLOduration=2.092708764 podStartE2EDuration="5.5898484s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.497139636s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:02:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:02:09.588132832 +0000 UTC m=+4126.374007961" watchObservedRunningTime="2026-07-28 14:02:09.5898484 +0000 UTC m=+4126.375723513"
Jul 28 14:02:10.570119 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:02:10.570085    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-5mb6r"
Jul 28 14:02:10.571482 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:02:10.571456    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-5mb6r" podUID="b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 28 14:02:11.574535 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:02:11.574495    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-5mb6r" podUID="b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 28 14:02:16.579244 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:02:16.579161    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-5mb6r"
Jul 28 14:02:16.579736 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:02:16.579708    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-5mb6r" podUID="b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 28 14:02:26.579924 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:02:26.579887    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-5mb6r" podUID="b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 28 14:02:36.580492 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:02:36.580453    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-5mb6r" podUID="b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 28 14:02:46.579840 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:02:46.579804    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-5mb6r" podUID="b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 28 14:02:56.580338 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:02:56.580297    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-5mb6r" podUID="b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 28 14:03:06.580543 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:06.580501    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-5mb6r" podUID="b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 28 14:03:16.579630 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:16.579590    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-5mb6r" podUID="b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 28 14:03:17.797347 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:17.797308    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-5mb6r" podUID="b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 28 14:03:27.801934 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:27.801908    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-5mb6r"
Jul 28 14:03:34.582550 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:34.582514    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-5mb6r"]
Jul 28 14:03:34.585041 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:34.582769    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-5mb6r" podUID="b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91" containerName="kserve-container" containerID="cri-o://0869ff81b68ea3e8824dc705660f85e6d3107cb4c29490aa70193d74b0a61307" gracePeriod=30
Jul 28 14:03:34.585041 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:34.582810    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-5mb6r" podUID="b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91" containerName="kube-rbac-proxy" containerID="cri-o://d92cd083af8cb77b183773f4610f4bcebb49f868c9d59d4f11be309fb3454882" gracePeriod=30
Jul 28 14:03:34.903319 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:34.903213    2554 generic.go:356] "Generic (PLEG): container finished" podID="b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91" containerID="d92cd083af8cb77b183773f4610f4bcebb49f868c9d59d4f11be309fb3454882" exitCode=2
Jul 28 14:03:34.903319 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:34.903251    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-5mb6r" event={"ID":"b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91","Type":"ContainerDied","Data":"d92cd083af8cb77b183773f4610f4bcebb49f868c9d59d4f11be309fb3454882"}
Jul 28 14:03:36.574925 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:36.574886    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-5mb6r" podUID="b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.51:8643/healthz\": dial tcp 10.134.0.51:8643: connect: connection refused"
Jul 28 14:03:37.797839 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:37.797799    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-5mb6r" podUID="b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 28 14:03:39.235617 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:39.235592    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-5mb6r"
Jul 28 14:03:39.420243 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:39.420144    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljjxd\" (UniqueName: \"kubernetes.io/projected/b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91-kube-api-access-ljjxd\") pod \"b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91\" (UID: \"b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91\") "
Jul 28 14:03:39.420243 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:39.420190    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91-cabundle-cert\") pod \"b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91\" (UID: \"b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91\") "
Jul 28 14:03:39.420243 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:39.420231    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91-kserve-provision-location\") pod \"b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91\" (UID: \"b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91\") "
Jul 28 14:03:39.420243 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:39.420247    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91-kube-rbac-proxy-sar-config\") pod \"b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91\" (UID: \"b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91\") "
Jul 28 14:03:39.420610 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:39.420270    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91-proxy-tls\") pod \"b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91\" (UID: \"b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91\") "
Jul 28 14:03:39.420610 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:39.420594    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91" (UID: "b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:03:39.420708 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:39.420621    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91" (UID: "b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:03:39.420708 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:39.420668    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91" (UID: "b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:03:39.422540 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:39.422519    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91-kube-api-access-ljjxd" (OuterVolumeSpecName: "kube-api-access-ljjxd") pod "b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91" (UID: "b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91"). InnerVolumeSpecName "kube-api-access-ljjxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:03:39.422639 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:39.422543    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91" (UID: "b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:03:39.521743 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:39.521708    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91-kserve-provision-location\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:03:39.521743 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:39.521735    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:03:39.521743 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:39.521747    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91-proxy-tls\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:03:39.521955 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:39.521757    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljjxd\" (UniqueName: \"kubernetes.io/projected/b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91-kube-api-access-ljjxd\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:03:39.521955 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:39.521766    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91-cabundle-cert\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:03:39.887894 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:39.887863    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6br5b"]
Jul 28 14:03:39.888199 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:39.888186    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91" containerName="storage-initializer"
Jul 28 14:03:39.888238 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:39.888204    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91" containerName="storage-initializer"
Jul 28 14:03:39.888238 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:39.888215    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91" containerName="kube-rbac-proxy"
Jul 28 14:03:39.888238 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:39.888220    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91" containerName="kube-rbac-proxy"
Jul 28 14:03:39.888238 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:39.888233    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91" containerName="kserve-container"
Jul 28 14:03:39.888392 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:39.888240    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91" containerName="kserve-container"
Jul 28 14:03:39.888392 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:39.888311    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91" containerName="kserve-container"
Jul 28 14:03:39.888392 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:39.888323    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91" containerName="kube-rbac-proxy"
Jul 28 14:03:39.891744 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:39.891725    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6br5b"
Jul 28 14:03:39.897364 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:39.897341    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
Jul 28 14:03:39.897461 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:39.897360    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
Jul 28 14:03:39.927840 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:39.925579    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6br5b"]
Jul 28 14:03:39.930465 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:39.930434    2554 generic.go:356] "Generic (PLEG): container finished" podID="b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91" containerID="0869ff81b68ea3e8824dc705660f85e6d3107cb4c29490aa70193d74b0a61307" exitCode=0
Jul 28 14:03:39.930602 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:39.930508    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-5mb6r"
Jul 28 14:03:39.930602 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:39.930522    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-5mb6r" event={"ID":"b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91","Type":"ContainerDied","Data":"0869ff81b68ea3e8824dc705660f85e6d3107cb4c29490aa70193d74b0a61307"}
Jul 28 14:03:39.930602 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:39.930588    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-5mb6r" event={"ID":"b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91","Type":"ContainerDied","Data":"375c6024dc2e53d1c84183e79f0f52f2320da8a5660a6b82c105c70201794779"}
Jul 28 14:03:39.930772 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:39.930625    2554 scope.go:117] "RemoveContainer" containerID="d92cd083af8cb77b183773f4610f4bcebb49f868c9d59d4f11be309fb3454882"
Jul 28 14:03:39.939213 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:39.939195    2554 scope.go:117] "RemoveContainer" containerID="0869ff81b68ea3e8824dc705660f85e6d3107cb4c29490aa70193d74b0a61307"
Jul 28 14:03:39.947198 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:39.947168    2554 scope.go:117] "RemoveContainer" containerID="e0c7dc19ad9242558592b4c2612b0218bf99918a3a369622c6175e2159181879"
Jul 28 14:03:39.955002 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:39.954980    2554 scope.go:117] "RemoveContainer" containerID="d92cd083af8cb77b183773f4610f4bcebb49f868c9d59d4f11be309fb3454882"
Jul 28 14:03:39.955329 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:03:39.955265    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d92cd083af8cb77b183773f4610f4bcebb49f868c9d59d4f11be309fb3454882\": container with ID starting with d92cd083af8cb77b183773f4610f4bcebb49f868c9d59d4f11be309fb3454882 not found: ID does not exist" containerID="d92cd083af8cb77b183773f4610f4bcebb49f868c9d59d4f11be309fb3454882"
Jul 28 14:03:39.955403 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:39.955339    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d92cd083af8cb77b183773f4610f4bcebb49f868c9d59d4f11be309fb3454882"} err="failed to get container status \"d92cd083af8cb77b183773f4610f4bcebb49f868c9d59d4f11be309fb3454882\": rpc error: code = NotFound desc = could not find container \"d92cd083af8cb77b183773f4610f4bcebb49f868c9d59d4f11be309fb3454882\": container with ID starting with d92cd083af8cb77b183773f4610f4bcebb49f868c9d59d4f11be309fb3454882 not found: ID does not exist"
Jul 28 14:03:39.955403 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:39.955358    2554 scope.go:117] "RemoveContainer" containerID="0869ff81b68ea3e8824dc705660f85e6d3107cb4c29490aa70193d74b0a61307"
Jul 28 14:03:39.955619 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:03:39.955604    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0869ff81b68ea3e8824dc705660f85e6d3107cb4c29490aa70193d74b0a61307\": container with ID starting with 0869ff81b68ea3e8824dc705660f85e6d3107cb4c29490aa70193d74b0a61307 not found: ID does not exist" containerID="0869ff81b68ea3e8824dc705660f85e6d3107cb4c29490aa70193d74b0a61307"
Jul 28 14:03:39.955671 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:39.955622    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0869ff81b68ea3e8824dc705660f85e6d3107cb4c29490aa70193d74b0a61307"} err="failed to get container status \"0869ff81b68ea3e8824dc705660f85e6d3107cb4c29490aa70193d74b0a61307\": rpc error: code = NotFound desc = could not find container \"0869ff81b68ea3e8824dc705660f85e6d3107cb4c29490aa70193d74b0a61307\": container with ID starting with 0869ff81b68ea3e8824dc705660f85e6d3107cb4c29490aa70193d74b0a61307 not found: ID does not exist"
Jul 28 14:03:39.955671 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:39.955634    2554 scope.go:117] "RemoveContainer" containerID="e0c7dc19ad9242558592b4c2612b0218bf99918a3a369622c6175e2159181879"
Jul 28 14:03:39.955890 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:03:39.955871    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0c7dc19ad9242558592b4c2612b0218bf99918a3a369622c6175e2159181879\": container with ID starting with e0c7dc19ad9242558592b4c2612b0218bf99918a3a369622c6175e2159181879 not found: ID does not exist" containerID="e0c7dc19ad9242558592b4c2612b0218bf99918a3a369622c6175e2159181879"
Jul 28 14:03:39.955934 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:39.955895    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0c7dc19ad9242558592b4c2612b0218bf99918a3a369622c6175e2159181879"} err="failed to get container status \"e0c7dc19ad9242558592b4c2612b0218bf99918a3a369622c6175e2159181879\": rpc error: code = NotFound desc = could not find container \"e0c7dc19ad9242558592b4c2612b0218bf99918a3a369622c6175e2159181879\": container with ID starting with e0c7dc19ad9242558592b4c2612b0218bf99918a3a369622c6175e2159181879 not found: ID does not exist"
Jul 28 14:03:39.976353 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:39.976329    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-5mb6r"]
Jul 28 14:03:39.988873 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:39.988840    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-5mb6r"]
Jul 28 14:03:40.026591 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:40.026557    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8e658c5-2624-4b4b-8476-89ffc2b11119-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-6br5b\" (UID: \"f8e658c5-2624-4b4b-8476-89ffc2b11119\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6br5b"
Jul 28 14:03:40.026715 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:40.026616    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8m8d\" (UniqueName: \"kubernetes.io/projected/f8e658c5-2624-4b4b-8476-89ffc2b11119-kube-api-access-j8m8d\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-6br5b\" (UID: \"f8e658c5-2624-4b4b-8476-89ffc2b11119\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6br5b"
Jul 28 14:03:40.026715 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:40.026642    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f8e658c5-2624-4b4b-8476-89ffc2b11119-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-6br5b\" (UID: \"f8e658c5-2624-4b4b-8476-89ffc2b11119\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6br5b"
Jul 28 14:03:40.026715 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:40.026691    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8e658c5-2624-4b4b-8476-89ffc2b11119-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-6br5b\" (UID: \"f8e658c5-2624-4b4b-8476-89ffc2b11119\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6br5b"
Jul 28 14:03:40.026845 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:40.026753    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8e658c5-2624-4b4b-8476-89ffc2b11119-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-6br5b\" (UID: \"f8e658c5-2624-4b4b-8476-89ffc2b11119\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6br5b"
Jul 28 14:03:40.127577 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:40.127497    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8e658c5-2624-4b4b-8476-89ffc2b11119-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-6br5b\" (UID: \"f8e658c5-2624-4b4b-8476-89ffc2b11119\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6br5b"
Jul 28 14:03:40.127577 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:40.127563    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8e658c5-2624-4b4b-8476-89ffc2b11119-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-6br5b\" (UID: \"f8e658c5-2624-4b4b-8476-89ffc2b11119\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6br5b"
Jul 28 14:03:40.127786 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:40.127614    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8m8d\" (UniqueName: \"kubernetes.io/projected/f8e658c5-2624-4b4b-8476-89ffc2b11119-kube-api-access-j8m8d\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-6br5b\" (UID: \"f8e658c5-2624-4b4b-8476-89ffc2b11119\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6br5b"
Jul 28 14:03:40.127786 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:40.127637    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f8e658c5-2624-4b4b-8476-89ffc2b11119-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-6br5b\" (UID: \"f8e658c5-2624-4b4b-8476-89ffc2b11119\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6br5b"
Jul 28 14:03:40.127786 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:40.127696    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8e658c5-2624-4b4b-8476-89ffc2b11119-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-6br5b\" (UID: \"f8e658c5-2624-4b4b-8476-89ffc2b11119\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6br5b"
Jul 28 14:03:40.128106 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:40.128085    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8e658c5-2624-4b4b-8476-89ffc2b11119-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-6br5b\" (UID: \"f8e658c5-2624-4b4b-8476-89ffc2b11119\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6br5b"
Jul 28 14:03:40.128388 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:40.128360    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8e658c5-2624-4b4b-8476-89ffc2b11119-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-6br5b\" (UID: \"f8e658c5-2624-4b4b-8476-89ffc2b11119\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6br5b"
Jul 28 14:03:40.128496 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:40.128360    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f8e658c5-2624-4b4b-8476-89ffc2b11119-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-6br5b\" (UID: \"f8e658c5-2624-4b4b-8476-89ffc2b11119\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6br5b"
Jul 28 14:03:40.130313 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:40.130291    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8e658c5-2624-4b4b-8476-89ffc2b11119-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-6br5b\" (UID: \"f8e658c5-2624-4b4b-8476-89ffc2b11119\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6br5b"
Jul 28 14:03:40.142033 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:40.142007    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8m8d\" (UniqueName: \"kubernetes.io/projected/f8e658c5-2624-4b4b-8476-89ffc2b11119-kube-api-access-j8m8d\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-6br5b\" (UID: \"f8e658c5-2624-4b4b-8476-89ffc2b11119\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6br5b"
Jul 28 14:03:40.201567 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:40.201540    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6br5b"
Jul 28 14:03:40.340130 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:40.340100    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6br5b"]
Jul 28 14:03:40.343044 ip-10-0-129-177 kubenswrapper[2554]: W0728 14:03:40.343018    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8e658c5_2624_4b4b_8476_89ffc2b11119.slice/crio-9e80aefaff0adc99c9abe5ef083bdd11c5f964b45d809903495dd7c97f8c21e1 WatchSource:0}: Error finding container 9e80aefaff0adc99c9abe5ef083bdd11c5f964b45d809903495dd7c97f8c21e1: Status 404 returned error can't find the container with id 9e80aefaff0adc99c9abe5ef083bdd11c5f964b45d809903495dd7c97f8c21e1
Jul 28 14:03:40.937813 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:40.937781    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6br5b" event={"ID":"f8e658c5-2624-4b4b-8476-89ffc2b11119","Type":"ContainerStarted","Data":"0a2effed15cbc86fc160a0541100d25d4bc363544c1978564a7cd1be9788c837"}
Jul 28 14:03:40.937987 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:40.937832    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6br5b" event={"ID":"f8e658c5-2624-4b4b-8476-89ffc2b11119","Type":"ContainerStarted","Data":"9e80aefaff0adc99c9abe5ef083bdd11c5f964b45d809903495dd7c97f8c21e1"}
Jul 28 14:03:41.802468 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:41.802436    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91" path="/var/lib/kubelet/pods/b87af0ff-1f2c-42d9-bb5e-48d32ad0fb91/volumes"
Jul 28 14:03:44.956334 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:44.956251    2554 generic.go:356] "Generic (PLEG): container finished" podID="f8e658c5-2624-4b4b-8476-89ffc2b11119" containerID="0a2effed15cbc86fc160a0541100d25d4bc363544c1978564a7cd1be9788c837" exitCode=0
Jul 28 14:03:44.956334 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:44.956325    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6br5b" event={"ID":"f8e658c5-2624-4b4b-8476-89ffc2b11119","Type":"ContainerDied","Data":"0a2effed15cbc86fc160a0541100d25d4bc363544c1978564a7cd1be9788c837"}
Jul 28 14:03:45.962905 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:45.962869    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6br5b" event={"ID":"f8e658c5-2624-4b4b-8476-89ffc2b11119","Type":"ContainerStarted","Data":"8b0908d0516170c96ae01fcc23a9aa8f01c4ed7eb09012efb2bbc2045555e27a"}
Jul 28 14:03:45.963255 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:45.962932    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6br5b" event={"ID":"f8e658c5-2624-4b4b-8476-89ffc2b11119","Type":"ContainerStarted","Data":"33945f64e8fec1f01dc5e5b3ca398df0df6e7889986069c3451d9812dcfef8d1"}
Jul 28 14:03:45.963255 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:45.963133    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6br5b"
Jul 28 14:03:45.963374 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:45.963307    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6br5b"
Jul 28 14:03:45.987016 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:45.986968    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6br5b" podStartSLOduration=3.080843643 podStartE2EDuration="6.986954099s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.906110456s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:03:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:03:45.984747109 +0000 UTC m=+4222.770622220" watchObservedRunningTime="2026-07-28 14:03:45.986954099 +0000 UTC m=+4222.772829278"
Jul 28 14:03:51.973616 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:03:51.973589    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6br5b"
Jul 28 14:04:21.990739 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:04:21.990697    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6br5b" podUID="f8e658c5-2624-4b4b-8476-89ffc2b11119" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Jul 28 14:04:31.976682 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:04:31.976648    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6br5b"
Jul 28 14:04:39.860846 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:04:39.860812    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6br5b"]
Jul 28 14:04:39.861240 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:04:39.861107    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6br5b" podUID="f8e658c5-2624-4b4b-8476-89ffc2b11119" containerName="kserve-container" containerID="cri-o://33945f64e8fec1f01dc5e5b3ca398df0df6e7889986069c3451d9812dcfef8d1" gracePeriod=30
Jul 28 14:04:39.861240 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:04:39.861144    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6br5b" podUID="f8e658c5-2624-4b4b-8476-89ffc2b11119" containerName="kube-rbac-proxy" containerID="cri-o://8b0908d0516170c96ae01fcc23a9aa8f01c4ed7eb09012efb2bbc2045555e27a" gracePeriod=30
Jul 28 14:04:40.191315 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:04:40.191196    2554 generic.go:356] "Generic (PLEG): container finished" podID="f8e658c5-2624-4b4b-8476-89ffc2b11119" containerID="8b0908d0516170c96ae01fcc23a9aa8f01c4ed7eb09012efb2bbc2045555e27a" exitCode=2
Jul 28 14:04:40.191315 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:04:40.191263    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6br5b" event={"ID":"f8e658c5-2624-4b4b-8476-89ffc2b11119","Type":"ContainerDied","Data":"8b0908d0516170c96ae01fcc23a9aa8f01c4ed7eb09012efb2bbc2045555e27a"}
Jul 28 14:04:41.968967 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:04:41.968933    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6br5b" podUID="f8e658c5-2624-4b4b-8476-89ffc2b11119" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.52:8643/healthz\": dial tcp 10.134.0.52:8643: connect: connection refused"
Jul 28 14:04:43.016456 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:04:43.016415    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6br5b" podUID="f8e658c5-2624-4b4b-8476-89ffc2b11119" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.52:8080/v2/models/sklearn-v2-mlserver/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 28 14:04:46.909479 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:04:46.909453    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6br5b"
Jul 28 14:04:46.977520 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:04:46.977450    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j8m8d\" (UniqueName: \"kubernetes.io/projected/f8e658c5-2624-4b4b-8476-89ffc2b11119-kube-api-access-j8m8d\") pod \"f8e658c5-2624-4b4b-8476-89ffc2b11119\" (UID: \"f8e658c5-2624-4b4b-8476-89ffc2b11119\") "
Jul 28 14:04:46.977520 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:04:46.977515    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f8e658c5-2624-4b4b-8476-89ffc2b11119-cabundle-cert\") pod \"f8e658c5-2624-4b4b-8476-89ffc2b11119\" (UID: \"f8e658c5-2624-4b4b-8476-89ffc2b11119\") "
Jul 28 14:04:46.977744 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:04:46.977538    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8e658c5-2624-4b4b-8476-89ffc2b11119-kserve-provision-location\") pod \"f8e658c5-2624-4b4b-8476-89ffc2b11119\" (UID: \"f8e658c5-2624-4b4b-8476-89ffc2b11119\") "
Jul 28 14:04:46.977744 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:04:46.977582    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8e658c5-2624-4b4b-8476-89ffc2b11119-proxy-tls\") pod \"f8e658c5-2624-4b4b-8476-89ffc2b11119\" (UID: \"f8e658c5-2624-4b4b-8476-89ffc2b11119\") "
Jul 28 14:04:46.977744 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:04:46.977601    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8e658c5-2624-4b4b-8476-89ffc2b11119-kube-rbac-proxy-sar-config\") pod \"f8e658c5-2624-4b4b-8476-89ffc2b11119\" (UID: \"f8e658c5-2624-4b4b-8476-89ffc2b11119\") "
Jul 28 14:04:46.977929 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:04:46.977905    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8e658c5-2624-4b4b-8476-89ffc2b11119-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f8e658c5-2624-4b4b-8476-89ffc2b11119" (UID: "f8e658c5-2624-4b4b-8476-89ffc2b11119"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:04:46.977988 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:04:46.977947    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8e658c5-2624-4b4b-8476-89ffc2b11119-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f8e658c5-2624-4b4b-8476-89ffc2b11119" (UID: "f8e658c5-2624-4b4b-8476-89ffc2b11119"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:04:46.978027 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:04:46.977992    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8e658c5-2624-4b4b-8476-89ffc2b11119-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f8e658c5-2624-4b4b-8476-89ffc2b11119" (UID: "f8e658c5-2624-4b4b-8476-89ffc2b11119"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:04:46.979873 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:04:46.979842    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8e658c5-2624-4b4b-8476-89ffc2b11119-kube-api-access-j8m8d" (OuterVolumeSpecName: "kube-api-access-j8m8d") pod "f8e658c5-2624-4b4b-8476-89ffc2b11119" (UID: "f8e658c5-2624-4b4b-8476-89ffc2b11119"). InnerVolumeSpecName "kube-api-access-j8m8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:04:46.979873 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:04:46.979859    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8e658c5-2624-4b4b-8476-89ffc2b11119-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f8e658c5-2624-4b4b-8476-89ffc2b11119" (UID: "f8e658c5-2624-4b4b-8476-89ffc2b11119"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:04:47.078266 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:04:47.078232    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f8e658c5-2624-4b4b-8476-89ffc2b11119-proxy-tls\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:04:47.078266 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:04:47.078261    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f8e658c5-2624-4b4b-8476-89ffc2b11119-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:04:47.078266 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:04:47.078289    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8m8d\" (UniqueName: \"kubernetes.io/projected/f8e658c5-2624-4b4b-8476-89ffc2b11119-kube-api-access-j8m8d\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:04:47.078518 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:04:47.078298    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f8e658c5-2624-4b4b-8476-89ffc2b11119-cabundle-cert\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:04:47.078518 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:04:47.078307    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8e658c5-2624-4b4b-8476-89ffc2b11119-kserve-provision-location\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:04:47.221369 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:04:47.221323    2554 generic.go:356] "Generic (PLEG): container finished" podID="f8e658c5-2624-4b4b-8476-89ffc2b11119" containerID="33945f64e8fec1f01dc5e5b3ca398df0df6e7889986069c3451d9812dcfef8d1" exitCode=0
Jul 28 14:04:47.221369 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:04:47.221381    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6br5b"
Jul 28 14:04:47.221586 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:04:47.221400    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6br5b" event={"ID":"f8e658c5-2624-4b4b-8476-89ffc2b11119","Type":"ContainerDied","Data":"33945f64e8fec1f01dc5e5b3ca398df0df6e7889986069c3451d9812dcfef8d1"}
Jul 28 14:04:47.221586 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:04:47.221458    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6br5b" event={"ID":"f8e658c5-2624-4b4b-8476-89ffc2b11119","Type":"ContainerDied","Data":"9e80aefaff0adc99c9abe5ef083bdd11c5f964b45d809903495dd7c97f8c21e1"}
Jul 28 14:04:47.221586 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:04:47.221486    2554 scope.go:117] "RemoveContainer" containerID="8b0908d0516170c96ae01fcc23a9aa8f01c4ed7eb09012efb2bbc2045555e27a"
Jul 28 14:04:47.231054 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:04:47.231036    2554 scope.go:117] "RemoveContainer" containerID="33945f64e8fec1f01dc5e5b3ca398df0df6e7889986069c3451d9812dcfef8d1"
Jul 28 14:04:47.241301 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:04:47.241255    2554 scope.go:117] "RemoveContainer" containerID="0a2effed15cbc86fc160a0541100d25d4bc363544c1978564a7cd1be9788c837"
Jul 28 14:04:47.248809 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:04:47.248792    2554 scope.go:117] "RemoveContainer" containerID="8b0908d0516170c96ae01fcc23a9aa8f01c4ed7eb09012efb2bbc2045555e27a"
Jul 28 14:04:47.249092 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:04:47.249071    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b0908d0516170c96ae01fcc23a9aa8f01c4ed7eb09012efb2bbc2045555e27a\": container with ID starting with 8b0908d0516170c96ae01fcc23a9aa8f01c4ed7eb09012efb2bbc2045555e27a not found: ID does not exist" containerID="8b0908d0516170c96ae01fcc23a9aa8f01c4ed7eb09012efb2bbc2045555e27a"
Jul 28 14:04:47.249167 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:04:47.249105    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b0908d0516170c96ae01fcc23a9aa8f01c4ed7eb09012efb2bbc2045555e27a"} err="failed to get container status \"8b0908d0516170c96ae01fcc23a9aa8f01c4ed7eb09012efb2bbc2045555e27a\": rpc error: code = NotFound desc = could not find container \"8b0908d0516170c96ae01fcc23a9aa8f01c4ed7eb09012efb2bbc2045555e27a\": container with ID starting with 8b0908d0516170c96ae01fcc23a9aa8f01c4ed7eb09012efb2bbc2045555e27a not found: ID does not exist"
Jul 28 14:04:47.249167 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:04:47.249131    2554 scope.go:117] "RemoveContainer" containerID="33945f64e8fec1f01dc5e5b3ca398df0df6e7889986069c3451d9812dcfef8d1"
Jul 28 14:04:47.249414 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:04:47.249396    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33945f64e8fec1f01dc5e5b3ca398df0df6e7889986069c3451d9812dcfef8d1\": container with ID starting with 33945f64e8fec1f01dc5e5b3ca398df0df6e7889986069c3451d9812dcfef8d1 not found: ID does not exist" containerID="33945f64e8fec1f01dc5e5b3ca398df0df6e7889986069c3451d9812dcfef8d1"
Jul 28 14:04:47.249487 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:04:47.249420    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33945f64e8fec1f01dc5e5b3ca398df0df6e7889986069c3451d9812dcfef8d1"} err="failed to get container status \"33945f64e8fec1f01dc5e5b3ca398df0df6e7889986069c3451d9812dcfef8d1\": rpc error: code = NotFound desc = could not find container \"33945f64e8fec1f01dc5e5b3ca398df0df6e7889986069c3451d9812dcfef8d1\": container with ID starting with 33945f64e8fec1f01dc5e5b3ca398df0df6e7889986069c3451d9812dcfef8d1 not found: ID does not exist"
Jul 28 14:04:47.249487 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:04:47.249439    2554 scope.go:117] "RemoveContainer" containerID="0a2effed15cbc86fc160a0541100d25d4bc363544c1978564a7cd1be9788c837"
Jul 28 14:04:47.249700 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:04:47.249681    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a2effed15cbc86fc160a0541100d25d4bc363544c1978564a7cd1be9788c837\": container with ID starting with 0a2effed15cbc86fc160a0541100d25d4bc363544c1978564a7cd1be9788c837 not found: ID does not exist" containerID="0a2effed15cbc86fc160a0541100d25d4bc363544c1978564a7cd1be9788c837"
Jul 28 14:04:47.249738 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:04:47.249706    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a2effed15cbc86fc160a0541100d25d4bc363544c1978564a7cd1be9788c837"} err="failed to get container status \"0a2effed15cbc86fc160a0541100d25d4bc363544c1978564a7cd1be9788c837\": rpc error: code = NotFound desc = could not find container \"0a2effed15cbc86fc160a0541100d25d4bc363544c1978564a7cd1be9788c837\": container with ID starting with 0a2effed15cbc86fc160a0541100d25d4bc363544c1978564a7cd1be9788c837 not found: ID does not exist"
Jul 28 14:04:47.269802 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:04:47.269774    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6br5b"]
Jul 28 14:04:47.284082 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:04:47.284049    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-6br5b"]
Jul 28 14:04:47.802390 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:04:47.802350    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8e658c5-2624-4b4b-8476-89ffc2b11119" path="/var/lib/kubelet/pods/f8e658c5-2624-4b4b-8476-89ffc2b11119/volumes"
Jul 28 14:05:22.327371 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:05:22.327283    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh"]
Jul 28 14:05:22.327948 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:05:22.327747    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8e658c5-2624-4b4b-8476-89ffc2b11119" containerName="kserve-container"
Jul 28 14:05:22.327948 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:05:22.327765    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8e658c5-2624-4b4b-8476-89ffc2b11119" containerName="kserve-container"
Jul 28 14:05:22.327948 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:05:22.327785    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8e658c5-2624-4b4b-8476-89ffc2b11119" containerName="storage-initializer"
Jul 28 14:05:22.327948 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:05:22.327793    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8e658c5-2624-4b4b-8476-89ffc2b11119" containerName="storage-initializer"
Jul 28 14:05:22.327948 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:05:22.327823    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8e658c5-2624-4b4b-8476-89ffc2b11119" containerName="kube-rbac-proxy"
Jul 28 14:05:22.327948 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:05:22.327832    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8e658c5-2624-4b4b-8476-89ffc2b11119" containerName="kube-rbac-proxy"
Jul 28 14:05:22.327948 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:05:22.327914    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8e658c5-2624-4b4b-8476-89ffc2b11119" containerName="kserve-container"
Jul 28 14:05:22.327948 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:05:22.327927    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8e658c5-2624-4b4b-8476-89ffc2b11119" containerName="kube-rbac-proxy"
Jul 28 14:05:22.331386 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:05:22.331360    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh"
Jul 28 14:05:22.334036 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:05:22.334015    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 14:05:22.334165 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:05:22.334063    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rxfwd\""
Jul 28 14:05:22.334165 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:05:22.334151    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 28 14:05:22.334419 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:05:22.334403    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 14:05:22.334925 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:05:22.334909    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
Jul 28 14:05:22.334996 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:05:22.334964    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 14:05:22.334996 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:05:22.334977    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 14:05:22.342614 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:05:22.342592    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh"]
Jul 28 14:05:22.464186 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:05:22.464154    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mhjk\" (UniqueName: \"kubernetes.io/projected/3e5006de-18cd-40d4-9690-bad86b0afaeb-kube-api-access-4mhjk\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh\" (UID: \"3e5006de-18cd-40d4-9690-bad86b0afaeb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh"
Jul 28 14:05:22.464360 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:05:22.464198    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e5006de-18cd-40d4-9690-bad86b0afaeb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh\" (UID: \"3e5006de-18cd-40d4-9690-bad86b0afaeb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh"
Jul 28 14:05:22.464360 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:05:22.464235    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e5006de-18cd-40d4-9690-bad86b0afaeb-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh\" (UID: \"3e5006de-18cd-40d4-9690-bad86b0afaeb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh"
Jul 28 14:05:22.464360 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:05:22.464316    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e5006de-18cd-40d4-9690-bad86b0afaeb-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh\" (UID: \"3e5006de-18cd-40d4-9690-bad86b0afaeb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh"
Jul 28 14:05:22.464360 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:05:22.464334    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e5006de-18cd-40d4-9690-bad86b0afaeb-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh\" (UID: \"3e5006de-18cd-40d4-9690-bad86b0afaeb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh"
Jul 28 14:05:22.565442 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:05:22.565412    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e5006de-18cd-40d4-9690-bad86b0afaeb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh\" (UID: \"3e5006de-18cd-40d4-9690-bad86b0afaeb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh"
Jul 28 14:05:22.565559 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:05:22.565453    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e5006de-18cd-40d4-9690-bad86b0afaeb-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh\" (UID: \"3e5006de-18cd-40d4-9690-bad86b0afaeb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh"
Jul 28 14:05:22.565559 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:05:22.565481    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e5006de-18cd-40d4-9690-bad86b0afaeb-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh\" (UID: \"3e5006de-18cd-40d4-9690-bad86b0afaeb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh"
Jul 28 14:05:22.565559 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:05:22.565505    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e5006de-18cd-40d4-9690-bad86b0afaeb-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh\" (UID: \"3e5006de-18cd-40d4-9690-bad86b0afaeb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh"
Jul 28 14:05:22.565688 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:05:22.565570    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mhjk\" (UniqueName: \"kubernetes.io/projected/3e5006de-18cd-40d4-9690-bad86b0afaeb-kube-api-access-4mhjk\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh\" (UID: \"3e5006de-18cd-40d4-9690-bad86b0afaeb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh"
Jul 28 14:05:22.565688 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:05:22.565584    2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-serving-cert: secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
Jul 28 14:05:22.565688 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:05:22.565650    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e5006de-18cd-40d4-9690-bad86b0afaeb-proxy-tls podName:3e5006de-18cd-40d4-9690-bad86b0afaeb nodeName:}" failed. No retries permitted until 2026-07-28 14:05:23.065628122 +0000 UTC m=+4319.851503213 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3e5006de-18cd-40d4-9690-bad86b0afaeb-proxy-tls") pod "isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh" (UID: "3e5006de-18cd-40d4-9690-bad86b0afaeb") : secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
Jul 28 14:05:22.565966 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:05:22.565944    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e5006de-18cd-40d4-9690-bad86b0afaeb-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh\" (UID: \"3e5006de-18cd-40d4-9690-bad86b0afaeb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh"
Jul 28 14:05:22.566220 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:05:22.566199    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e5006de-18cd-40d4-9690-bad86b0afaeb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh\" (UID: \"3e5006de-18cd-40d4-9690-bad86b0afaeb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh"
Jul 28 14:05:22.566258 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:05:22.566235    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e5006de-18cd-40d4-9690-bad86b0afaeb-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh\" (UID: \"3e5006de-18cd-40d4-9690-bad86b0afaeb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh"
Jul 28 14:05:22.574721 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:05:22.574693    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mhjk\" (UniqueName: \"kubernetes.io/projected/3e5006de-18cd-40d4-9690-bad86b0afaeb-kube-api-access-4mhjk\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh\" (UID: \"3e5006de-18cd-40d4-9690-bad86b0afaeb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh"
Jul 28 14:05:23.070752 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:05:23.070715    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e5006de-18cd-40d4-9690-bad86b0afaeb-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh\" (UID: \"3e5006de-18cd-40d4-9690-bad86b0afaeb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh"
Jul 28 14:05:23.073190 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:05:23.073166    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e5006de-18cd-40d4-9690-bad86b0afaeb-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh\" (UID: \"3e5006de-18cd-40d4-9690-bad86b0afaeb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh"
Jul 28 14:05:23.243381 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:05:23.243343    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh"
Jul 28 14:05:23.377057 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:05:23.377033    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh"]
Jul 28 14:05:23.379591 ip-10-0-129-177 kubenswrapper[2554]: W0728 14:05:23.379553    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e5006de_18cd_40d4_9690_bad86b0afaeb.slice/crio-24ae1f38969e5861d9d4bd24d0171080f341bfb231322356b36fdc520bb22502 WatchSource:0}: Error finding container 24ae1f38969e5861d9d4bd24d0171080f341bfb231322356b36fdc520bb22502: Status 404 returned error can't find the container with id 24ae1f38969e5861d9d4bd24d0171080f341bfb231322356b36fdc520bb22502
Jul 28 14:05:24.370019 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:05:24.369987    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh" event={"ID":"3e5006de-18cd-40d4-9690-bad86b0afaeb","Type":"ContainerStarted","Data":"6dc23096908f29af0ac23950e7f601f15b1617793161b1f9006ca06c04ea15fc"}
Jul 28 14:05:24.370205 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:05:24.370051    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh" event={"ID":"3e5006de-18cd-40d4-9690-bad86b0afaeb","Type":"ContainerStarted","Data":"24ae1f38969e5861d9d4bd24d0171080f341bfb231322356b36fdc520bb22502"}
Jul 28 14:05:27.383427 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:05:27.383397    2554 generic.go:356] "Generic (PLEG): container finished" podID="3e5006de-18cd-40d4-9690-bad86b0afaeb" containerID="6dc23096908f29af0ac23950e7f601f15b1617793161b1f9006ca06c04ea15fc" exitCode=0
Jul 28 14:05:27.383781 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:05:27.383472    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh" event={"ID":"3e5006de-18cd-40d4-9690-bad86b0afaeb","Type":"ContainerDied","Data":"6dc23096908f29af0ac23950e7f601f15b1617793161b1f9006ca06c04ea15fc"}
Jul 28 14:05:28.389006 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:05:28.388972    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh" event={"ID":"3e5006de-18cd-40d4-9690-bad86b0afaeb","Type":"ContainerStarted","Data":"20f033e8fa1657d1a9d3045933db8cfb03ff0907288229290dbdd2a79ee5bed1"}
Jul 28 14:05:28.389472 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:05:28.389029    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh" event={"ID":"3e5006de-18cd-40d4-9690-bad86b0afaeb","Type":"ContainerStarted","Data":"1f3e456ec09272b9c68840e5400ef7b9f9ca78aac33a2e6a5287ee0fe7394dc5"}
Jul 28 14:05:28.389472 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:05:28.389233    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh"
Jul 28 14:05:28.389472 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:05:28.389298    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh"
Jul 28 14:05:28.416087 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:05:28.416038    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh" podStartSLOduration=2.5468893550000002 podStartE2EDuration="6.416020628s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.869131273s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:05:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:05:28.413841187 +0000 UTC m=+4325.199716297" watchObservedRunningTime="2026-07-28 14:05:28.416020628 +0000 UTC m=+4325.201895739"
Jul 28 14:05:34.399791 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:05:34.399754    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh"
Jul 28 14:06:04.488584 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:04.488533    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh" podUID="3e5006de-18cd-40d4-9690-bad86b0afaeb" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Jul 28 14:06:14.403106 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:14.403074    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh"
Jul 28 14:06:22.266564 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:22.266531    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh"]
Jul 28 14:06:22.266986 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:22.266782    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh" podUID="3e5006de-18cd-40d4-9690-bad86b0afaeb" containerName="kserve-container" containerID="cri-o://1f3e456ec09272b9c68840e5400ef7b9f9ca78aac33a2e6a5287ee0fe7394dc5" gracePeriod=30
Jul 28 14:06:22.266986 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:22.266809    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh" podUID="3e5006de-18cd-40d4-9690-bad86b0afaeb" containerName="kube-rbac-proxy" containerID="cri-o://20f033e8fa1657d1a9d3045933db8cfb03ff0907288229290dbdd2a79ee5bed1" gracePeriod=30
Jul 28 14:06:22.617710 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:22.617628    2554 generic.go:356] "Generic (PLEG): container finished" podID="3e5006de-18cd-40d4-9690-bad86b0afaeb" containerID="20f033e8fa1657d1a9d3045933db8cfb03ff0907288229290dbdd2a79ee5bed1" exitCode=2
Jul 28 14:06:22.617710 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:22.617698    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh" event={"ID":"3e5006de-18cd-40d4-9690-bad86b0afaeb","Type":"ContainerDied","Data":"20f033e8fa1657d1a9d3045933db8cfb03ff0907288229290dbdd2a79ee5bed1"}
Jul 28 14:06:24.394750 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:24.394705    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh" podUID="3e5006de-18cd-40d4-9690-bad86b0afaeb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.53:8643/healthz\": dial tcp 10.134.0.53:8643: connect: connection refused"
Jul 28 14:06:25.442437 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:25.442393    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh" podUID="3e5006de-18cd-40d4-9690-bad86b0afaeb" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.53:8080/v2/models/isvc-sklearn-v2-runtime/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 28 14:06:27.422177 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:27.422142    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt"]
Jul 28 14:06:27.425870 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:27.425850    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt"
Jul 28 14:06:27.428351 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:27.428311    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
Jul 28 14:06:27.428475 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:27.428372    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
Jul 28 14:06:27.437292 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:27.437247    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt"]
Jul 28 14:06:27.498394 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:27.498358    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a89a0dc4-1d31-4ad2-a275-60c68a3c8839-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt\" (UID: \"a89a0dc4-1d31-4ad2-a275-60c68a3c8839\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt"
Jul 28 14:06:27.498572 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:27.498427    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a89a0dc4-1d31-4ad2-a275-60c68a3c8839-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt\" (UID: \"a89a0dc4-1d31-4ad2-a275-60c68a3c8839\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt"
Jul 28 14:06:27.498572 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:27.498514    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9tl8\" (UniqueName: \"kubernetes.io/projected/a89a0dc4-1d31-4ad2-a275-60c68a3c8839-kube-api-access-c9tl8\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt\" (UID: \"a89a0dc4-1d31-4ad2-a275-60c68a3c8839\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt"
Jul 28 14:06:27.498761 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:27.498592    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a89a0dc4-1d31-4ad2-a275-60c68a3c8839-proxy-tls\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt\" (UID: \"a89a0dc4-1d31-4ad2-a275-60c68a3c8839\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt"
Jul 28 14:06:27.498761 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:27.498678    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a89a0dc4-1d31-4ad2-a275-60c68a3c8839-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt\" (UID: \"a89a0dc4-1d31-4ad2-a275-60c68a3c8839\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt"
Jul 28 14:06:27.600007 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:27.599969    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a89a0dc4-1d31-4ad2-a275-60c68a3c8839-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt\" (UID: \"a89a0dc4-1d31-4ad2-a275-60c68a3c8839\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt"
Jul 28 14:06:27.600007 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:27.600013    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a89a0dc4-1d31-4ad2-a275-60c68a3c8839-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt\" (UID: \"a89a0dc4-1d31-4ad2-a275-60c68a3c8839\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt"
Jul 28 14:06:27.600244 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:27.600070    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a89a0dc4-1d31-4ad2-a275-60c68a3c8839-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt\" (UID: \"a89a0dc4-1d31-4ad2-a275-60c68a3c8839\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt"
Jul 28 14:06:27.600244 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:27.600103    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9tl8\" (UniqueName: \"kubernetes.io/projected/a89a0dc4-1d31-4ad2-a275-60c68a3c8839-kube-api-access-c9tl8\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt\" (UID: \"a89a0dc4-1d31-4ad2-a275-60c68a3c8839\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt"
Jul 28 14:06:27.600244 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:27.600142    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a89a0dc4-1d31-4ad2-a275-60c68a3c8839-proxy-tls\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt\" (UID: \"a89a0dc4-1d31-4ad2-a275-60c68a3c8839\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt"
Jul 28 14:06:27.600428 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:06:27.600305    2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-predictor-serving-cert: secret "isvc-sklearn-v2-predictor-serving-cert" not found
Jul 28 14:06:27.600428 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:06:27.600382    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a89a0dc4-1d31-4ad2-a275-60c68a3c8839-proxy-tls podName:a89a0dc4-1d31-4ad2-a275-60c68a3c8839 nodeName:}" failed. No retries permitted until 2026-07-28 14:06:28.100359366 +0000 UTC m=+4384.886234455 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a89a0dc4-1d31-4ad2-a275-60c68a3c8839-proxy-tls") pod "isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt" (UID: "a89a0dc4-1d31-4ad2-a275-60c68a3c8839") : secret "isvc-sklearn-v2-predictor-serving-cert" not found
Jul 28 14:06:27.600613 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:27.600590    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a89a0dc4-1d31-4ad2-a275-60c68a3c8839-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt\" (UID: \"a89a0dc4-1d31-4ad2-a275-60c68a3c8839\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt"
Jul 28 14:06:27.600775 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:27.600758    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a89a0dc4-1d31-4ad2-a275-60c68a3c8839-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt\" (UID: \"a89a0dc4-1d31-4ad2-a275-60c68a3c8839\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt"
Jul 28 14:06:27.600822 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:27.600805    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a89a0dc4-1d31-4ad2-a275-60c68a3c8839-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt\" (UID: \"a89a0dc4-1d31-4ad2-a275-60c68a3c8839\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt"
Jul 28 14:06:27.611669 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:27.611636    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9tl8\" (UniqueName: \"kubernetes.io/projected/a89a0dc4-1d31-4ad2-a275-60c68a3c8839-kube-api-access-c9tl8\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt\" (UID: \"a89a0dc4-1d31-4ad2-a275-60c68a3c8839\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt"
Jul 28 14:06:28.104425 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:28.104385    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a89a0dc4-1d31-4ad2-a275-60c68a3c8839-proxy-tls\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt\" (UID: \"a89a0dc4-1d31-4ad2-a275-60c68a3c8839\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt"
Jul 28 14:06:28.106953 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:28.106933    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a89a0dc4-1d31-4ad2-a275-60c68a3c8839-proxy-tls\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt\" (UID: \"a89a0dc4-1d31-4ad2-a275-60c68a3c8839\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt"
Jul 28 14:06:28.337172 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:28.337136    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt"
Jul 28 14:06:28.477698 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:28.477674    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt"]
Jul 28 14:06:28.480426 ip-10-0-129-177 kubenswrapper[2554]: W0728 14:06:28.480391    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda89a0dc4_1d31_4ad2_a275_60c68a3c8839.slice/crio-78052176bee0a1bda3aa60afc55ba4a63a5371d02951a33db8629d771bf9b4f0 WatchSource:0}: Error finding container 78052176bee0a1bda3aa60afc55ba4a63a5371d02951a33db8629d771bf9b4f0: Status 404 returned error can't find the container with id 78052176bee0a1bda3aa60afc55ba4a63a5371d02951a33db8629d771bf9b4f0
Jul 28 14:06:28.642505 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:28.642402    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt" event={"ID":"a89a0dc4-1d31-4ad2-a275-60c68a3c8839","Type":"ContainerStarted","Data":"3bf0ac00809790b5ad80cc4a5e2f26c88c6e5411a9fde63d05fa372339c8073d"}
Jul 28 14:06:28.642505 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:28.642489    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt" event={"ID":"a89a0dc4-1d31-4ad2-a275-60c68a3c8839","Type":"ContainerStarted","Data":"78052176bee0a1bda3aa60afc55ba4a63a5371d02951a33db8629d771bf9b4f0"}
Jul 28 14:06:29.316924 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:29.316899    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh"
Jul 28 14:06:29.417064 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:29.416993    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e5006de-18cd-40d4-9690-bad86b0afaeb-proxy-tls\") pod \"3e5006de-18cd-40d4-9690-bad86b0afaeb\" (UID: \"3e5006de-18cd-40d4-9690-bad86b0afaeb\") "
Jul 28 14:06:29.417064 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:29.417032    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e5006de-18cd-40d4-9690-bad86b0afaeb-kserve-provision-location\") pod \"3e5006de-18cd-40d4-9690-bad86b0afaeb\" (UID: \"3e5006de-18cd-40d4-9690-bad86b0afaeb\") "
Jul 28 14:06:29.417253 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:29.417103    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e5006de-18cd-40d4-9690-bad86b0afaeb-cabundle-cert\") pod \"3e5006de-18cd-40d4-9690-bad86b0afaeb\" (UID: \"3e5006de-18cd-40d4-9690-bad86b0afaeb\") "
Jul 28 14:06:29.417253 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:29.417120    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4mhjk\" (UniqueName: \"kubernetes.io/projected/3e5006de-18cd-40d4-9690-bad86b0afaeb-kube-api-access-4mhjk\") pod \"3e5006de-18cd-40d4-9690-bad86b0afaeb\" (UID: \"3e5006de-18cd-40d4-9690-bad86b0afaeb\") "
Jul 28 14:06:29.417253 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:29.417147    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e5006de-18cd-40d4-9690-bad86b0afaeb-kube-rbac-proxy-sar-config\") pod \"3e5006de-18cd-40d4-9690-bad86b0afaeb\" (UID: \"3e5006de-18cd-40d4-9690-bad86b0afaeb\") "
Jul 28 14:06:29.417482 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:29.417424    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e5006de-18cd-40d4-9690-bad86b0afaeb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3e5006de-18cd-40d4-9690-bad86b0afaeb" (UID: "3e5006de-18cd-40d4-9690-bad86b0afaeb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:06:29.417565 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:29.417531    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e5006de-18cd-40d4-9690-bad86b0afaeb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3e5006de-18cd-40d4-9690-bad86b0afaeb" (UID: "3e5006de-18cd-40d4-9690-bad86b0afaeb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:06:29.417624 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:29.417596    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e5006de-18cd-40d4-9690-bad86b0afaeb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3e5006de-18cd-40d4-9690-bad86b0afaeb" (UID: "3e5006de-18cd-40d4-9690-bad86b0afaeb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:06:29.419312 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:29.419295    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e5006de-18cd-40d4-9690-bad86b0afaeb-kube-api-access-4mhjk" (OuterVolumeSpecName: "kube-api-access-4mhjk") pod "3e5006de-18cd-40d4-9690-bad86b0afaeb" (UID: "3e5006de-18cd-40d4-9690-bad86b0afaeb"). InnerVolumeSpecName "kube-api-access-4mhjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:06:29.419409 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:29.419391    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e5006de-18cd-40d4-9690-bad86b0afaeb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3e5006de-18cd-40d4-9690-bad86b0afaeb" (UID: "3e5006de-18cd-40d4-9690-bad86b0afaeb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:06:29.517902 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:29.517862    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e5006de-18cd-40d4-9690-bad86b0afaeb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:06:29.517902 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:29.517893    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e5006de-18cd-40d4-9690-bad86b0afaeb-proxy-tls\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:06:29.517902 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:29.517903    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e5006de-18cd-40d4-9690-bad86b0afaeb-kserve-provision-location\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:06:29.518393 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:29.517914    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e5006de-18cd-40d4-9690-bad86b0afaeb-cabundle-cert\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:06:29.518393 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:29.517923    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mhjk\" (UniqueName: \"kubernetes.io/projected/3e5006de-18cd-40d4-9690-bad86b0afaeb-kube-api-access-4mhjk\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:06:29.649465 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:29.649426    2554 generic.go:356] "Generic (PLEG): container finished" podID="3e5006de-18cd-40d4-9690-bad86b0afaeb" containerID="1f3e456ec09272b9c68840e5400ef7b9f9ca78aac33a2e6a5287ee0fe7394dc5" exitCode=0
Jul 28 14:06:29.649665 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:29.649481    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh" event={"ID":"3e5006de-18cd-40d4-9690-bad86b0afaeb","Type":"ContainerDied","Data":"1f3e456ec09272b9c68840e5400ef7b9f9ca78aac33a2e6a5287ee0fe7394dc5"}
Jul 28 14:06:29.649665 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:29.649496    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh"
Jul 28 14:06:29.649665 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:29.649540    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh" event={"ID":"3e5006de-18cd-40d4-9690-bad86b0afaeb","Type":"ContainerDied","Data":"24ae1f38969e5861d9d4bd24d0171080f341bfb231322356b36fdc520bb22502"}
Jul 28 14:06:29.649665 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:29.649566    2554 scope.go:117] "RemoveContainer" containerID="20f033e8fa1657d1a9d3045933db8cfb03ff0907288229290dbdd2a79ee5bed1"
Jul 28 14:06:29.661855 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:29.661824    2554 scope.go:117] "RemoveContainer" containerID="1f3e456ec09272b9c68840e5400ef7b9f9ca78aac33a2e6a5287ee0fe7394dc5"
Jul 28 14:06:29.669571 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:29.669534    2554 scope.go:117] "RemoveContainer" containerID="6dc23096908f29af0ac23950e7f601f15b1617793161b1f9006ca06c04ea15fc"
Jul 28 14:06:29.677528 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:29.677501    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh"]
Jul 28 14:06:29.678113 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:29.678095    2554 scope.go:117] "RemoveContainer" containerID="20f033e8fa1657d1a9d3045933db8cfb03ff0907288229290dbdd2a79ee5bed1"
Jul 28 14:06:29.678381 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:06:29.678363    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20f033e8fa1657d1a9d3045933db8cfb03ff0907288229290dbdd2a79ee5bed1\": container with ID starting with 20f033e8fa1657d1a9d3045933db8cfb03ff0907288229290dbdd2a79ee5bed1 not found: ID does not exist" containerID="20f033e8fa1657d1a9d3045933db8cfb03ff0907288229290dbdd2a79ee5bed1"
Jul 28 14:06:29.678438 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:29.678388    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20f033e8fa1657d1a9d3045933db8cfb03ff0907288229290dbdd2a79ee5bed1"} err="failed to get container status \"20f033e8fa1657d1a9d3045933db8cfb03ff0907288229290dbdd2a79ee5bed1\": rpc error: code = NotFound desc = could not find container \"20f033e8fa1657d1a9d3045933db8cfb03ff0907288229290dbdd2a79ee5bed1\": container with ID starting with 20f033e8fa1657d1a9d3045933db8cfb03ff0907288229290dbdd2a79ee5bed1 not found: ID does not exist"
Jul 28 14:06:29.678438 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:29.678404    2554 scope.go:117] "RemoveContainer" containerID="1f3e456ec09272b9c68840e5400ef7b9f9ca78aac33a2e6a5287ee0fe7394dc5"
Jul 28 14:06:29.678607 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:06:29.678591    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f3e456ec09272b9c68840e5400ef7b9f9ca78aac33a2e6a5287ee0fe7394dc5\": container with ID starting with 1f3e456ec09272b9c68840e5400ef7b9f9ca78aac33a2e6a5287ee0fe7394dc5 not found: ID does not exist" containerID="1f3e456ec09272b9c68840e5400ef7b9f9ca78aac33a2e6a5287ee0fe7394dc5"
Jul 28 14:06:29.678644 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:29.678610    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f3e456ec09272b9c68840e5400ef7b9f9ca78aac33a2e6a5287ee0fe7394dc5"} err="failed to get container status \"1f3e456ec09272b9c68840e5400ef7b9f9ca78aac33a2e6a5287ee0fe7394dc5\": rpc error: code = NotFound desc = could not find container \"1f3e456ec09272b9c68840e5400ef7b9f9ca78aac33a2e6a5287ee0fe7394dc5\": container with ID starting with 1f3e456ec09272b9c68840e5400ef7b9f9ca78aac33a2e6a5287ee0fe7394dc5 not found: ID does not exist"
Jul 28 14:06:29.678644 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:29.678623    2554 scope.go:117] "RemoveContainer" containerID="6dc23096908f29af0ac23950e7f601f15b1617793161b1f9006ca06c04ea15fc"
Jul 28 14:06:29.678835 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:06:29.678821    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dc23096908f29af0ac23950e7f601f15b1617793161b1f9006ca06c04ea15fc\": container with ID starting with 6dc23096908f29af0ac23950e7f601f15b1617793161b1f9006ca06c04ea15fc not found: ID does not exist" containerID="6dc23096908f29af0ac23950e7f601f15b1617793161b1f9006ca06c04ea15fc"
Jul 28 14:06:29.678875 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:29.678839    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dc23096908f29af0ac23950e7f601f15b1617793161b1f9006ca06c04ea15fc"} err="failed to get container status \"6dc23096908f29af0ac23950e7f601f15b1617793161b1f9006ca06c04ea15fc\": rpc error: code = NotFound desc = could not find container \"6dc23096908f29af0ac23950e7f601f15b1617793161b1f9006ca06c04ea15fc\": container with ID starting with 6dc23096908f29af0ac23950e7f601f15b1617793161b1f9006ca06c04ea15fc not found: ID does not exist"
Jul 28 14:06:29.687210 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:29.687191    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-k7xrh"]
Jul 28 14:06:29.801648 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:29.801612    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e5006de-18cd-40d4-9690-bad86b0afaeb" path="/var/lib/kubelet/pods/3e5006de-18cd-40d4-9690-bad86b0afaeb/volumes"
Jul 28 14:06:32.664290 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:32.664240    2554 generic.go:356] "Generic (PLEG): container finished" podID="a89a0dc4-1d31-4ad2-a275-60c68a3c8839" containerID="3bf0ac00809790b5ad80cc4a5e2f26c88c6e5411a9fde63d05fa372339c8073d" exitCode=0
Jul 28 14:06:32.664651 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:32.664319    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt" event={"ID":"a89a0dc4-1d31-4ad2-a275-60c68a3c8839","Type":"ContainerDied","Data":"3bf0ac00809790b5ad80cc4a5e2f26c88c6e5411a9fde63d05fa372339c8073d"}
Jul 28 14:06:33.670369 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:33.670338    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt" event={"ID":"a89a0dc4-1d31-4ad2-a275-60c68a3c8839","Type":"ContainerStarted","Data":"1a09695dff06b3b717a531b6c758bedb27927b699915ed3afef7b4e1831b2151"}
Jul 28 14:06:33.670781 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:33.670396    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt" event={"ID":"a89a0dc4-1d31-4ad2-a275-60c68a3c8839","Type":"ContainerStarted","Data":"72d7cb64a5509348f13059b5cc7b7a0082efd7ca18548150299f9af6cc514953"}
Jul 28 14:06:33.670781 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:33.670666    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt"
Jul 28 14:06:33.670887 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:33.670821    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt"
Jul 28 14:06:33.672099 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:33.672071    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt" podUID="a89a0dc4-1d31-4ad2-a275-60c68a3c8839" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Jul 28 14:06:33.693893 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:33.693844    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt" podStartSLOduration=3.271433239 podStartE2EDuration="6.693826316s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.422393077s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:06:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:06:33.691506402 +0000 UTC m=+4390.477381525" watchObservedRunningTime="2026-07-28 14:06:33.693826316 +0000 UTC m=+4390.479701428"
Jul 28 14:06:34.674930 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:34.674892    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt" podUID="a89a0dc4-1d31-4ad2-a275-60c68a3c8839" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Jul 28 14:06:39.679165 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:39.679132    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt"
Jul 28 14:06:39.679620 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:39.679597    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt" podUID="a89a0dc4-1d31-4ad2-a275-60c68a3c8839" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Jul 28 14:06:49.679865 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:49.679779    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt" podUID="a89a0dc4-1d31-4ad2-a275-60c68a3c8839" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Jul 28 14:06:59.680507 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:06:59.680471    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt" podUID="a89a0dc4-1d31-4ad2-a275-60c68a3c8839" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Jul 28 14:07:09.679533 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:07:09.679495    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt" podUID="a89a0dc4-1d31-4ad2-a275-60c68a3c8839" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Jul 28 14:07:19.680324 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:07:19.680262    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt" podUID="a89a0dc4-1d31-4ad2-a275-60c68a3c8839" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Jul 28 14:07:29.680584 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:07:29.680543    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt" podUID="a89a0dc4-1d31-4ad2-a275-60c68a3c8839" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Jul 28 14:07:39.679520 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:07:39.679485    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt" podUID="a89a0dc4-1d31-4ad2-a275-60c68a3c8839" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Jul 28 14:07:49.681106 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:07:49.681074    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt"
Jul 28 14:07:57.721051 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:07:57.721017    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt"]
Jul 28 14:07:57.721537 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:07:57.721373    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt" podUID="a89a0dc4-1d31-4ad2-a275-60c68a3c8839" containerName="kserve-container" containerID="cri-o://72d7cb64a5509348f13059b5cc7b7a0082efd7ca18548150299f9af6cc514953" gracePeriod=30
Jul 28 14:07:57.721537 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:07:57.721398    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt" podUID="a89a0dc4-1d31-4ad2-a275-60c68a3c8839" containerName="kube-rbac-proxy" containerID="cri-o://1a09695dff06b3b717a531b6c758bedb27927b699915ed3afef7b4e1831b2151" gracePeriod=30
Jul 28 14:07:58.029232 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:07:58.029141    2554 generic.go:356] "Generic (PLEG): container finished" podID="a89a0dc4-1d31-4ad2-a275-60c68a3c8839" containerID="1a09695dff06b3b717a531b6c758bedb27927b699915ed3afef7b4e1831b2151" exitCode=2
Jul 28 14:07:58.029400 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:07:58.029222    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt" event={"ID":"a89a0dc4-1d31-4ad2-a275-60c68a3c8839","Type":"ContainerDied","Data":"1a09695dff06b3b717a531b6c758bedb27927b699915ed3afef7b4e1831b2151"}
Jul 28 14:07:59.675819 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:07:59.675777    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt" podUID="a89a0dc4-1d31-4ad2-a275-60c68a3c8839" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.54:8643/healthz\": dial tcp 10.134.0.54:8643: connect: connection refused"
Jul 28 14:07:59.680106 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:07:59.680085    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt" podUID="a89a0dc4-1d31-4ad2-a275-60c68a3c8839" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Jul 28 14:08:02.273244 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:02.273220    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt"
Jul 28 14:08:02.297815 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:02.297789    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a89a0dc4-1d31-4ad2-a275-60c68a3c8839-kserve-provision-location\") pod \"a89a0dc4-1d31-4ad2-a275-60c68a3c8839\" (UID: \"a89a0dc4-1d31-4ad2-a275-60c68a3c8839\") "
Jul 28 14:08:02.297980 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:02.297832    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a89a0dc4-1d31-4ad2-a275-60c68a3c8839-proxy-tls\") pod \"a89a0dc4-1d31-4ad2-a275-60c68a3c8839\" (UID: \"a89a0dc4-1d31-4ad2-a275-60c68a3c8839\") "
Jul 28 14:08:02.297980 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:02.297851    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a89a0dc4-1d31-4ad2-a275-60c68a3c8839-kube-rbac-proxy-sar-config\") pod \"a89a0dc4-1d31-4ad2-a275-60c68a3c8839\" (UID: \"a89a0dc4-1d31-4ad2-a275-60c68a3c8839\") "
Jul 28 14:08:02.297980 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:02.297926    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c9tl8\" (UniqueName: \"kubernetes.io/projected/a89a0dc4-1d31-4ad2-a275-60c68a3c8839-kube-api-access-c9tl8\") pod \"a89a0dc4-1d31-4ad2-a275-60c68a3c8839\" (UID: \"a89a0dc4-1d31-4ad2-a275-60c68a3c8839\") "
Jul 28 14:08:02.297980 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:02.297953    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a89a0dc4-1d31-4ad2-a275-60c68a3c8839-cabundle-cert\") pod \"a89a0dc4-1d31-4ad2-a275-60c68a3c8839\" (UID: \"a89a0dc4-1d31-4ad2-a275-60c68a3c8839\") "
Jul 28 14:08:02.298195 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:02.298147    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a89a0dc4-1d31-4ad2-a275-60c68a3c8839-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a89a0dc4-1d31-4ad2-a275-60c68a3c8839" (UID: "a89a0dc4-1d31-4ad2-a275-60c68a3c8839"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:08:02.298297 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:02.298252    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a89a0dc4-1d31-4ad2-a275-60c68a3c8839-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a89a0dc4-1d31-4ad2-a275-60c68a3c8839" (UID: "a89a0dc4-1d31-4ad2-a275-60c68a3c8839"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:08:02.298492 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:02.298469    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a89a0dc4-1d31-4ad2-a275-60c68a3c8839-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a89a0dc4-1d31-4ad2-a275-60c68a3c8839" (UID: "a89a0dc4-1d31-4ad2-a275-60c68a3c8839"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:08:02.300528 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:02.300472    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a89a0dc4-1d31-4ad2-a275-60c68a3c8839-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a89a0dc4-1d31-4ad2-a275-60c68a3c8839" (UID: "a89a0dc4-1d31-4ad2-a275-60c68a3c8839"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:08:02.300646 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:02.300577    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a89a0dc4-1d31-4ad2-a275-60c68a3c8839-kube-api-access-c9tl8" (OuterVolumeSpecName: "kube-api-access-c9tl8") pod "a89a0dc4-1d31-4ad2-a275-60c68a3c8839" (UID: "a89a0dc4-1d31-4ad2-a275-60c68a3c8839"). InnerVolumeSpecName "kube-api-access-c9tl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:08:02.398873 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:02.398788    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9tl8\" (UniqueName: \"kubernetes.io/projected/a89a0dc4-1d31-4ad2-a275-60c68a3c8839-kube-api-access-c9tl8\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:08:02.398873 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:02.398816    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a89a0dc4-1d31-4ad2-a275-60c68a3c8839-cabundle-cert\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:08:02.398873 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:02.398826    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a89a0dc4-1d31-4ad2-a275-60c68a3c8839-kserve-provision-location\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:08:02.398873 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:02.398835    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a89a0dc4-1d31-4ad2-a275-60c68a3c8839-proxy-tls\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:08:02.398873 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:02.398844    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a89a0dc4-1d31-4ad2-a275-60c68a3c8839-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:08:02.919504 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:02.919468    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t"]
Jul 28 14:08:02.919822 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:02.919809    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a89a0dc4-1d31-4ad2-a275-60c68a3c8839" containerName="storage-initializer"
Jul 28 14:08:02.919870 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:02.919825    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a89a0dc4-1d31-4ad2-a275-60c68a3c8839" containerName="storage-initializer"
Jul 28 14:08:02.919870 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:02.919838    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a89a0dc4-1d31-4ad2-a275-60c68a3c8839" containerName="kube-rbac-proxy"
Jul 28 14:08:02.919870 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:02.919843    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a89a0dc4-1d31-4ad2-a275-60c68a3c8839" containerName="kube-rbac-proxy"
Jul 28 14:08:02.919870 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:02.919856    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e5006de-18cd-40d4-9690-bad86b0afaeb" containerName="storage-initializer"
Jul 28 14:08:02.919870 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:02.919862    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e5006de-18cd-40d4-9690-bad86b0afaeb" containerName="storage-initializer"
Jul 28 14:08:02.920028 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:02.919874    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e5006de-18cd-40d4-9690-bad86b0afaeb" containerName="kserve-container"
Jul 28 14:08:02.920028 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:02.919879    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e5006de-18cd-40d4-9690-bad86b0afaeb" containerName="kserve-container"
Jul 28 14:08:02.920028 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:02.919884    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a89a0dc4-1d31-4ad2-a275-60c68a3c8839" containerName="kserve-container"
Jul 28 14:08:02.920028 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:02.919889    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a89a0dc4-1d31-4ad2-a275-60c68a3c8839" containerName="kserve-container"
Jul 28 14:08:02.920028 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:02.919897    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e5006de-18cd-40d4-9690-bad86b0afaeb" containerName="kube-rbac-proxy"
Jul 28 14:08:02.920028 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:02.919902    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e5006de-18cd-40d4-9690-bad86b0afaeb" containerName="kube-rbac-proxy"
Jul 28 14:08:02.920028 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:02.919948    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="a89a0dc4-1d31-4ad2-a275-60c68a3c8839" containerName="kube-rbac-proxy"
Jul 28 14:08:02.920028 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:02.919956    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e5006de-18cd-40d4-9690-bad86b0afaeb" containerName="kserve-container"
Jul 28 14:08:02.920028 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:02.919964    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e5006de-18cd-40d4-9690-bad86b0afaeb" containerName="kube-rbac-proxy"
Jul 28 14:08:02.920028 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:02.919969    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="a89a0dc4-1d31-4ad2-a275-60c68a3c8839" containerName="kserve-container"
Jul 28 14:08:02.923385 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:02.923368    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t"
Jul 28 14:08:02.928104 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:02.928082    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Jul 28 14:08:02.928208 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:02.928130    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Jul 28 14:08:02.936169 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:02.936145    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t"]
Jul 28 14:08:03.003158 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:03.003122    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/048f90cd-ed22-45df-8809-27bce99cc953-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t\" (UID: \"048f90cd-ed22-45df-8809-27bce99cc953\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t"
Jul 28 14:08:03.003374 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:03.003164    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/048f90cd-ed22-45df-8809-27bce99cc953-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t\" (UID: \"048f90cd-ed22-45df-8809-27bce99cc953\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t"
Jul 28 14:08:03.003374 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:03.003185    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vvqd\" (UniqueName: \"kubernetes.io/projected/048f90cd-ed22-45df-8809-27bce99cc953-kube-api-access-7vvqd\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t\" (UID: \"048f90cd-ed22-45df-8809-27bce99cc953\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t"
Jul 28 14:08:03.003374 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:03.003210    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/048f90cd-ed22-45df-8809-27bce99cc953-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t\" (UID: \"048f90cd-ed22-45df-8809-27bce99cc953\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t"
Jul 28 14:08:03.003374 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:03.003264    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/048f90cd-ed22-45df-8809-27bce99cc953-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t\" (UID: \"048f90cd-ed22-45df-8809-27bce99cc953\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t"
Jul 28 14:08:03.049568 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:03.049531    2554 generic.go:356] "Generic (PLEG): container finished" podID="a89a0dc4-1d31-4ad2-a275-60c68a3c8839" containerID="72d7cb64a5509348f13059b5cc7b7a0082efd7ca18548150299f9af6cc514953" exitCode=0
Jul 28 14:08:03.049769 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:03.049599    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt"
Jul 28 14:08:03.049769 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:03.049613    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt" event={"ID":"a89a0dc4-1d31-4ad2-a275-60c68a3c8839","Type":"ContainerDied","Data":"72d7cb64a5509348f13059b5cc7b7a0082efd7ca18548150299f9af6cc514953"}
Jul 28 14:08:03.049769 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:03.049673    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt" event={"ID":"a89a0dc4-1d31-4ad2-a275-60c68a3c8839","Type":"ContainerDied","Data":"78052176bee0a1bda3aa60afc55ba4a63a5371d02951a33db8629d771bf9b4f0"}
Jul 28 14:08:03.049769 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:03.049699    2554 scope.go:117] "RemoveContainer" containerID="1a09695dff06b3b717a531b6c758bedb27927b699915ed3afef7b4e1831b2151"
Jul 28 14:08:03.059362 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:03.059259    2554 scope.go:117] "RemoveContainer" containerID="72d7cb64a5509348f13059b5cc7b7a0082efd7ca18548150299f9af6cc514953"
Jul 28 14:08:03.067233 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:03.067217    2554 scope.go:117] "RemoveContainer" containerID="3bf0ac00809790b5ad80cc4a5e2f26c88c6e5411a9fde63d05fa372339c8073d"
Jul 28 14:08:03.075096 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:03.075077    2554 scope.go:117] "RemoveContainer" containerID="1a09695dff06b3b717a531b6c758bedb27927b699915ed3afef7b4e1831b2151"
Jul 28 14:08:03.075420 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:08:03.075400    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a09695dff06b3b717a531b6c758bedb27927b699915ed3afef7b4e1831b2151\": container with ID starting with 1a09695dff06b3b717a531b6c758bedb27927b699915ed3afef7b4e1831b2151 not found: ID does not exist" containerID="1a09695dff06b3b717a531b6c758bedb27927b699915ed3afef7b4e1831b2151"
Jul 28 14:08:03.075507 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:03.075430    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a09695dff06b3b717a531b6c758bedb27927b699915ed3afef7b4e1831b2151"} err="failed to get container status \"1a09695dff06b3b717a531b6c758bedb27927b699915ed3afef7b4e1831b2151\": rpc error: code = NotFound desc = could not find container \"1a09695dff06b3b717a531b6c758bedb27927b699915ed3afef7b4e1831b2151\": container with ID starting with 1a09695dff06b3b717a531b6c758bedb27927b699915ed3afef7b4e1831b2151 not found: ID does not exist"
Jul 28 14:08:03.075507 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:03.075448    2554 scope.go:117] "RemoveContainer" containerID="72d7cb64a5509348f13059b5cc7b7a0082efd7ca18548150299f9af6cc514953"
Jul 28 14:08:03.075711 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:08:03.075689    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72d7cb64a5509348f13059b5cc7b7a0082efd7ca18548150299f9af6cc514953\": container with ID starting with 72d7cb64a5509348f13059b5cc7b7a0082efd7ca18548150299f9af6cc514953 not found: ID does not exist" containerID="72d7cb64a5509348f13059b5cc7b7a0082efd7ca18548150299f9af6cc514953"
Jul 28 14:08:03.075762 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:03.075722    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72d7cb64a5509348f13059b5cc7b7a0082efd7ca18548150299f9af6cc514953"} err="failed to get container status \"72d7cb64a5509348f13059b5cc7b7a0082efd7ca18548150299f9af6cc514953\": rpc error: code = NotFound desc = could not find container \"72d7cb64a5509348f13059b5cc7b7a0082efd7ca18548150299f9af6cc514953\": container with ID starting with 72d7cb64a5509348f13059b5cc7b7a0082efd7ca18548150299f9af6cc514953 not found: ID does not exist"
Jul 28 14:08:03.075762 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:03.075744    2554 scope.go:117] "RemoveContainer" containerID="3bf0ac00809790b5ad80cc4a5e2f26c88c6e5411a9fde63d05fa372339c8073d"
Jul 28 14:08:03.075900 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:03.075880    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt"]
Jul 28 14:08:03.076023 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:08:03.076007    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bf0ac00809790b5ad80cc4a5e2f26c88c6e5411a9fde63d05fa372339c8073d\": container with ID starting with 3bf0ac00809790b5ad80cc4a5e2f26c88c6e5411a9fde63d05fa372339c8073d not found: ID does not exist" containerID="3bf0ac00809790b5ad80cc4a5e2f26c88c6e5411a9fde63d05fa372339c8073d"
Jul 28 14:08:03.076069 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:03.076030    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bf0ac00809790b5ad80cc4a5e2f26c88c6e5411a9fde63d05fa372339c8073d"} err="failed to get container status \"3bf0ac00809790b5ad80cc4a5e2f26c88c6e5411a9fde63d05fa372339c8073d\": rpc error: code = NotFound desc = could not find container \"3bf0ac00809790b5ad80cc4a5e2f26c88c6e5411a9fde63d05fa372339c8073d\": container with ID starting with 3bf0ac00809790b5ad80cc4a5e2f26c88c6e5411a9fde63d05fa372339c8073d not found: ID does not exist"
Jul 28 14:08:03.080252 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:03.080228    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-vsnrt"]
Jul 28 14:08:03.104669 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:03.104642    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/048f90cd-ed22-45df-8809-27bce99cc953-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t\" (UID: \"048f90cd-ed22-45df-8809-27bce99cc953\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t"
Jul 28 14:08:03.104790 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:03.104678    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/048f90cd-ed22-45df-8809-27bce99cc953-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t\" (UID: \"048f90cd-ed22-45df-8809-27bce99cc953\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t"
Jul 28 14:08:03.104790 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:03.104701    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vvqd\" (UniqueName: \"kubernetes.io/projected/048f90cd-ed22-45df-8809-27bce99cc953-kube-api-access-7vvqd\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t\" (UID: \"048f90cd-ed22-45df-8809-27bce99cc953\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t"
Jul 28 14:08:03.104790 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:03.104724    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/048f90cd-ed22-45df-8809-27bce99cc953-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t\" (UID: \"048f90cd-ed22-45df-8809-27bce99cc953\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t"
Jul 28 14:08:03.104790 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:03.104771    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/048f90cd-ed22-45df-8809-27bce99cc953-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t\" (UID: \"048f90cd-ed22-45df-8809-27bce99cc953\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t"
Jul 28 14:08:03.105007 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:08:03.104915    2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-serving-cert: secret "isvc-sklearn-v2-mixed-predictor-serving-cert" not found
Jul 28 14:08:03.105007 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:08:03.104984    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/048f90cd-ed22-45df-8809-27bce99cc953-proxy-tls podName:048f90cd-ed22-45df-8809-27bce99cc953 nodeName:}" failed. No retries permitted until 2026-07-28 14:08:03.604963306 +0000 UTC m=+4480.390838402 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/048f90cd-ed22-45df-8809-27bce99cc953-proxy-tls") pod "isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t" (UID: "048f90cd-ed22-45df-8809-27bce99cc953") : secret "isvc-sklearn-v2-mixed-predictor-serving-cert" not found
Jul 28 14:08:03.105134 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:03.105113    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/048f90cd-ed22-45df-8809-27bce99cc953-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t\" (UID: \"048f90cd-ed22-45df-8809-27bce99cc953\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t"
Jul 28 14:08:03.105393 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:03.105378    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/048f90cd-ed22-45df-8809-27bce99cc953-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t\" (UID: \"048f90cd-ed22-45df-8809-27bce99cc953\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t"
Jul 28 14:08:03.105438 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:03.105417    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/048f90cd-ed22-45df-8809-27bce99cc953-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t\" (UID: \"048f90cd-ed22-45df-8809-27bce99cc953\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t"
Jul 28 14:08:03.115249 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:03.115232    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vvqd\" (UniqueName: \"kubernetes.io/projected/048f90cd-ed22-45df-8809-27bce99cc953-kube-api-access-7vvqd\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t\" (UID: \"048f90cd-ed22-45df-8809-27bce99cc953\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t"
Jul 28 14:08:03.609471 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:03.609433    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/048f90cd-ed22-45df-8809-27bce99cc953-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t\" (UID: \"048f90cd-ed22-45df-8809-27bce99cc953\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t"
Jul 28 14:08:03.611844 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:03.611817    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/048f90cd-ed22-45df-8809-27bce99cc953-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t\" (UID: \"048f90cd-ed22-45df-8809-27bce99cc953\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t"
Jul 28 14:08:03.803471 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:03.803436    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a89a0dc4-1d31-4ad2-a275-60c68a3c8839" path="/var/lib/kubelet/pods/a89a0dc4-1d31-4ad2-a275-60c68a3c8839/volumes"
Jul 28 14:08:03.836807 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:03.836770    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t"
Jul 28 14:08:03.973466 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:03.973441    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t"]
Jul 28 14:08:03.975194 ip-10-0-129-177 kubenswrapper[2554]: W0728 14:08:03.975160    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod048f90cd_ed22_45df_8809_27bce99cc953.slice/crio-29e038ab33c3227f8c9eeb94cc06d8827f6ead14f5f50ca6beda457224dbe1ec WatchSource:0}: Error finding container 29e038ab33c3227f8c9eeb94cc06d8827f6ead14f5f50ca6beda457224dbe1ec: Status 404 returned error can't find the container with id 29e038ab33c3227f8c9eeb94cc06d8827f6ead14f5f50ca6beda457224dbe1ec
Jul 28 14:08:04.056887 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:04.056849    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t" event={"ID":"048f90cd-ed22-45df-8809-27bce99cc953","Type":"ContainerStarted","Data":"208f76c601603800f75bb7fd1e4ec4ccdb48c88aeae8527c03ccd190ed2ec617"}
Jul 28 14:08:04.057043 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:04.056913    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t" event={"ID":"048f90cd-ed22-45df-8809-27bce99cc953","Type":"ContainerStarted","Data":"29e038ab33c3227f8c9eeb94cc06d8827f6ead14f5f50ca6beda457224dbe1ec"}
Jul 28 14:08:08.075057 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:08.075016    2554 generic.go:356] "Generic (PLEG): container finished" podID="048f90cd-ed22-45df-8809-27bce99cc953" containerID="208f76c601603800f75bb7fd1e4ec4ccdb48c88aeae8527c03ccd190ed2ec617" exitCode=0
Jul 28 14:08:08.075573 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:08.075076    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t" event={"ID":"048f90cd-ed22-45df-8809-27bce99cc953","Type":"ContainerDied","Data":"208f76c601603800f75bb7fd1e4ec4ccdb48c88aeae8527c03ccd190ed2ec617"}
Jul 28 14:08:09.081603 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:09.081572    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t" event={"ID":"048f90cd-ed22-45df-8809-27bce99cc953","Type":"ContainerStarted","Data":"378c4d132bfac2db3445187438449125196c271c4c2ee195b7e6164882dd1368"}
Jul 28 14:08:09.082033 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:09.081634    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t" event={"ID":"048f90cd-ed22-45df-8809-27bce99cc953","Type":"ContainerStarted","Data":"806a86b5d56942f1dbf68c9a2d2c1ddad55ddac2d5a68ce4cb79a9e97ad8a0f1"}
Jul 28 14:08:09.082033 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:09.081801    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t"
Jul 28 14:08:09.106808 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:09.106753    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t" podStartSLOduration=3.295259495 podStartE2EDuration="7.106739178s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.811479683s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:08:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:08:09.106001431 +0000 UTC m=+4485.891876554" watchObservedRunningTime="2026-07-28 14:08:09.106739178 +0000 UTC m=+4485.892614351"
Jul 28 14:08:10.085894 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:10.085857    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t"
Jul 28 14:08:10.087152 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:10.087124    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t" podUID="048f90cd-ed22-45df-8809-27bce99cc953" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Jul 28 14:08:11.090082 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:11.090041    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t" podUID="048f90cd-ed22-45df-8809-27bce99cc953" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Jul 28 14:08:16.095728 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:16.095630    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t"
Jul 28 14:08:16.096250 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:16.096223    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t" podUID="048f90cd-ed22-45df-8809-27bce99cc953" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Jul 28 14:08:26.096150 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:26.096110    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t" podUID="048f90cd-ed22-45df-8809-27bce99cc953" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Jul 28 14:08:36.096968 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:36.096919    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t" podUID="048f90cd-ed22-45df-8809-27bce99cc953" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Jul 28 14:08:46.096353 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:46.096313    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t" podUID="048f90cd-ed22-45df-8809-27bce99cc953" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Jul 28 14:08:56.096350 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:08:56.096308    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t" podUID="048f90cd-ed22-45df-8809-27bce99cc953" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Jul 28 14:09:06.096509 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:06.096468    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t" podUID="048f90cd-ed22-45df-8809-27bce99cc953" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Jul 28 14:09:16.096672 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:16.096627    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t" podUID="048f90cd-ed22-45df-8809-27bce99cc953" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Jul 28 14:09:19.801545 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:19.801519    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t"
Jul 28 14:09:22.986387 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:22.986333    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t"]
Jul 28 14:09:22.986820 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:22.986588    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t" podUID="048f90cd-ed22-45df-8809-27bce99cc953" containerName="kserve-container" containerID="cri-o://806a86b5d56942f1dbf68c9a2d2c1ddad55ddac2d5a68ce4cb79a9e97ad8a0f1" gracePeriod=30
Jul 28 14:09:22.986820 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:22.986637    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t" podUID="048f90cd-ed22-45df-8809-27bce99cc953" containerName="kube-rbac-proxy" containerID="cri-o://378c4d132bfac2db3445187438449125196c271c4c2ee195b7e6164882dd1368" gracePeriod=30
Jul 28 14:09:23.380391 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:23.380306    2554 generic.go:356] "Generic (PLEG): container finished" podID="048f90cd-ed22-45df-8809-27bce99cc953" containerID="378c4d132bfac2db3445187438449125196c271c4c2ee195b7e6164882dd1368" exitCode=2
Jul 28 14:09:23.380538 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:23.380383    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t" event={"ID":"048f90cd-ed22-45df-8809-27bce99cc953","Type":"ContainerDied","Data":"378c4d132bfac2db3445187438449125196c271c4c2ee195b7e6164882dd1368"}
Jul 28 14:09:26.091031 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:26.090987    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t" podUID="048f90cd-ed22-45df-8809-27bce99cc953" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.55:8643/healthz\": dial tcp 10.134.0.55:8643: connect: connection refused"
Jul 28 14:09:27.627879 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:27.627853    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t"
Jul 28 14:09:27.680244 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:27.680171    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/048f90cd-ed22-45df-8809-27bce99cc953-kube-rbac-proxy-sar-config\") pod \"048f90cd-ed22-45df-8809-27bce99cc953\" (UID: \"048f90cd-ed22-45df-8809-27bce99cc953\") "
Jul 28 14:09:27.680244 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:27.680211    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7vvqd\" (UniqueName: \"kubernetes.io/projected/048f90cd-ed22-45df-8809-27bce99cc953-kube-api-access-7vvqd\") pod \"048f90cd-ed22-45df-8809-27bce99cc953\" (UID: \"048f90cd-ed22-45df-8809-27bce99cc953\") "
Jul 28 14:09:27.680467 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:27.680255    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/048f90cd-ed22-45df-8809-27bce99cc953-cabundle-cert\") pod \"048f90cd-ed22-45df-8809-27bce99cc953\" (UID: \"048f90cd-ed22-45df-8809-27bce99cc953\") "
Jul 28 14:09:27.680467 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:27.680298    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/048f90cd-ed22-45df-8809-27bce99cc953-kserve-provision-location\") pod \"048f90cd-ed22-45df-8809-27bce99cc953\" (UID: \"048f90cd-ed22-45df-8809-27bce99cc953\") "
Jul 28 14:09:27.680467 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:27.680340    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/048f90cd-ed22-45df-8809-27bce99cc953-proxy-tls\") pod \"048f90cd-ed22-45df-8809-27bce99cc953\" (UID: \"048f90cd-ed22-45df-8809-27bce99cc953\") "
Jul 28 14:09:27.680635 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:27.680617    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/048f90cd-ed22-45df-8809-27bce99cc953-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "048f90cd-ed22-45df-8809-27bce99cc953" (UID: "048f90cd-ed22-45df-8809-27bce99cc953"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:09:27.680687 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:27.680614    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/048f90cd-ed22-45df-8809-27bce99cc953-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "048f90cd-ed22-45df-8809-27bce99cc953" (UID: "048f90cd-ed22-45df-8809-27bce99cc953"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:09:27.680687 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:27.680656    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/048f90cd-ed22-45df-8809-27bce99cc953-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "048f90cd-ed22-45df-8809-27bce99cc953" (UID: "048f90cd-ed22-45df-8809-27bce99cc953"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:09:27.682520 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:27.682493    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/048f90cd-ed22-45df-8809-27bce99cc953-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "048f90cd-ed22-45df-8809-27bce99cc953" (UID: "048f90cd-ed22-45df-8809-27bce99cc953"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:09:27.682632 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:27.682527    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/048f90cd-ed22-45df-8809-27bce99cc953-kube-api-access-7vvqd" (OuterVolumeSpecName: "kube-api-access-7vvqd") pod "048f90cd-ed22-45df-8809-27bce99cc953" (UID: "048f90cd-ed22-45df-8809-27bce99cc953"). InnerVolumeSpecName "kube-api-access-7vvqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:09:27.781905 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:27.781869    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/048f90cd-ed22-45df-8809-27bce99cc953-kserve-provision-location\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:09:27.781905 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:27.781902    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/048f90cd-ed22-45df-8809-27bce99cc953-proxy-tls\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:09:27.781905 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:27.781915    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/048f90cd-ed22-45df-8809-27bce99cc953-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:09:27.782165 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:27.781924    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vvqd\" (UniqueName: \"kubernetes.io/projected/048f90cd-ed22-45df-8809-27bce99cc953-kube-api-access-7vvqd\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:09:27.782165 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:27.781933    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/048f90cd-ed22-45df-8809-27bce99cc953-cabundle-cert\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:09:28.404672 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:28.404640    2554 generic.go:356] "Generic (PLEG): container finished" podID="048f90cd-ed22-45df-8809-27bce99cc953" containerID="806a86b5d56942f1dbf68c9a2d2c1ddad55ddac2d5a68ce4cb79a9e97ad8a0f1" exitCode=0
Jul 28 14:09:28.404840 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:28.404715    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t"
Jul 28 14:09:28.404840 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:28.404720    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t" event={"ID":"048f90cd-ed22-45df-8809-27bce99cc953","Type":"ContainerDied","Data":"806a86b5d56942f1dbf68c9a2d2c1ddad55ddac2d5a68ce4cb79a9e97ad8a0f1"}
Jul 28 14:09:28.404840 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:28.404820    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t" event={"ID":"048f90cd-ed22-45df-8809-27bce99cc953","Type":"ContainerDied","Data":"29e038ab33c3227f8c9eeb94cc06d8827f6ead14f5f50ca6beda457224dbe1ec"}
Jul 28 14:09:28.404945 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:28.404860    2554 scope.go:117] "RemoveContainer" containerID="378c4d132bfac2db3445187438449125196c271c4c2ee195b7e6164882dd1368"
Jul 28 14:09:28.412911 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:28.412894    2554 scope.go:117] "RemoveContainer" containerID="806a86b5d56942f1dbf68c9a2d2c1ddad55ddac2d5a68ce4cb79a9e97ad8a0f1"
Jul 28 14:09:28.420237 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:28.420218    2554 scope.go:117] "RemoveContainer" containerID="208f76c601603800f75bb7fd1e4ec4ccdb48c88aeae8527c03ccd190ed2ec617"
Jul 28 14:09:28.427082 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:28.427060    2554 scope.go:117] "RemoveContainer" containerID="378c4d132bfac2db3445187438449125196c271c4c2ee195b7e6164882dd1368"
Jul 28 14:09:28.427344 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:09:28.427325    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"378c4d132bfac2db3445187438449125196c271c4c2ee195b7e6164882dd1368\": container with ID starting with 378c4d132bfac2db3445187438449125196c271c4c2ee195b7e6164882dd1368 not found: ID does not exist" containerID="378c4d132bfac2db3445187438449125196c271c4c2ee195b7e6164882dd1368"
Jul 28 14:09:28.427412 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:28.427354    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"378c4d132bfac2db3445187438449125196c271c4c2ee195b7e6164882dd1368"} err="failed to get container status \"378c4d132bfac2db3445187438449125196c271c4c2ee195b7e6164882dd1368\": rpc error: code = NotFound desc = could not find container \"378c4d132bfac2db3445187438449125196c271c4c2ee195b7e6164882dd1368\": container with ID starting with 378c4d132bfac2db3445187438449125196c271c4c2ee195b7e6164882dd1368 not found: ID does not exist"
Jul 28 14:09:28.427412 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:28.427372    2554 scope.go:117] "RemoveContainer" containerID="806a86b5d56942f1dbf68c9a2d2c1ddad55ddac2d5a68ce4cb79a9e97ad8a0f1"
Jul 28 14:09:28.427609 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:09:28.427583    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"806a86b5d56942f1dbf68c9a2d2c1ddad55ddac2d5a68ce4cb79a9e97ad8a0f1\": container with ID starting with 806a86b5d56942f1dbf68c9a2d2c1ddad55ddac2d5a68ce4cb79a9e97ad8a0f1 not found: ID does not exist" containerID="806a86b5d56942f1dbf68c9a2d2c1ddad55ddac2d5a68ce4cb79a9e97ad8a0f1"
Jul 28 14:09:28.427657 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:28.427616    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"806a86b5d56942f1dbf68c9a2d2c1ddad55ddac2d5a68ce4cb79a9e97ad8a0f1"} err="failed to get container status \"806a86b5d56942f1dbf68c9a2d2c1ddad55ddac2d5a68ce4cb79a9e97ad8a0f1\": rpc error: code = NotFound desc = could not find container \"806a86b5d56942f1dbf68c9a2d2c1ddad55ddac2d5a68ce4cb79a9e97ad8a0f1\": container with ID starting with 806a86b5d56942f1dbf68c9a2d2c1ddad55ddac2d5a68ce4cb79a9e97ad8a0f1 not found: ID does not exist"
Jul 28 14:09:28.427657 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:28.427631    2554 scope.go:117] "RemoveContainer" containerID="208f76c601603800f75bb7fd1e4ec4ccdb48c88aeae8527c03ccd190ed2ec617"
Jul 28 14:09:28.427864 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:09:28.427848    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"208f76c601603800f75bb7fd1e4ec4ccdb48c88aeae8527c03ccd190ed2ec617\": container with ID starting with 208f76c601603800f75bb7fd1e4ec4ccdb48c88aeae8527c03ccd190ed2ec617 not found: ID does not exist" containerID="208f76c601603800f75bb7fd1e4ec4ccdb48c88aeae8527c03ccd190ed2ec617"
Jul 28 14:09:28.427902 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:28.427870    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"208f76c601603800f75bb7fd1e4ec4ccdb48c88aeae8527c03ccd190ed2ec617"} err="failed to get container status \"208f76c601603800f75bb7fd1e4ec4ccdb48c88aeae8527c03ccd190ed2ec617\": rpc error: code = NotFound desc = could not find container \"208f76c601603800f75bb7fd1e4ec4ccdb48c88aeae8527c03ccd190ed2ec617\": container with ID starting with 208f76c601603800f75bb7fd1e4ec4ccdb48c88aeae8527c03ccd190ed2ec617 not found: ID does not exist"
Jul 28 14:09:28.433695 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:28.433674    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t"]
Jul 28 14:09:28.437388 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:28.437368    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-9vb2t"]
Jul 28 14:09:29.801580 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:29.801550    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="048f90cd-ed22-45df-8809-27bce99cc953" path="/var/lib/kubelet/pods/048f90cd-ed22-45df-8809-27bce99cc953/volumes"
Jul 28 14:09:48.186533 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:48.186444    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-tngzb"]
Jul 28 14:09:48.186955 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:48.186765    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="048f90cd-ed22-45df-8809-27bce99cc953" containerName="kserve-container"
Jul 28 14:09:48.186955 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:48.186777    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="048f90cd-ed22-45df-8809-27bce99cc953" containerName="kserve-container"
Jul 28 14:09:48.186955 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:48.186797    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="048f90cd-ed22-45df-8809-27bce99cc953" containerName="storage-initializer"
Jul 28 14:09:48.186955 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:48.186806    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="048f90cd-ed22-45df-8809-27bce99cc953" containerName="storage-initializer"
Jul 28 14:09:48.186955 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:48.186814    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="048f90cd-ed22-45df-8809-27bce99cc953" containerName="kube-rbac-proxy"
Jul 28 14:09:48.186955 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:48.186820    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="048f90cd-ed22-45df-8809-27bce99cc953" containerName="kube-rbac-proxy"
Jul 28 14:09:48.186955 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:48.186876    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="048f90cd-ed22-45df-8809-27bce99cc953" containerName="kserve-container"
Jul 28 14:09:48.186955 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:48.186888    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="048f90cd-ed22-45df-8809-27bce99cc953" containerName="kube-rbac-proxy"
Jul 28 14:09:48.190089 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:48.190069    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-tngzb"
Jul 28 14:09:48.197014 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:48.196988    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-predictor-serving-cert\""
Jul 28 14:09:48.197115 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:48.197030    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 14:09:48.197115 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:48.197056    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 14:09:48.197115 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:48.197083    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 14:09:48.197300 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:48.197118    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
Jul 28 14:09:48.197300 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:48.197152    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rxfwd\""
Jul 28 14:09:48.197483 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:48.197458    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 14:09:48.207696 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:48.207673    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-tngzb"]
Jul 28 14:09:48.353700 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:48.353665    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a386c451-fd83-41f3-8913-e4d11d70e758-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-tngzb\" (UID: \"a386c451-fd83-41f3-8913-e4d11d70e758\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-tngzb"
Jul 28 14:09:48.353887 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:48.353713    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qncs\" (UniqueName: \"kubernetes.io/projected/a386c451-fd83-41f3-8913-e4d11d70e758-kube-api-access-6qncs\") pod \"isvc-tensorflow-predictor-547dfc5669-tngzb\" (UID: \"a386c451-fd83-41f3-8913-e4d11d70e758\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-tngzb"
Jul 28 14:09:48.353887 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:48.353803    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a386c451-fd83-41f3-8913-e4d11d70e758-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-tngzb\" (UID: \"a386c451-fd83-41f3-8913-e4d11d70e758\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-tngzb"
Jul 28 14:09:48.353887 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:48.353880    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a386c451-fd83-41f3-8913-e4d11d70e758-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-tngzb\" (UID: \"a386c451-fd83-41f3-8913-e4d11d70e758\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-tngzb"
Jul 28 14:09:48.354028 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:48.353899    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a386c451-fd83-41f3-8913-e4d11d70e758-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-tngzb\" (UID: \"a386c451-fd83-41f3-8913-e4d11d70e758\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-tngzb"
Jul 28 14:09:48.454953 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:48.454923    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a386c451-fd83-41f3-8913-e4d11d70e758-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-tngzb\" (UID: \"a386c451-fd83-41f3-8913-e4d11d70e758\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-tngzb"
Jul 28 14:09:48.455110 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:48.454979    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a386c451-fd83-41f3-8913-e4d11d70e758-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-tngzb\" (UID: \"a386c451-fd83-41f3-8913-e4d11d70e758\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-tngzb"
Jul 28 14:09:48.455110 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:48.454998    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a386c451-fd83-41f3-8913-e4d11d70e758-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-tngzb\" (UID: \"a386c451-fd83-41f3-8913-e4d11d70e758\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-tngzb"
Jul 28 14:09:48.455110 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:48.455021    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a386c451-fd83-41f3-8913-e4d11d70e758-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-tngzb\" (UID: \"a386c451-fd83-41f3-8913-e4d11d70e758\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-tngzb"
Jul 28 14:09:48.455110 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:48.455042    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qncs\" (UniqueName: \"kubernetes.io/projected/a386c451-fd83-41f3-8913-e4d11d70e758-kube-api-access-6qncs\") pod \"isvc-tensorflow-predictor-547dfc5669-tngzb\" (UID: \"a386c451-fd83-41f3-8913-e4d11d70e758\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-tngzb"
Jul 28 14:09:48.455524 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:48.455503    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a386c451-fd83-41f3-8913-e4d11d70e758-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-tngzb\" (UID: \"a386c451-fd83-41f3-8913-e4d11d70e758\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-tngzb"
Jul 28 14:09:48.455724 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:48.455706    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a386c451-fd83-41f3-8913-e4d11d70e758-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-tngzb\" (UID: \"a386c451-fd83-41f3-8913-e4d11d70e758\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-tngzb"
Jul 28 14:09:48.455775 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:48.455722    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a386c451-fd83-41f3-8913-e4d11d70e758-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-tngzb\" (UID: \"a386c451-fd83-41f3-8913-e4d11d70e758\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-tngzb"
Jul 28 14:09:48.457756 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:48.457735    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a386c451-fd83-41f3-8913-e4d11d70e758-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-tngzb\" (UID: \"a386c451-fd83-41f3-8913-e4d11d70e758\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-tngzb"
Jul 28 14:09:48.463712 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:48.463692    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qncs\" (UniqueName: \"kubernetes.io/projected/a386c451-fd83-41f3-8913-e4d11d70e758-kube-api-access-6qncs\") pod \"isvc-tensorflow-predictor-547dfc5669-tngzb\" (UID: \"a386c451-fd83-41f3-8913-e4d11d70e758\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-tngzb"
Jul 28 14:09:48.500822 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:48.500800    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-tngzb"
Jul 28 14:09:48.636976 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:48.636944    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-tngzb"]
Jul 28 14:09:48.640289 ip-10-0-129-177 kubenswrapper[2554]: W0728 14:09:48.640254    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda386c451_fd83_41f3_8913_e4d11d70e758.slice/crio-7197a514a55e8c250718b95d36ad40a405d569d32c2986f6e6cdc1bd0cde10fb WatchSource:0}: Error finding container 7197a514a55e8c250718b95d36ad40a405d569d32c2986f6e6cdc1bd0cde10fb: Status 404 returned error can't find the container with id 7197a514a55e8c250718b95d36ad40a405d569d32c2986f6e6cdc1bd0cde10fb
Jul 28 14:09:49.487937 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:49.487899    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-tngzb" event={"ID":"a386c451-fd83-41f3-8913-e4d11d70e758","Type":"ContainerStarted","Data":"beacedd0098b9ae69336ab277fa426f5ccd458e8cbe5026dd1aa9a608c068b04"}
Jul 28 14:09:49.488357 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:49.488008    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-tngzb" event={"ID":"a386c451-fd83-41f3-8913-e4d11d70e758","Type":"ContainerStarted","Data":"7197a514a55e8c250718b95d36ad40a405d569d32c2986f6e6cdc1bd0cde10fb"}
Jul 28 14:09:53.507872 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:53.507774    2554 generic.go:356] "Generic (PLEG): container finished" podID="a386c451-fd83-41f3-8913-e4d11d70e758" containerID="beacedd0098b9ae69336ab277fa426f5ccd458e8cbe5026dd1aa9a608c068b04" exitCode=0
Jul 28 14:09:53.507872 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:53.507858    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-tngzb" event={"ID":"a386c451-fd83-41f3-8913-e4d11d70e758","Type":"ContainerDied","Data":"beacedd0098b9ae69336ab277fa426f5ccd458e8cbe5026dd1aa9a608c068b04"}
Jul 28 14:09:53.509158 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:53.509143    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 14:09:57.530177 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:57.530090    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-tngzb" event={"ID":"a386c451-fd83-41f3-8913-e4d11d70e758","Type":"ContainerStarted","Data":"db4845abc3c8b7f6e9bfadd187ca4b66de9861bff5838264a236ed6c4178f25e"}
Jul 28 14:09:57.530177 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:57.530152    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-tngzb" event={"ID":"a386c451-fd83-41f3-8913-e4d11d70e758","Type":"ContainerStarted","Data":"ca2e5955d72eb5535810a452d853515ba421e5f1b709a53b528df9ec0c8d5e0d"}
Jul 28 14:09:57.530640 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:57.530415    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-tngzb"
Jul 28 14:09:57.552481 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:57.552438    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-tngzb" podStartSLOduration=1.263857933 podStartE2EDuration="9.552424466s" totalImagesPullingTime="3.740590267s" totalInitContainerRuntime="4.547976266s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:09:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:09:57.551157581 +0000 UTC m=+4594.337032689" watchObservedRunningTime="2026-07-28 14:09:57.552424466 +0000 UTC m=+4594.338299576"
Jul 28 14:09:58.534761 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:58.534728    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-tngzb"
Jul 28 14:09:58.536061 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:58.536033    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-tngzb" podUID="a386c451-fd83-41f3-8913-e4d11d70e758" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Jul 28 14:09:59.538821 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:09:59.538783    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-tngzb" podUID="a386c451-fd83-41f3-8913-e4d11d70e758" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Jul 28 14:10:04.543801 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:10:04.543773    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-tngzb"
Jul 28 14:10:04.544245 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:10:04.544220    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-tngzb" podUID="a386c451-fd83-41f3-8913-e4d11d70e758" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Jul 28 14:10:14.544838 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:10:14.544812    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-tngzb"
Jul 28 14:10:28.678695 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:10:28.678663    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-tngzb"]
Jul 28 14:10:28.679101 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:10:28.678908    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-tngzb" podUID="a386c451-fd83-41f3-8913-e4d11d70e758" containerName="kserve-container" containerID="cri-o://ca2e5955d72eb5535810a452d853515ba421e5f1b709a53b528df9ec0c8d5e0d" gracePeriod=30
Jul 28 14:10:28.679101 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:10:28.678950    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-tngzb" podUID="a386c451-fd83-41f3-8913-e4d11d70e758" containerName="kube-rbac-proxy" containerID="cri-o://db4845abc3c8b7f6e9bfadd187ca4b66de9861bff5838264a236ed6c4178f25e" gracePeriod=30
Jul 28 14:10:29.539407 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:10:29.539367    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-tngzb" podUID="a386c451-fd83-41f3-8913-e4d11d70e758" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.56:8643/healthz\": dial tcp 10.134.0.56:8643: connect: connection refused"
Jul 28 14:10:29.664132 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:10:29.664097    2554 generic.go:356] "Generic (PLEG): container finished" podID="a386c451-fd83-41f3-8913-e4d11d70e758" containerID="db4845abc3c8b7f6e9bfadd187ca4b66de9861bff5838264a236ed6c4178f25e" exitCode=2
Jul 28 14:10:29.664308 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:10:29.664157    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-tngzb" event={"ID":"a386c451-fd83-41f3-8913-e4d11d70e758","Type":"ContainerDied","Data":"db4845abc3c8b7f6e9bfadd187ca4b66de9861bff5838264a236ed6c4178f25e"}
Jul 28 14:10:34.539644 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:10:34.539597    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-tngzb" podUID="a386c451-fd83-41f3-8913-e4d11d70e758" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.56:8643/healthz\": dial tcp 10.134.0.56:8643: connect: connection refused"
Jul 28 14:10:39.539217 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:10:39.539177    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-tngzb" podUID="a386c451-fd83-41f3-8913-e4d11d70e758" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.56:8643/healthz\": dial tcp 10.134.0.56:8643: connect: connection refused"
Jul 28 14:10:39.539620 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:10:39.539358    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-tngzb"
Jul 28 14:10:44.539306 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:10:44.539249    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-tngzb" podUID="a386c451-fd83-41f3-8913-e4d11d70e758" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.56:8643/healthz\": dial tcp 10.134.0.56:8643: connect: connection refused"
Jul 28 14:10:49.539173 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:10:49.539133    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-tngzb" podUID="a386c451-fd83-41f3-8913-e4d11d70e758" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.56:8643/healthz\": dial tcp 10.134.0.56:8643: connect: connection refused"
Jul 28 14:10:54.539590 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:10:54.539550    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-tngzb" podUID="a386c451-fd83-41f3-8913-e4d11d70e758" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.56:8643/healthz\": dial tcp 10.134.0.56:8643: connect: connection refused"
Jul 28 14:10:58.783948 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:10:58.783911    2554 generic.go:356] "Generic (PLEG): container finished" podID="a386c451-fd83-41f3-8913-e4d11d70e758" containerID="ca2e5955d72eb5535810a452d853515ba421e5f1b709a53b528df9ec0c8d5e0d" exitCode=137
Jul 28 14:10:58.784312 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:10:58.783983    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-tngzb" event={"ID":"a386c451-fd83-41f3-8913-e4d11d70e758","Type":"ContainerDied","Data":"ca2e5955d72eb5535810a452d853515ba421e5f1b709a53b528df9ec0c8d5e0d"}
Jul 28 14:10:59.330250 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:10:59.330228    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-tngzb"
Jul 28 14:10:59.436089 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:10:59.436053    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a386c451-fd83-41f3-8913-e4d11d70e758-kserve-provision-location\") pod \"a386c451-fd83-41f3-8913-e4d11d70e758\" (UID: \"a386c451-fd83-41f3-8913-e4d11d70e758\") "
Jul 28 14:10:59.436248 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:10:59.436108    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6qncs\" (UniqueName: \"kubernetes.io/projected/a386c451-fd83-41f3-8913-e4d11d70e758-kube-api-access-6qncs\") pod \"a386c451-fd83-41f3-8913-e4d11d70e758\" (UID: \"a386c451-fd83-41f3-8913-e4d11d70e758\") "
Jul 28 14:10:59.436248 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:10:59.436140    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a386c451-fd83-41f3-8913-e4d11d70e758-proxy-tls\") pod \"a386c451-fd83-41f3-8913-e4d11d70e758\" (UID: \"a386c451-fd83-41f3-8913-e4d11d70e758\") "
Jul 28 14:10:59.436248 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:10:59.436158    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a386c451-fd83-41f3-8913-e4d11d70e758-kube-rbac-proxy-sar-config\") pod \"a386c451-fd83-41f3-8913-e4d11d70e758\" (UID: \"a386c451-fd83-41f3-8913-e4d11d70e758\") "
Jul 28 14:10:59.436248 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:10:59.436173    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a386c451-fd83-41f3-8913-e4d11d70e758-cabundle-cert\") pod \"a386c451-fd83-41f3-8913-e4d11d70e758\" (UID: \"a386c451-fd83-41f3-8913-e4d11d70e758\") "
Jul 28 14:10:59.436604 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:10:59.436574    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a386c451-fd83-41f3-8913-e4d11d70e758-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a386c451-fd83-41f3-8913-e4d11d70e758" (UID: "a386c451-fd83-41f3-8913-e4d11d70e758"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:10:59.436686 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:10:59.436578    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a386c451-fd83-41f3-8913-e4d11d70e758-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a386c451-fd83-41f3-8913-e4d11d70e758" (UID: "a386c451-fd83-41f3-8913-e4d11d70e758"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:10:59.438457 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:10:59.438428    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a386c451-fd83-41f3-8913-e4d11d70e758-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a386c451-fd83-41f3-8913-e4d11d70e758" (UID: "a386c451-fd83-41f3-8913-e4d11d70e758"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:10:59.438559 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:10:59.438453    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a386c451-fd83-41f3-8913-e4d11d70e758-kube-api-access-6qncs" (OuterVolumeSpecName: "kube-api-access-6qncs") pod "a386c451-fd83-41f3-8913-e4d11d70e758" (UID: "a386c451-fd83-41f3-8913-e4d11d70e758"). InnerVolumeSpecName "kube-api-access-6qncs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:10:59.446991 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:10:59.446941    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a386c451-fd83-41f3-8913-e4d11d70e758-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a386c451-fd83-41f3-8913-e4d11d70e758" (UID: "a386c451-fd83-41f3-8913-e4d11d70e758"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:10:59.536965 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:10:59.536932    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a386c451-fd83-41f3-8913-e4d11d70e758-kserve-provision-location\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:10:59.537096 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:10:59.536969    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qncs\" (UniqueName: \"kubernetes.io/projected/a386c451-fd83-41f3-8913-e4d11d70e758-kube-api-access-6qncs\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:10:59.537096 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:10:59.536981    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a386c451-fd83-41f3-8913-e4d11d70e758-proxy-tls\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:10:59.537096 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:10:59.536989    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a386c451-fd83-41f3-8913-e4d11d70e758-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:10:59.537096 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:10:59.536998    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a386c451-fd83-41f3-8913-e4d11d70e758-cabundle-cert\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:10:59.790428 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:10:59.790394    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-tngzb" event={"ID":"a386c451-fd83-41f3-8913-e4d11d70e758","Type":"ContainerDied","Data":"7197a514a55e8c250718b95d36ad40a405d569d32c2986f6e6cdc1bd0cde10fb"}
Jul 28 14:10:59.790428 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:10:59.790415    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-tngzb"
Jul 28 14:10:59.790834 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:10:59.790475    2554 scope.go:117] "RemoveContainer" containerID="db4845abc3c8b7f6e9bfadd187ca4b66de9861bff5838264a236ed6c4178f25e"
Jul 28 14:10:59.798712 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:10:59.798664    2554 scope.go:117] "RemoveContainer" containerID="ca2e5955d72eb5535810a452d853515ba421e5f1b709a53b528df9ec0c8d5e0d"
Jul 28 14:10:59.806236 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:10:59.806220    2554 scope.go:117] "RemoveContainer" containerID="beacedd0098b9ae69336ab277fa426f5ccd458e8cbe5026dd1aa9a608c068b04"
Jul 28 14:10:59.820260 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:10:59.820239    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-tngzb"]
Jul 28 14:10:59.826988 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:10:59.826969    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-tngzb"]
Jul 28 14:11:01.801094 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:11:01.801056    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a386c451-fd83-41f3-8913-e4d11d70e758" path="/var/lib/kubelet/pods/a386c451-fd83-41f3-8913-e4d11d70e758/volumes"
Jul 28 14:11:18.492012 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:11:18.491925    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-kngj9"]
Jul 28 14:11:18.492571 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:11:18.492434    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a386c451-fd83-41f3-8913-e4d11d70e758" containerName="kserve-container"
Jul 28 14:11:18.492571 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:11:18.492455    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a386c451-fd83-41f3-8913-e4d11d70e758" containerName="kserve-container"
Jul 28 14:11:18.492571 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:11:18.492469    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a386c451-fd83-41f3-8913-e4d11d70e758" containerName="storage-initializer"
Jul 28 14:11:18.492571 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:11:18.492478    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a386c451-fd83-41f3-8913-e4d11d70e758" containerName="storage-initializer"
Jul 28 14:11:18.492571 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:11:18.492499    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a386c451-fd83-41f3-8913-e4d11d70e758" containerName="kube-rbac-proxy"
Jul 28 14:11:18.492571 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:11:18.492507    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a386c451-fd83-41f3-8913-e4d11d70e758" containerName="kube-rbac-proxy"
Jul 28 14:11:18.492891 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:11:18.492580    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="a386c451-fd83-41f3-8913-e4d11d70e758" containerName="kube-rbac-proxy"
Jul 28 14:11:18.492891 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:11:18.492593    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="a386c451-fd83-41f3-8913-e4d11d70e758" containerName="kserve-container"
Jul 28 14:11:18.495928 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:11:18.495908    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-kngj9"
Jul 28 14:11:18.498531 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:11:18.498515    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 14:11:18.498620 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:11:18.498515    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-kube-rbac-proxy-sar-config\""
Jul 28 14:11:18.498620 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:11:18.498515    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 14:11:18.499344 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:11:18.499330    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 14:11:18.499508 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:11:18.499491    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rxfwd\""
Jul 28 14:11:18.499568 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:11:18.499523    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-predictor-serving-cert\""
Jul 28 14:11:18.499625 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:11:18.499577    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 14:11:18.505584 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:11:18.505564    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-kngj9"]
Jul 28 14:11:18.577724 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:11:18.577687    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79eaa522-f069-4f1b-a399-242268782d37-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-kngj9\" (UID: \"79eaa522-f069-4f1b-a399-242268782d37\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-kngj9"
Jul 28 14:11:18.577724 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:11:18.577721    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79eaa522-f069-4f1b-a399-242268782d37-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-kngj9\" (UID: \"79eaa522-f069-4f1b-a399-242268782d37\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-kngj9"
Jul 28 14:11:18.577928 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:11:18.577777    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79eaa522-f069-4f1b-a399-242268782d37-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-kngj9\" (UID: \"79eaa522-f069-4f1b-a399-242268782d37\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-kngj9"
Jul 28 14:11:18.577928 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:11:18.577803    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79eaa522-f069-4f1b-a399-242268782d37-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-kngj9\" (UID: \"79eaa522-f069-4f1b-a399-242268782d37\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-kngj9"
Jul 28 14:11:18.577928 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:11:18.577830    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb4tj\" (UniqueName: \"kubernetes.io/projected/79eaa522-f069-4f1b-a399-242268782d37-kube-api-access-pb4tj\") pod \"isvc-triton-predictor-5d7db58676-kngj9\" (UID: \"79eaa522-f069-4f1b-a399-242268782d37\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-kngj9"
Jul 28 14:11:18.679002 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:11:18.678967    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79eaa522-f069-4f1b-a399-242268782d37-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-kngj9\" (UID: \"79eaa522-f069-4f1b-a399-242268782d37\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-kngj9"
Jul 28 14:11:18.679002 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:11:18.679006    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79eaa522-f069-4f1b-a399-242268782d37-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-kngj9\" (UID: \"79eaa522-f069-4f1b-a399-242268782d37\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-kngj9"
Jul 28 14:11:18.679216 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:11:18.679027    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb4tj\" (UniqueName: \"kubernetes.io/projected/79eaa522-f069-4f1b-a399-242268782d37-kube-api-access-pb4tj\") pod \"isvc-triton-predictor-5d7db58676-kngj9\" (UID: \"79eaa522-f069-4f1b-a399-242268782d37\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-kngj9"
Jul 28 14:11:18.679216 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:11:18.679074    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79eaa522-f069-4f1b-a399-242268782d37-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-kngj9\" (UID: \"79eaa522-f069-4f1b-a399-242268782d37\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-kngj9"
Jul 28 14:11:18.679216 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:11:18.679093    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79eaa522-f069-4f1b-a399-242268782d37-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-kngj9\" (UID: \"79eaa522-f069-4f1b-a399-242268782d37\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-kngj9"
Jul 28 14:11:18.679471 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:11:18.679448    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79eaa522-f069-4f1b-a399-242268782d37-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-kngj9\" (UID: \"79eaa522-f069-4f1b-a399-242268782d37\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-kngj9"
Jul 28 14:11:18.679825 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:11:18.679808    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79eaa522-f069-4f1b-a399-242268782d37-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-kngj9\" (UID: \"79eaa522-f069-4f1b-a399-242268782d37\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-kngj9"
Jul 28 14:11:18.679865 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:11:18.679808    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79eaa522-f069-4f1b-a399-242268782d37-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-kngj9\" (UID: \"79eaa522-f069-4f1b-a399-242268782d37\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-kngj9"
Jul 28 14:11:18.681663 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:11:18.681647    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79eaa522-f069-4f1b-a399-242268782d37-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-kngj9\" (UID: \"79eaa522-f069-4f1b-a399-242268782d37\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-kngj9"
Jul 28 14:11:18.687893 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:11:18.687861    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb4tj\" (UniqueName: \"kubernetes.io/projected/79eaa522-f069-4f1b-a399-242268782d37-kube-api-access-pb4tj\") pod \"isvc-triton-predictor-5d7db58676-kngj9\" (UID: \"79eaa522-f069-4f1b-a399-242268782d37\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-kngj9"
Jul 28 14:11:18.806667 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:11:18.806576    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-kngj9"
Jul 28 14:11:18.940830 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:11:18.940799    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-kngj9"]
Jul 28 14:11:18.943849 ip-10-0-129-177 kubenswrapper[2554]: W0728 14:11:18.943811    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79eaa522_f069_4f1b_a399_242268782d37.slice/crio-7c6f168fb66a40bb574a1fa053ac2263ebddd0576741f182974388c9eb57f8ec WatchSource:0}: Error finding container 7c6f168fb66a40bb574a1fa053ac2263ebddd0576741f182974388c9eb57f8ec: Status 404 returned error can't find the container with id 7c6f168fb66a40bb574a1fa053ac2263ebddd0576741f182974388c9eb57f8ec
Jul 28 14:11:19.871387 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:11:19.871347    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-kngj9" event={"ID":"79eaa522-f069-4f1b-a399-242268782d37","Type":"ContainerStarted","Data":"947e1f5e33c099cb2294ddbdf349b471d3f4dbb5ace6eac271d4c410b04860db"}
Jul 28 14:11:19.871769 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:11:19.871417    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-kngj9" event={"ID":"79eaa522-f069-4f1b-a399-242268782d37","Type":"ContainerStarted","Data":"7c6f168fb66a40bb574a1fa053ac2263ebddd0576741f182974388c9eb57f8ec"}
Jul 28 14:11:23.890441 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:11:23.890402    2554 generic.go:356] "Generic (PLEG): container finished" podID="79eaa522-f069-4f1b-a399-242268782d37" containerID="947e1f5e33c099cb2294ddbdf349b471d3f4dbb5ace6eac271d4c410b04860db" exitCode=0
Jul 28 14:11:23.890851 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:11:23.890477    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-kngj9" event={"ID":"79eaa522-f069-4f1b-a399-242268782d37","Type":"ContainerDied","Data":"947e1f5e33c099cb2294ddbdf349b471d3f4dbb5ace6eac271d4c410b04860db"}
Jul 28 14:13:19.483027 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:19.482985    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-kngj9" event={"ID":"79eaa522-f069-4f1b-a399-242268782d37","Type":"ContainerStarted","Data":"8621aeeb1417a8b3e6c09fe439731cc1a59389a6ec1205fb92431f9a479f5c30"}
Jul 28 14:13:19.483561 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:19.483052    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-kngj9" event={"ID":"79eaa522-f069-4f1b-a399-242268782d37","Type":"ContainerStarted","Data":"2009eb47b9b233edfc4ce27e5de6b6da9e92b8552bf2da0424ae03cff98d9dba"}
Jul 28 14:13:19.483561 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:19.483116    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-kngj9"
Jul 28 14:13:19.518075 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:19.518026    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-kngj9" podStartSLOduration=2.588471032 podStartE2EDuration="2m1.518010588s" totalImagesPullingTime="1m54.952420717s" totalInitContainerRuntime="3.977118839s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:11:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:13:19.516568923 +0000 UTC m=+4796.302444055" watchObservedRunningTime="2026-07-28 14:13:19.518010588 +0000 UTC m=+4796.303885699"
Jul 28 14:13:20.487066 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:20.487035    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-kngj9"
Jul 28 14:13:20.488012 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:20.487986    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-kngj9" podUID="79eaa522-f069-4f1b-a399-242268782d37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Jul 28 14:13:21.491919 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:21.491863    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-kngj9" podUID="79eaa522-f069-4f1b-a399-242268782d37" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Jul 28 14:13:26.496699 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:26.496665    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-kngj9"
Jul 28 14:13:26.497402 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:26.497384    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-kngj9"
Jul 28 14:13:41.098422 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:41.098374    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-kngj9"]
Jul 28 14:13:41.098968 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:41.098698    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-kngj9" podUID="79eaa522-f069-4f1b-a399-242268782d37" containerName="kserve-container" containerID="cri-o://2009eb47b9b233edfc4ce27e5de6b6da9e92b8552bf2da0424ae03cff98d9dba" gracePeriod=30
Jul 28 14:13:41.098968 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:41.098868    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-kngj9" podUID="79eaa522-f069-4f1b-a399-242268782d37" containerName="kube-rbac-proxy" containerID="cri-o://8621aeeb1417a8b3e6c09fe439731cc1a59389a6ec1205fb92431f9a479f5c30" gracePeriod=30
Jul 28 14:13:41.492315 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:41.492255    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-kngj9" podUID="79eaa522-f069-4f1b-a399-242268782d37" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.57:8643/healthz\": dial tcp 10.134.0.57:8643: connect: connection refused"
Jul 28 14:13:41.571297 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:41.571243    2554 generic.go:356] "Generic (PLEG): container finished" podID="79eaa522-f069-4f1b-a399-242268782d37" containerID="8621aeeb1417a8b3e6c09fe439731cc1a59389a6ec1205fb92431f9a479f5c30" exitCode=2
Jul 28 14:13:41.571446 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:41.571312    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-kngj9" event={"ID":"79eaa522-f069-4f1b-a399-242268782d37","Type":"ContainerDied","Data":"8621aeeb1417a8b3e6c09fe439731cc1a59389a6ec1205fb92431f9a479f5c30"}
Jul 28 14:13:44.336357 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:44.336330    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-kngj9"
Jul 28 14:13:44.449307 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:44.449261    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79eaa522-f069-4f1b-a399-242268782d37-kube-rbac-proxy-sar-config\") pod \"79eaa522-f069-4f1b-a399-242268782d37\" (UID: \"79eaa522-f069-4f1b-a399-242268782d37\") "
Jul 28 14:13:44.449465 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:44.449383    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pb4tj\" (UniqueName: \"kubernetes.io/projected/79eaa522-f069-4f1b-a399-242268782d37-kube-api-access-pb4tj\") pod \"79eaa522-f069-4f1b-a399-242268782d37\" (UID: \"79eaa522-f069-4f1b-a399-242268782d37\") "
Jul 28 14:13:44.449465 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:44.449445    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79eaa522-f069-4f1b-a399-242268782d37-kserve-provision-location\") pod \"79eaa522-f069-4f1b-a399-242268782d37\" (UID: \"79eaa522-f069-4f1b-a399-242268782d37\") "
Jul 28 14:13:44.449565 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:44.449473    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79eaa522-f069-4f1b-a399-242268782d37-cabundle-cert\") pod \"79eaa522-f069-4f1b-a399-242268782d37\" (UID: \"79eaa522-f069-4f1b-a399-242268782d37\") "
Jul 28 14:13:44.449565 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:44.449529    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79eaa522-f069-4f1b-a399-242268782d37-proxy-tls\") pod \"79eaa522-f069-4f1b-a399-242268782d37\" (UID: \"79eaa522-f069-4f1b-a399-242268782d37\") "
Jul 28 14:13:44.449676 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:44.449660    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79eaa522-f069-4f1b-a399-242268782d37-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "79eaa522-f069-4f1b-a399-242268782d37" (UID: "79eaa522-f069-4f1b-a399-242268782d37"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:13:44.449817 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:44.449789    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79eaa522-f069-4f1b-a399-242268782d37-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "79eaa522-f069-4f1b-a399-242268782d37" (UID: "79eaa522-f069-4f1b-a399-242268782d37"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:13:44.449817 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:44.449814    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79eaa522-f069-4f1b-a399-242268782d37-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:13:44.449958 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:44.449860    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79eaa522-f069-4f1b-a399-242268782d37-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "79eaa522-f069-4f1b-a399-242268782d37" (UID: "79eaa522-f069-4f1b-a399-242268782d37"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:13:44.451669 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:44.451641    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79eaa522-f069-4f1b-a399-242268782d37-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "79eaa522-f069-4f1b-a399-242268782d37" (UID: "79eaa522-f069-4f1b-a399-242268782d37"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:13:44.451799 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:44.451782    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79eaa522-f069-4f1b-a399-242268782d37-kube-api-access-pb4tj" (OuterVolumeSpecName: "kube-api-access-pb4tj") pod "79eaa522-f069-4f1b-a399-242268782d37" (UID: "79eaa522-f069-4f1b-a399-242268782d37"). InnerVolumeSpecName "kube-api-access-pb4tj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:13:44.550988 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:44.550960    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79eaa522-f069-4f1b-a399-242268782d37-kserve-provision-location\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:13:44.550988 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:44.550985    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79eaa522-f069-4f1b-a399-242268782d37-cabundle-cert\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:13:44.550988 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:44.550995    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79eaa522-f069-4f1b-a399-242268782d37-proxy-tls\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:13:44.551213 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:44.551005    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pb4tj\" (UniqueName: \"kubernetes.io/projected/79eaa522-f069-4f1b-a399-242268782d37-kube-api-access-pb4tj\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:13:44.584874 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:44.584840    2554 generic.go:356] "Generic (PLEG): container finished" podID="79eaa522-f069-4f1b-a399-242268782d37" containerID="2009eb47b9b233edfc4ce27e5de6b6da9e92b8552bf2da0424ae03cff98d9dba" exitCode=0
Jul 28 14:13:44.585011 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:44.584899    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-kngj9"
Jul 28 14:13:44.585011 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:44.584912    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-kngj9" event={"ID":"79eaa522-f069-4f1b-a399-242268782d37","Type":"ContainerDied","Data":"2009eb47b9b233edfc4ce27e5de6b6da9e92b8552bf2da0424ae03cff98d9dba"}
Jul 28 14:13:44.585011 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:44.584976    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-kngj9" event={"ID":"79eaa522-f069-4f1b-a399-242268782d37","Type":"ContainerDied","Data":"7c6f168fb66a40bb574a1fa053ac2263ebddd0576741f182974388c9eb57f8ec"}
Jul 28 14:13:44.585185 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:44.585013    2554 scope.go:117] "RemoveContainer" containerID="8621aeeb1417a8b3e6c09fe439731cc1a59389a6ec1205fb92431f9a479f5c30"
Jul 28 14:13:44.593655 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:44.593639    2554 scope.go:117] "RemoveContainer" containerID="2009eb47b9b233edfc4ce27e5de6b6da9e92b8552bf2da0424ae03cff98d9dba"
Jul 28 14:13:44.600885 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:44.600868    2554 scope.go:117] "RemoveContainer" containerID="947e1f5e33c099cb2294ddbdf349b471d3f4dbb5ace6eac271d4c410b04860db"
Jul 28 14:13:44.608564 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:44.608549    2554 scope.go:117] "RemoveContainer" containerID="8621aeeb1417a8b3e6c09fe439731cc1a59389a6ec1205fb92431f9a479f5c30"
Jul 28 14:13:44.608830 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:13:44.608800    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8621aeeb1417a8b3e6c09fe439731cc1a59389a6ec1205fb92431f9a479f5c30\": container with ID starting with 8621aeeb1417a8b3e6c09fe439731cc1a59389a6ec1205fb92431f9a479f5c30 not found: ID does not exist" containerID="8621aeeb1417a8b3e6c09fe439731cc1a59389a6ec1205fb92431f9a479f5c30"
Jul 28 14:13:44.608920 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:44.608831    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8621aeeb1417a8b3e6c09fe439731cc1a59389a6ec1205fb92431f9a479f5c30"} err="failed to get container status \"8621aeeb1417a8b3e6c09fe439731cc1a59389a6ec1205fb92431f9a479f5c30\": rpc error: code = NotFound desc = could not find container \"8621aeeb1417a8b3e6c09fe439731cc1a59389a6ec1205fb92431f9a479f5c30\": container with ID starting with 8621aeeb1417a8b3e6c09fe439731cc1a59389a6ec1205fb92431f9a479f5c30 not found: ID does not exist"
Jul 28 14:13:44.608920 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:44.608847    2554 scope.go:117] "RemoveContainer" containerID="2009eb47b9b233edfc4ce27e5de6b6da9e92b8552bf2da0424ae03cff98d9dba"
Jul 28 14:13:44.609069 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:13:44.609052    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2009eb47b9b233edfc4ce27e5de6b6da9e92b8552bf2da0424ae03cff98d9dba\": container with ID starting with 2009eb47b9b233edfc4ce27e5de6b6da9e92b8552bf2da0424ae03cff98d9dba not found: ID does not exist" containerID="2009eb47b9b233edfc4ce27e5de6b6da9e92b8552bf2da0424ae03cff98d9dba"
Jul 28 14:13:44.609108 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:44.609075    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2009eb47b9b233edfc4ce27e5de6b6da9e92b8552bf2da0424ae03cff98d9dba"} err="failed to get container status \"2009eb47b9b233edfc4ce27e5de6b6da9e92b8552bf2da0424ae03cff98d9dba\": rpc error: code = NotFound desc = could not find container \"2009eb47b9b233edfc4ce27e5de6b6da9e92b8552bf2da0424ae03cff98d9dba\": container with ID starting with 2009eb47b9b233edfc4ce27e5de6b6da9e92b8552bf2da0424ae03cff98d9dba not found: ID does not exist"
Jul 28 14:13:44.609108 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:44.609088    2554 scope.go:117] "RemoveContainer" containerID="947e1f5e33c099cb2294ddbdf349b471d3f4dbb5ace6eac271d4c410b04860db"
Jul 28 14:13:44.609341 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:13:44.609313    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"947e1f5e33c099cb2294ddbdf349b471d3f4dbb5ace6eac271d4c410b04860db\": container with ID starting with 947e1f5e33c099cb2294ddbdf349b471d3f4dbb5ace6eac271d4c410b04860db not found: ID does not exist" containerID="947e1f5e33c099cb2294ddbdf349b471d3f4dbb5ace6eac271d4c410b04860db"
Jul 28 14:13:44.609451 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:44.609350    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"947e1f5e33c099cb2294ddbdf349b471d3f4dbb5ace6eac271d4c410b04860db"} err="failed to get container status \"947e1f5e33c099cb2294ddbdf349b471d3f4dbb5ace6eac271d4c410b04860db\": rpc error: code = NotFound desc = could not find container \"947e1f5e33c099cb2294ddbdf349b471d3f4dbb5ace6eac271d4c410b04860db\": container with ID starting with 947e1f5e33c099cb2294ddbdf349b471d3f4dbb5ace6eac271d4c410b04860db not found: ID does not exist"
Jul 28 14:13:44.609573 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:44.609528    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-kngj9"]
Jul 28 14:13:44.611420 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:44.611403    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-kngj9"]
Jul 28 14:13:45.801216 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:45.801186    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79eaa522-f069-4f1b-a399-242268782d37" path="/var/lib/kubelet/pods/79eaa522-f069-4f1b-a399-242268782d37/volumes"
Jul 28 14:13:46.254936 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:46.254905    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6fzcg"]
Jul 28 14:13:46.255231 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:46.255218    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79eaa522-f069-4f1b-a399-242268782d37" containerName="kube-rbac-proxy"
Jul 28 14:13:46.255324 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:46.255235    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="79eaa522-f069-4f1b-a399-242268782d37" containerName="kube-rbac-proxy"
Jul 28 14:13:46.255324 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:46.255253    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79eaa522-f069-4f1b-a399-242268782d37" containerName="kserve-container"
Jul 28 14:13:46.255324 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:46.255262    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="79eaa522-f069-4f1b-a399-242268782d37" containerName="kserve-container"
Jul 28 14:13:46.255324 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:46.255300    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79eaa522-f069-4f1b-a399-242268782d37" containerName="storage-initializer"
Jul 28 14:13:46.255324 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:46.255310    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="79eaa522-f069-4f1b-a399-242268782d37" containerName="storage-initializer"
Jul 28 14:13:46.255675 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:46.255423    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="79eaa522-f069-4f1b-a399-242268782d37" containerName="kube-rbac-proxy"
Jul 28 14:13:46.255675 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:46.255439    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="79eaa522-f069-4f1b-a399-242268782d37" containerName="kserve-container"
Jul 28 14:13:46.266423 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:46.266400    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6fzcg"
Jul 28 14:13:46.269301 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:46.269261    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 14:13:46.269430 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:46.269268    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 14:13:46.269430 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:46.269288    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 14:13:46.269430 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:46.269376    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 14:13:46.269655 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:46.269634    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
Jul 28 14:13:46.270229 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:46.270209    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rxfwd\""
Jul 28 14:13:46.270415 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:46.270252    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\""
Jul 28 14:13:46.272091 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:46.272072    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6fzcg"]
Jul 28 14:13:46.367709 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:46.367674    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be163698-d313-407f-9f10-67905b11e8d7-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-6fzcg\" (UID: \"be163698-d313-407f-9f10-67905b11e8d7\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6fzcg"
Jul 28 14:13:46.367870 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:46.367718    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be163698-d313-407f-9f10-67905b11e8d7-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-6fzcg\" (UID: \"be163698-d313-407f-9f10-67905b11e8d7\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6fzcg"
Jul 28 14:13:46.367870 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:46.367751    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be163698-d313-407f-9f10-67905b11e8d7-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-6fzcg\" (UID: \"be163698-d313-407f-9f10-67905b11e8d7\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6fzcg"
Jul 28 14:13:46.367870 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:46.367795    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be163698-d313-407f-9f10-67905b11e8d7-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-6fzcg\" (UID: \"be163698-d313-407f-9f10-67905b11e8d7\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6fzcg"
Jul 28 14:13:46.367870 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:46.367847    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5f69\" (UniqueName: \"kubernetes.io/projected/be163698-d313-407f-9f10-67905b11e8d7-kube-api-access-p5f69\") pod \"isvc-xgboost-predictor-7566d65bf7-6fzcg\" (UID: \"be163698-d313-407f-9f10-67905b11e8d7\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6fzcg"
Jul 28 14:13:46.469200 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:46.469165    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be163698-d313-407f-9f10-67905b11e8d7-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-6fzcg\" (UID: \"be163698-d313-407f-9f10-67905b11e8d7\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6fzcg"
Jul 28 14:13:46.469370 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:46.469212    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be163698-d313-407f-9f10-67905b11e8d7-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-6fzcg\" (UID: \"be163698-d313-407f-9f10-67905b11e8d7\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6fzcg"
Jul 28 14:13:46.469430 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:46.469410    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be163698-d313-407f-9f10-67905b11e8d7-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-6fzcg\" (UID: \"be163698-d313-407f-9f10-67905b11e8d7\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6fzcg"
Jul 28 14:13:46.469499 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:46.469484    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5f69\" (UniqueName: \"kubernetes.io/projected/be163698-d313-407f-9f10-67905b11e8d7-kube-api-access-p5f69\") pod \"isvc-xgboost-predictor-7566d65bf7-6fzcg\" (UID: \"be163698-d313-407f-9f10-67905b11e8d7\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6fzcg"
Jul 28 14:13:46.469572 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:46.469556    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be163698-d313-407f-9f10-67905b11e8d7-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-6fzcg\" (UID: \"be163698-d313-407f-9f10-67905b11e8d7\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6fzcg"
Jul 28 14:13:46.469697 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:46.469671    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be163698-d313-407f-9f10-67905b11e8d7-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-6fzcg\" (UID: \"be163698-d313-407f-9f10-67905b11e8d7\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6fzcg"
Jul 28 14:13:46.470028 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:46.469997    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be163698-d313-407f-9f10-67905b11e8d7-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-6fzcg\" (UID: \"be163698-d313-407f-9f10-67905b11e8d7\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6fzcg"
Jul 28 14:13:46.470231 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:46.470138    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be163698-d313-407f-9f10-67905b11e8d7-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-6fzcg\" (UID: \"be163698-d313-407f-9f10-67905b11e8d7\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6fzcg"
Jul 28 14:13:46.471949 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:46.471928    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be163698-d313-407f-9f10-67905b11e8d7-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-6fzcg\" (UID: \"be163698-d313-407f-9f10-67905b11e8d7\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6fzcg"
Jul 28 14:13:46.478298 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:46.478255    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5f69\" (UniqueName: \"kubernetes.io/projected/be163698-d313-407f-9f10-67905b11e8d7-kube-api-access-p5f69\") pod \"isvc-xgboost-predictor-7566d65bf7-6fzcg\" (UID: \"be163698-d313-407f-9f10-67905b11e8d7\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6fzcg"
Jul 28 14:13:46.578636 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:46.578562    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6fzcg"
Jul 28 14:13:46.755056 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:46.755032    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6fzcg"]
Jul 28 14:13:46.757932 ip-10-0-129-177 kubenswrapper[2554]: W0728 14:13:46.757909    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe163698_d313_407f_9f10_67905b11e8d7.slice/crio-4ac6d6098fe21f9a8d425c6c47ac888840fe5acbdad599c3d8123e9a01f584a0 WatchSource:0}: Error finding container 4ac6d6098fe21f9a8d425c6c47ac888840fe5acbdad599c3d8123e9a01f584a0: Status 404 returned error can't find the container with id 4ac6d6098fe21f9a8d425c6c47ac888840fe5acbdad599c3d8123e9a01f584a0
Jul 28 14:13:47.599817 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:47.599778    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6fzcg" event={"ID":"be163698-d313-407f-9f10-67905b11e8d7","Type":"ContainerStarted","Data":"3bdab887fc848b3744da22bec35bdb2a46411aba67da7505452a6bbfc17a6c84"}
Jul 28 14:13:47.600385 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:47.599846    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6fzcg" event={"ID":"be163698-d313-407f-9f10-67905b11e8d7","Type":"ContainerStarted","Data":"4ac6d6098fe21f9a8d425c6c47ac888840fe5acbdad599c3d8123e9a01f584a0"}
Jul 28 14:13:51.617366 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:51.617330    2554 generic.go:356] "Generic (PLEG): container finished" podID="be163698-d313-407f-9f10-67905b11e8d7" containerID="3bdab887fc848b3744da22bec35bdb2a46411aba67da7505452a6bbfc17a6c84" exitCode=0
Jul 28 14:13:51.617761 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:13:51.617382    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6fzcg" event={"ID":"be163698-d313-407f-9f10-67905b11e8d7","Type":"ContainerDied","Data":"3bdab887fc848b3744da22bec35bdb2a46411aba67da7505452a6bbfc17a6c84"}
Jul 28 14:14:10.712678 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:14:10.712644    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6fzcg" event={"ID":"be163698-d313-407f-9f10-67905b11e8d7","Type":"ContainerStarted","Data":"04bf2572c852ea05b76ca3961e0e262685ed28c081fdcf5aba884062e6334c9c"}
Jul 28 14:14:10.713079 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:14:10.712704    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6fzcg" event={"ID":"be163698-d313-407f-9f10-67905b11e8d7","Type":"ContainerStarted","Data":"f82366e3225dca7bd1803b1d52f85461d9acd31054ca1965bcb94d409a4f237e"}
Jul 28 14:14:10.713079 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:14:10.712985    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6fzcg"
Jul 28 14:14:10.713184 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:14:10.713146    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6fzcg"
Jul 28 14:14:10.714177 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:14:10.714153    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6fzcg" podUID="be163698-d313-407f-9f10-67905b11e8d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Jul 28 14:14:10.740370 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:14:10.740328    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6fzcg" podStartSLOduration=2.589216596 podStartE2EDuration="24.740313238s" totalImagesPullingTime="18.174298507s" totalInitContainerRuntime="3.976798135s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:13:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:14:10.738480021 +0000 UTC m=+4847.524355151" watchObservedRunningTime="2026-07-28 14:14:10.740313238 +0000 UTC m=+4847.526188350"
Jul 28 14:14:11.716710 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:14:11.716668    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6fzcg" podUID="be163698-d313-407f-9f10-67905b11e8d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Jul 28 14:14:16.721548 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:14:16.721480    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6fzcg"
Jul 28 14:14:16.721989 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:14:16.721955    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6fzcg" podUID="be163698-d313-407f-9f10-67905b11e8d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Jul 28 14:14:26.721929 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:14:26.721887    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6fzcg" podUID="be163698-d313-407f-9f10-67905b11e8d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Jul 28 14:14:36.722251 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:14:36.722211    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6fzcg" podUID="be163698-d313-407f-9f10-67905b11e8d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Jul 28 14:14:46.722390 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:14:46.722351    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6fzcg" podUID="be163698-d313-407f-9f10-67905b11e8d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Jul 28 14:14:56.722919 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:14:56.722878    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6fzcg" podUID="be163698-d313-407f-9f10-67905b11e8d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Jul 28 14:15:06.723428 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:06.723393    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6fzcg"
Jul 28 14:15:16.470130 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:16.470092    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6fzcg"]
Jul 28 14:15:16.470638 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:16.470350    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6fzcg" podUID="be163698-d313-407f-9f10-67905b11e8d7" containerName="kserve-container" containerID="cri-o://f82366e3225dca7bd1803b1d52f85461d9acd31054ca1965bcb94d409a4f237e" gracePeriod=30
Jul 28 14:15:16.470638 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:16.470420    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6fzcg" podUID="be163698-d313-407f-9f10-67905b11e8d7" containerName="kube-rbac-proxy" containerID="cri-o://04bf2572c852ea05b76ca3961e0e262685ed28c081fdcf5aba884062e6334c9c" gracePeriod=30
Jul 28 14:15:16.717495 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:16.717451    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6fzcg" podUID="be163698-d313-407f-9f10-67905b11e8d7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.58:8643/healthz\": dial tcp 10.134.0.58:8643: connect: connection refused"
Jul 28 14:15:16.722907 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:16.722841    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6fzcg" podUID="be163698-d313-407f-9f10-67905b11e8d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Jul 28 14:15:16.987895 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:16.987809    2554 generic.go:356] "Generic (PLEG): container finished" podID="be163698-d313-407f-9f10-67905b11e8d7" containerID="04bf2572c852ea05b76ca3961e0e262685ed28c081fdcf5aba884062e6334c9c" exitCode=2
Jul 28 14:15:16.988036 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:16.987890    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6fzcg" event={"ID":"be163698-d313-407f-9f10-67905b11e8d7","Type":"ContainerDied","Data":"04bf2572c852ea05b76ca3961e0e262685ed28c081fdcf5aba884062e6334c9c"}
Jul 28 14:15:20.004481 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:20.004453    2554 generic.go:356] "Generic (PLEG): container finished" podID="be163698-d313-407f-9f10-67905b11e8d7" containerID="f82366e3225dca7bd1803b1d52f85461d9acd31054ca1965bcb94d409a4f237e" exitCode=0
Jul 28 14:15:20.004862 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:20.004492    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6fzcg" event={"ID":"be163698-d313-407f-9f10-67905b11e8d7","Type":"ContainerDied","Data":"f82366e3225dca7bd1803b1d52f85461d9acd31054ca1965bcb94d409a4f237e"}
Jul 28 14:15:20.018447 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:20.018426    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6fzcg"
Jul 28 14:15:20.056637 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:20.056574    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be163698-d313-407f-9f10-67905b11e8d7-kube-rbac-proxy-sar-config\") pod \"be163698-d313-407f-9f10-67905b11e8d7\" (UID: \"be163698-d313-407f-9f10-67905b11e8d7\") "
Jul 28 14:15:20.056637 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:20.056608    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be163698-d313-407f-9f10-67905b11e8d7-kserve-provision-location\") pod \"be163698-d313-407f-9f10-67905b11e8d7\" (UID: \"be163698-d313-407f-9f10-67905b11e8d7\") "
Jul 28 14:15:20.056637 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:20.056636    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be163698-d313-407f-9f10-67905b11e8d7-cabundle-cert\") pod \"be163698-d313-407f-9f10-67905b11e8d7\" (UID: \"be163698-d313-407f-9f10-67905b11e8d7\") "
Jul 28 14:15:20.056882 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:20.056670    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be163698-d313-407f-9f10-67905b11e8d7-proxy-tls\") pod \"be163698-d313-407f-9f10-67905b11e8d7\" (UID: \"be163698-d313-407f-9f10-67905b11e8d7\") "
Jul 28 14:15:20.056882 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:20.056717    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p5f69\" (UniqueName: \"kubernetes.io/projected/be163698-d313-407f-9f10-67905b11e8d7-kube-api-access-p5f69\") pod \"be163698-d313-407f-9f10-67905b11e8d7\" (UID: \"be163698-d313-407f-9f10-67905b11e8d7\") "
Jul 28 14:15:20.056990 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:20.056933    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be163698-d313-407f-9f10-67905b11e8d7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "be163698-d313-407f-9f10-67905b11e8d7" (UID: "be163698-d313-407f-9f10-67905b11e8d7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:15:20.056990 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:20.056957    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be163698-d313-407f-9f10-67905b11e8d7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "be163698-d313-407f-9f10-67905b11e8d7" (UID: "be163698-d313-407f-9f10-67905b11e8d7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:15:20.057102 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:20.057061    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be163698-d313-407f-9f10-67905b11e8d7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "be163698-d313-407f-9f10-67905b11e8d7" (UID: "be163698-d313-407f-9f10-67905b11e8d7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:15:20.058967 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:20.058942    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be163698-d313-407f-9f10-67905b11e8d7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "be163698-d313-407f-9f10-67905b11e8d7" (UID: "be163698-d313-407f-9f10-67905b11e8d7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:15:20.059058 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:20.059003    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be163698-d313-407f-9f10-67905b11e8d7-kube-api-access-p5f69" (OuterVolumeSpecName: "kube-api-access-p5f69") pod "be163698-d313-407f-9f10-67905b11e8d7" (UID: "be163698-d313-407f-9f10-67905b11e8d7"). InnerVolumeSpecName "kube-api-access-p5f69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:15:20.157282 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:20.157251    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be163698-d313-407f-9f10-67905b11e8d7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:15:20.157426 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:20.157296    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be163698-d313-407f-9f10-67905b11e8d7-kserve-provision-location\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:15:20.157426 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:20.157307    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/be163698-d313-407f-9f10-67905b11e8d7-cabundle-cert\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:15:20.157426 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:20.157318    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be163698-d313-407f-9f10-67905b11e8d7-proxy-tls\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:15:20.157426 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:20.157326    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5f69\" (UniqueName: \"kubernetes.io/projected/be163698-d313-407f-9f10-67905b11e8d7-kube-api-access-p5f69\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:15:21.011185 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:21.011154    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6fzcg"
Jul 28 14:15:21.011682 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:21.011152    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6fzcg" event={"ID":"be163698-d313-407f-9f10-67905b11e8d7","Type":"ContainerDied","Data":"4ac6d6098fe21f9a8d425c6c47ac888840fe5acbdad599c3d8123e9a01f584a0"}
Jul 28 14:15:21.011682 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:21.011332    2554 scope.go:117] "RemoveContainer" containerID="04bf2572c852ea05b76ca3961e0e262685ed28c081fdcf5aba884062e6334c9c"
Jul 28 14:15:21.020203 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:21.020186    2554 scope.go:117] "RemoveContainer" containerID="f82366e3225dca7bd1803b1d52f85461d9acd31054ca1965bcb94d409a4f237e"
Jul 28 14:15:21.027476 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:21.027456    2554 scope.go:117] "RemoveContainer" containerID="3bdab887fc848b3744da22bec35bdb2a46411aba67da7505452a6bbfc17a6c84"
Jul 28 14:15:21.035097 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:21.035077    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6fzcg"]
Jul 28 14:15:21.039441 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:21.039418    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6fzcg"]
Jul 28 14:15:21.634097 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:21.634060    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-gvvzk"]
Jul 28 14:15:21.634468 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:21.634449    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be163698-d313-407f-9f10-67905b11e8d7" containerName="storage-initializer"
Jul 28 14:15:21.634468 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:21.634469    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="be163698-d313-407f-9f10-67905b11e8d7" containerName="storage-initializer"
Jul 28 14:15:21.634602 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:21.634488    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be163698-d313-407f-9f10-67905b11e8d7" containerName="kube-rbac-proxy"
Jul 28 14:15:21.634602 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:21.634493    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="be163698-d313-407f-9f10-67905b11e8d7" containerName="kube-rbac-proxy"
Jul 28 14:15:21.634602 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:21.634500    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be163698-d313-407f-9f10-67905b11e8d7" containerName="kserve-container"
Jul 28 14:15:21.634602 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:21.634507    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="be163698-d313-407f-9f10-67905b11e8d7" containerName="kserve-container"
Jul 28 14:15:21.634602 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:21.634598    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="be163698-d313-407f-9f10-67905b11e8d7" containerName="kserve-container"
Jul 28 14:15:21.634864 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:21.634612    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="be163698-d313-407f-9f10-67905b11e8d7" containerName="kube-rbac-proxy"
Jul 28 14:15:21.639251 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:21.639230    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-gvvzk"
Jul 28 14:15:21.642670 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:21.642650    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 14:15:21.642782 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:21.642749    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 14:15:21.643237 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:21.643216    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 14:15:21.643410 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:21.643304    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rxfwd\""
Jul 28 14:15:21.643410 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:21.643313    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Jul 28 14:15:21.643410 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:21.643324    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 14:15:21.643893 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:21.643875    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
Jul 28 14:15:21.649418 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:21.649396    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-gvvzk"]
Jul 28 14:15:21.769876 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:21.769844    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d1e67e6e-9932-4097-9820-68866162135b-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-gvvzk\" (UID: \"d1e67e6e-9932-4097-9820-68866162135b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-gvvzk"
Jul 28 14:15:21.770029 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:21.769899    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1e67e6e-9932-4097-9820-68866162135b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-gvvzk\" (UID: \"d1e67e6e-9932-4097-9820-68866162135b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-gvvzk"
Jul 28 14:15:21.770029 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:21.769964    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1e67e6e-9932-4097-9820-68866162135b-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-gvvzk\" (UID: \"d1e67e6e-9932-4097-9820-68866162135b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-gvvzk"
Jul 28 14:15:21.770110 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:21.770027    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1e67e6e-9932-4097-9820-68866162135b-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-gvvzk\" (UID: \"d1e67e6e-9932-4097-9820-68866162135b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-gvvzk"
Jul 28 14:15:21.770110 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:21.770056    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f97p\" (UniqueName: \"kubernetes.io/projected/d1e67e6e-9932-4097-9820-68866162135b-kube-api-access-9f97p\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-gvvzk\" (UID: \"d1e67e6e-9932-4097-9820-68866162135b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-gvvzk"
Jul 28 14:15:21.801150 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:21.801123    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be163698-d313-407f-9f10-67905b11e8d7" path="/var/lib/kubelet/pods/be163698-d313-407f-9f10-67905b11e8d7/volumes"
Jul 28 14:15:21.871058 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:21.871025    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f97p\" (UniqueName: \"kubernetes.io/projected/d1e67e6e-9932-4097-9820-68866162135b-kube-api-access-9f97p\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-gvvzk\" (UID: \"d1e67e6e-9932-4097-9820-68866162135b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-gvvzk"
Jul 28 14:15:21.871193 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:21.871064    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d1e67e6e-9932-4097-9820-68866162135b-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-gvvzk\" (UID: \"d1e67e6e-9932-4097-9820-68866162135b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-gvvzk"
Jul 28 14:15:21.871193 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:21.871105    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1e67e6e-9932-4097-9820-68866162135b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-gvvzk\" (UID: \"d1e67e6e-9932-4097-9820-68866162135b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-gvvzk"
Jul 28 14:15:21.871193 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:21.871130    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1e67e6e-9932-4097-9820-68866162135b-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-gvvzk\" (UID: \"d1e67e6e-9932-4097-9820-68866162135b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-gvvzk"
Jul 28 14:15:21.871193 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:21.871152    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1e67e6e-9932-4097-9820-68866162135b-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-gvvzk\" (UID: \"d1e67e6e-9932-4097-9820-68866162135b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-gvvzk"
Jul 28 14:15:21.871541 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:21.871525    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1e67e6e-9932-4097-9820-68866162135b-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-gvvzk\" (UID: \"d1e67e6e-9932-4097-9820-68866162135b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-gvvzk"
Jul 28 14:15:21.871972 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:21.871948    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d1e67e6e-9932-4097-9820-68866162135b-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-gvvzk\" (UID: \"d1e67e6e-9932-4097-9820-68866162135b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-gvvzk"
Jul 28 14:15:21.872065 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:21.871948    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1e67e6e-9932-4097-9820-68866162135b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-gvvzk\" (UID: \"d1e67e6e-9932-4097-9820-68866162135b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-gvvzk"
Jul 28 14:15:21.873692 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:21.873667    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1e67e6e-9932-4097-9820-68866162135b-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-gvvzk\" (UID: \"d1e67e6e-9932-4097-9820-68866162135b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-gvvzk"
Jul 28 14:15:21.883725 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:21.883703    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f97p\" (UniqueName: \"kubernetes.io/projected/d1e67e6e-9932-4097-9820-68866162135b-kube-api-access-9f97p\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-gvvzk\" (UID: \"d1e67e6e-9932-4097-9820-68866162135b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-gvvzk"
Jul 28 14:15:21.950730 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:21.950709    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-gvvzk"
Jul 28 14:15:22.103548 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:22.103236    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-gvvzk"]
Jul 28 14:15:22.106079 ip-10-0-129-177 kubenswrapper[2554]: W0728 14:15:22.106039    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1e67e6e_9932_4097_9820_68866162135b.slice/crio-f0f572c498749daecb7f0523d17d0bd316cbc918382378a7e0bf84dbde248c15 WatchSource:0}: Error finding container f0f572c498749daecb7f0523d17d0bd316cbc918382378a7e0bf84dbde248c15: Status 404 returned error can't find the container with id f0f572c498749daecb7f0523d17d0bd316cbc918382378a7e0bf84dbde248c15
Jul 28 14:15:23.025345 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:23.025310    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-gvvzk" event={"ID":"d1e67e6e-9932-4097-9820-68866162135b","Type":"ContainerStarted","Data":"9d06fa35c6c08c0f8b67ff4ef86612d586a305c2d826838d7948bf2071bb0a0d"}
Jul 28 14:15:23.025544 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:23.025382    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-gvvzk" event={"ID":"d1e67e6e-9932-4097-9820-68866162135b","Type":"ContainerStarted","Data":"f0f572c498749daecb7f0523d17d0bd316cbc918382378a7e0bf84dbde248c15"}
Jul 28 14:15:26.039411 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:26.039377    2554 generic.go:356] "Generic (PLEG): container finished" podID="d1e67e6e-9932-4097-9820-68866162135b" containerID="9d06fa35c6c08c0f8b67ff4ef86612d586a305c2d826838d7948bf2071bb0a0d" exitCode=0
Jul 28 14:15:26.039810 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:26.039433    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-gvvzk" event={"ID":"d1e67e6e-9932-4097-9820-68866162135b","Type":"ContainerDied","Data":"9d06fa35c6c08c0f8b67ff4ef86612d586a305c2d826838d7948bf2071bb0a0d"}
Jul 28 14:15:27.044705 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:27.044666    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-gvvzk" event={"ID":"d1e67e6e-9932-4097-9820-68866162135b","Type":"ContainerStarted","Data":"82240a6b23993634d8b636e62ae2dd09f2b1969471a215057f210fbcf233a74f"}
Jul 28 14:15:27.045138 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:27.044731    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-gvvzk" event={"ID":"d1e67e6e-9932-4097-9820-68866162135b","Type":"ContainerStarted","Data":"d98b1e3dfa064dc4128420244a25d242e13a72657d73f48651f67169e18f26d2"}
Jul 28 14:15:27.045138 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:27.044937    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-gvvzk"
Jul 28 14:15:27.069749 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:27.069698    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-gvvzk" podStartSLOduration=2.554896137 podStartE2EDuration="6.069680569s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.514784432s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:15:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:15:27.067495558 +0000 UTC m=+4923.853370666" watchObservedRunningTime="2026-07-28 14:15:27.069680569 +0000 UTC m=+4923.855555679"
Jul 28 14:15:28.048800 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:28.048773    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-gvvzk"
Jul 28 14:15:34.057939 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:15:34.057907    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-gvvzk"
Jul 28 14:16:04.062438 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:04.062409    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-gvvzk"
Jul 28 14:16:11.673662 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:11.673626    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-gvvzk"]
Jul 28 14:16:11.674212 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:11.673896    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-gvvzk" podUID="d1e67e6e-9932-4097-9820-68866162135b" containerName="kserve-container" containerID="cri-o://d98b1e3dfa064dc4128420244a25d242e13a72657d73f48651f67169e18f26d2" gracePeriod=30
Jul 28 14:16:11.674212 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:11.673916    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-gvvzk" podUID="d1e67e6e-9932-4097-9820-68866162135b" containerName="kube-rbac-proxy" containerID="cri-o://82240a6b23993634d8b636e62ae2dd09f2b1969471a215057f210fbcf233a74f" gracePeriod=30
Jul 28 14:16:12.229550 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:12.229520    2554 generic.go:356] "Generic (PLEG): container finished" podID="d1e67e6e-9932-4097-9820-68866162135b" containerID="82240a6b23993634d8b636e62ae2dd09f2b1969471a215057f210fbcf233a74f" exitCode=2
Jul 28 14:16:12.229728 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:12.229599    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-gvvzk" event={"ID":"d1e67e6e-9932-4097-9820-68866162135b","Type":"ContainerDied","Data":"82240a6b23993634d8b636e62ae2dd09f2b1969471a215057f210fbcf233a74f"}
Jul 28 14:16:14.053567 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:14.053529    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-gvvzk" podUID="d1e67e6e-9932-4097-9820-68866162135b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.59:8643/healthz\": dial tcp 10.134.0.59:8643: connect: connection refused"
Jul 28 14:16:14.058627 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:14.058601    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-gvvzk" podUID="d1e67e6e-9932-4097-9820-68866162135b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.59:8080/v2/models/isvc-xgboost-v2-mlserver/ready\": dial tcp 10.134.0.59:8080: connect: connection refused"
Jul 28 14:16:16.869953 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:16.869919    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6"]
Jul 28 14:16:16.873916 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:16.873891    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6"
Jul 28 14:16:16.877054 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:16.877032    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
Jul 28 14:16:16.877348 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:16.877287    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Jul 28 14:16:16.885491 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:16.885471    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6"]
Jul 28 14:16:16.932660 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:16.932633    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9409de62-685d-4284-bb5b-81f73b39da9a-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6\" (UID: \"9409de62-685d-4284-bb5b-81f73b39da9a\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6"
Jul 28 14:16:16.932801 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:16.932673    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9409de62-685d-4284-bb5b-81f73b39da9a-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6\" (UID: \"9409de62-685d-4284-bb5b-81f73b39da9a\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6"
Jul 28 14:16:16.932801 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:16.932730    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9409de62-685d-4284-bb5b-81f73b39da9a-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6\" (UID: \"9409de62-685d-4284-bb5b-81f73b39da9a\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6"
Jul 28 14:16:16.932801 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:16.932765    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh9zw\" (UniqueName: \"kubernetes.io/projected/9409de62-685d-4284-bb5b-81f73b39da9a-kube-api-access-vh9zw\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6\" (UID: \"9409de62-685d-4284-bb5b-81f73b39da9a\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6"
Jul 28 14:16:16.932926 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:16.932808    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9409de62-685d-4284-bb5b-81f73b39da9a-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6\" (UID: \"9409de62-685d-4284-bb5b-81f73b39da9a\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6"
Jul 28 14:16:17.033663 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:17.033628    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9409de62-685d-4284-bb5b-81f73b39da9a-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6\" (UID: \"9409de62-685d-4284-bb5b-81f73b39da9a\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6"
Jul 28 14:16:17.033830 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:17.033670    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh9zw\" (UniqueName: \"kubernetes.io/projected/9409de62-685d-4284-bb5b-81f73b39da9a-kube-api-access-vh9zw\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6\" (UID: \"9409de62-685d-4284-bb5b-81f73b39da9a\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6"
Jul 28 14:16:17.033830 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:17.033722    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9409de62-685d-4284-bb5b-81f73b39da9a-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6\" (UID: \"9409de62-685d-4284-bb5b-81f73b39da9a\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6"
Jul 28 14:16:17.033830 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:17.033752    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9409de62-685d-4284-bb5b-81f73b39da9a-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6\" (UID: \"9409de62-685d-4284-bb5b-81f73b39da9a\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6"
Jul 28 14:16:17.033830 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:17.033790    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9409de62-685d-4284-bb5b-81f73b39da9a-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6\" (UID: \"9409de62-685d-4284-bb5b-81f73b39da9a\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6"
Jul 28 14:16:17.034064 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:16:17.033912    2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-serving-cert: secret "xgboost-v2-mlserver-predictor-serving-cert" not found
Jul 28 14:16:17.034064 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:16:17.033971    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9409de62-685d-4284-bb5b-81f73b39da9a-proxy-tls podName:9409de62-685d-4284-bb5b-81f73b39da9a nodeName:}" failed. No retries permitted until 2026-07-28 14:16:17.533950515 +0000 UTC m=+4974.319825618 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9409de62-685d-4284-bb5b-81f73b39da9a-proxy-tls") pod "xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6" (UID: "9409de62-685d-4284-bb5b-81f73b39da9a") : secret "xgboost-v2-mlserver-predictor-serving-cert" not found
Jul 28 14:16:17.034192 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:17.034065    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9409de62-685d-4284-bb5b-81f73b39da9a-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6\" (UID: \"9409de62-685d-4284-bb5b-81f73b39da9a\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6"
Jul 28 14:16:17.034542 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:17.034514    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9409de62-685d-4284-bb5b-81f73b39da9a-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6\" (UID: \"9409de62-685d-4284-bb5b-81f73b39da9a\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6"
Jul 28 14:16:17.034627 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:17.034559    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9409de62-685d-4284-bb5b-81f73b39da9a-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6\" (UID: \"9409de62-685d-4284-bb5b-81f73b39da9a\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6"
Jul 28 14:16:17.052836 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:17.052808    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh9zw\" (UniqueName: \"kubernetes.io/projected/9409de62-685d-4284-bb5b-81f73b39da9a-kube-api-access-vh9zw\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6\" (UID: \"9409de62-685d-4284-bb5b-81f73b39da9a\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6"
Jul 28 14:16:17.539523 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:17.539481    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9409de62-685d-4284-bb5b-81f73b39da9a-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6\" (UID: \"9409de62-685d-4284-bb5b-81f73b39da9a\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6"
Jul 28 14:16:17.541969 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:17.541948    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9409de62-685d-4284-bb5b-81f73b39da9a-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6\" (UID: \"9409de62-685d-4284-bb5b-81f73b39da9a\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6"
Jul 28 14:16:17.785232 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:17.785199    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6"
Jul 28 14:16:17.922937 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:17.922910    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6"]
Jul 28 14:16:17.923233 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:17.923215    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-gvvzk"
Jul 28 14:16:17.924874 ip-10-0-129-177 kubenswrapper[2554]: W0728 14:16:17.924849    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9409de62_685d_4284_bb5b_81f73b39da9a.slice/crio-1bb679c2fabb18ee3f20c33adf938408e644e49cf08c2d6e2ac5b52572bbc970 WatchSource:0}: Error finding container 1bb679c2fabb18ee3f20c33adf938408e644e49cf08c2d6e2ac5b52572bbc970: Status 404 returned error can't find the container with id 1bb679c2fabb18ee3f20c33adf938408e644e49cf08c2d6e2ac5b52572bbc970
Jul 28 14:16:18.043587 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:18.043554    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d1e67e6e-9932-4097-9820-68866162135b-cabundle-cert\") pod \"d1e67e6e-9932-4097-9820-68866162135b\" (UID: \"d1e67e6e-9932-4097-9820-68866162135b\") "
Jul 28 14:16:18.043734 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:18.043612    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1e67e6e-9932-4097-9820-68866162135b-proxy-tls\") pod \"d1e67e6e-9932-4097-9820-68866162135b\" (UID: \"d1e67e6e-9932-4097-9820-68866162135b\") "
Jul 28 14:16:18.043734 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:18.043649    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1e67e6e-9932-4097-9820-68866162135b-kserve-provision-location\") pod \"d1e67e6e-9932-4097-9820-68866162135b\" (UID: \"d1e67e6e-9932-4097-9820-68866162135b\") "
Jul 28 14:16:18.043846 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:18.043797    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9f97p\" (UniqueName: \"kubernetes.io/projected/d1e67e6e-9932-4097-9820-68866162135b-kube-api-access-9f97p\") pod \"d1e67e6e-9932-4097-9820-68866162135b\" (UID: \"d1e67e6e-9932-4097-9820-68866162135b\") "
Jul 28 14:16:18.043846 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:18.043838    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1e67e6e-9932-4097-9820-68866162135b-kube-rbac-proxy-sar-config\") pod \"d1e67e6e-9932-4097-9820-68866162135b\" (UID: \"d1e67e6e-9932-4097-9820-68866162135b\") "
Jul 28 14:16:18.043973 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:18.043953    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1e67e6e-9932-4097-9820-68866162135b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d1e67e6e-9932-4097-9820-68866162135b" (UID: "d1e67e6e-9932-4097-9820-68866162135b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:16:18.044031 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:18.043985    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1e67e6e-9932-4097-9820-68866162135b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d1e67e6e-9932-4097-9820-68866162135b" (UID: "d1e67e6e-9932-4097-9820-68866162135b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:16:18.044221 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:18.044202    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1e67e6e-9932-4097-9820-68866162135b-kserve-provision-location\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:16:18.044320 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:18.044226    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d1e67e6e-9932-4097-9820-68866162135b-cabundle-cert\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:16:18.044320 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:18.044305    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1e67e6e-9932-4097-9820-68866162135b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d1e67e6e-9932-4097-9820-68866162135b" (UID: "d1e67e6e-9932-4097-9820-68866162135b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:16:18.045770 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:18.045728    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1e67e6e-9932-4097-9820-68866162135b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d1e67e6e-9932-4097-9820-68866162135b" (UID: "d1e67e6e-9932-4097-9820-68866162135b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:16:18.045866 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:18.045850    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1e67e6e-9932-4097-9820-68866162135b-kube-api-access-9f97p" (OuterVolumeSpecName: "kube-api-access-9f97p") pod "d1e67e6e-9932-4097-9820-68866162135b" (UID: "d1e67e6e-9932-4097-9820-68866162135b"). InnerVolumeSpecName "kube-api-access-9f97p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:16:18.145250 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:18.145214    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1e67e6e-9932-4097-9820-68866162135b-proxy-tls\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:16:18.145250 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:18.145246    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f97p\" (UniqueName: \"kubernetes.io/projected/d1e67e6e-9932-4097-9820-68866162135b-kube-api-access-9f97p\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:16:18.145250 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:18.145255    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1e67e6e-9932-4097-9820-68866162135b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:16:18.255910 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:18.255873    2554 generic.go:356] "Generic (PLEG): container finished" podID="d1e67e6e-9932-4097-9820-68866162135b" containerID="d98b1e3dfa064dc4128420244a25d242e13a72657d73f48651f67169e18f26d2" exitCode=0
Jul 28 14:16:18.256068 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:18.255924    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-gvvzk"
Jul 28 14:16:18.256068 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:18.255957    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-gvvzk" event={"ID":"d1e67e6e-9932-4097-9820-68866162135b","Type":"ContainerDied","Data":"d98b1e3dfa064dc4128420244a25d242e13a72657d73f48651f67169e18f26d2"}
Jul 28 14:16:18.256068 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:18.256025    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-gvvzk" event={"ID":"d1e67e6e-9932-4097-9820-68866162135b","Type":"ContainerDied","Data":"f0f572c498749daecb7f0523d17d0bd316cbc918382378a7e0bf84dbde248c15"}
Jul 28 14:16:18.256068 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:18.256060    2554 scope.go:117] "RemoveContainer" containerID="82240a6b23993634d8b636e62ae2dd09f2b1969471a215057f210fbcf233a74f"
Jul 28 14:16:18.257672 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:18.257652    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6" event={"ID":"9409de62-685d-4284-bb5b-81f73b39da9a","Type":"ContainerStarted","Data":"aa05fc9daa0c6e46b1c8774d49310bc003ccb9e1f8558ca4694c3af5f58b85b7"}
Jul 28 14:16:18.257778 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:18.257703    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6" event={"ID":"9409de62-685d-4284-bb5b-81f73b39da9a","Type":"ContainerStarted","Data":"1bb679c2fabb18ee3f20c33adf938408e644e49cf08c2d6e2ac5b52572bbc970"}
Jul 28 14:16:18.264968 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:18.264952    2554 scope.go:117] "RemoveContainer" containerID="d98b1e3dfa064dc4128420244a25d242e13a72657d73f48651f67169e18f26d2"
Jul 28 14:16:18.272525 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:18.272501    2554 scope.go:117] "RemoveContainer" containerID="9d06fa35c6c08c0f8b67ff4ef86612d586a305c2d826838d7948bf2071bb0a0d"
Jul 28 14:16:18.283369 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:18.283345    2554 scope.go:117] "RemoveContainer" containerID="82240a6b23993634d8b636e62ae2dd09f2b1969471a215057f210fbcf233a74f"
Jul 28 14:16:18.284029 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:16:18.283899    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82240a6b23993634d8b636e62ae2dd09f2b1969471a215057f210fbcf233a74f\": container with ID starting with 82240a6b23993634d8b636e62ae2dd09f2b1969471a215057f210fbcf233a74f not found: ID does not exist" containerID="82240a6b23993634d8b636e62ae2dd09f2b1969471a215057f210fbcf233a74f"
Jul 28 14:16:18.284029 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:18.283935    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82240a6b23993634d8b636e62ae2dd09f2b1969471a215057f210fbcf233a74f"} err="failed to get container status \"82240a6b23993634d8b636e62ae2dd09f2b1969471a215057f210fbcf233a74f\": rpc error: code = NotFound desc = could not find container \"82240a6b23993634d8b636e62ae2dd09f2b1969471a215057f210fbcf233a74f\": container with ID starting with 82240a6b23993634d8b636e62ae2dd09f2b1969471a215057f210fbcf233a74f not found: ID does not exist"
Jul 28 14:16:18.284029 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:18.283958    2554 scope.go:117] "RemoveContainer" containerID="d98b1e3dfa064dc4128420244a25d242e13a72657d73f48651f67169e18f26d2"
Jul 28 14:16:18.285164 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:16:18.284952    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d98b1e3dfa064dc4128420244a25d242e13a72657d73f48651f67169e18f26d2\": container with ID starting with d98b1e3dfa064dc4128420244a25d242e13a72657d73f48651f67169e18f26d2 not found: ID does not exist" containerID="d98b1e3dfa064dc4128420244a25d242e13a72657d73f48651f67169e18f26d2"
Jul 28 14:16:18.285164 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:18.284985    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d98b1e3dfa064dc4128420244a25d242e13a72657d73f48651f67169e18f26d2"} err="failed to get container status \"d98b1e3dfa064dc4128420244a25d242e13a72657d73f48651f67169e18f26d2\": rpc error: code = NotFound desc = could not find container \"d98b1e3dfa064dc4128420244a25d242e13a72657d73f48651f67169e18f26d2\": container with ID starting with d98b1e3dfa064dc4128420244a25d242e13a72657d73f48651f67169e18f26d2 not found: ID does not exist"
Jul 28 14:16:18.285164 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:18.285005    2554 scope.go:117] "RemoveContainer" containerID="9d06fa35c6c08c0f8b67ff4ef86612d586a305c2d826838d7948bf2071bb0a0d"
Jul 28 14:16:18.285427 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:16:18.285411    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d06fa35c6c08c0f8b67ff4ef86612d586a305c2d826838d7948bf2071bb0a0d\": container with ID starting with 9d06fa35c6c08c0f8b67ff4ef86612d586a305c2d826838d7948bf2071bb0a0d not found: ID does not exist" containerID="9d06fa35c6c08c0f8b67ff4ef86612d586a305c2d826838d7948bf2071bb0a0d"
Jul 28 14:16:18.285475 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:18.285433    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d06fa35c6c08c0f8b67ff4ef86612d586a305c2d826838d7948bf2071bb0a0d"} err="failed to get container status \"9d06fa35c6c08c0f8b67ff4ef86612d586a305c2d826838d7948bf2071bb0a0d\": rpc error: code = NotFound desc = could not find container \"9d06fa35c6c08c0f8b67ff4ef86612d586a305c2d826838d7948bf2071bb0a0d\": container with ID starting with 9d06fa35c6c08c0f8b67ff4ef86612d586a305c2d826838d7948bf2071bb0a0d not found: ID does not exist"
Jul 28 14:16:18.291670 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:18.291651    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-gvvzk"]
Jul 28 14:16:18.296099 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:18.296052    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-gvvzk"]
Jul 28 14:16:19.801780 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:19.801746    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1e67e6e-9932-4097-9820-68866162135b" path="/var/lib/kubelet/pods/d1e67e6e-9932-4097-9820-68866162135b/volumes"
Jul 28 14:16:22.276929 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:22.276893    2554 generic.go:356] "Generic (PLEG): container finished" podID="9409de62-685d-4284-bb5b-81f73b39da9a" containerID="aa05fc9daa0c6e46b1c8774d49310bc003ccb9e1f8558ca4694c3af5f58b85b7" exitCode=0
Jul 28 14:16:22.276929 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:22.276930    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6" event={"ID":"9409de62-685d-4284-bb5b-81f73b39da9a","Type":"ContainerDied","Data":"aa05fc9daa0c6e46b1c8774d49310bc003ccb9e1f8558ca4694c3af5f58b85b7"}
Jul 28 14:16:23.283520 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:23.283488    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6" event={"ID":"9409de62-685d-4284-bb5b-81f73b39da9a","Type":"ContainerStarted","Data":"da767f16a4777851e19d63505062029c1767530ea5d94ae2b4320ecf3d99027c"}
Jul 28 14:16:23.283998 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:23.283559    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6" event={"ID":"9409de62-685d-4284-bb5b-81f73b39da9a","Type":"ContainerStarted","Data":"b6235c421ebb235d6ae7037c930af9bef0e4c7c32ffd707c8a002c68f615c3b8"}
Jul 28 14:16:23.283998 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:23.283777    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6"
Jul 28 14:16:23.304380 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:23.304333    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6" podStartSLOduration=3.728787336 podStartE2EDuration="7.304315612s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.575528276s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:16:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:16:23.303128324 +0000 UTC m=+4980.089003429" watchObservedRunningTime="2026-07-28 14:16:23.304315612 +0000 UTC m=+4980.090190722"
Jul 28 14:16:24.287497 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:24.287466    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6"
Jul 28 14:16:30.298077 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:16:30.298050    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6"
Jul 28 14:17:00.302296 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:00.302243    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6"
Jul 28 14:17:07.018915 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:07.018882    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6"]
Jul 28 14:17:07.019458 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:07.019115    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6" podUID="9409de62-685d-4284-bb5b-81f73b39da9a" containerName="kserve-container" containerID="cri-o://b6235c421ebb235d6ae7037c930af9bef0e4c7c32ffd707c8a002c68f615c3b8" gracePeriod=30
Jul 28 14:17:07.019458 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:07.019154    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6" podUID="9409de62-685d-4284-bb5b-81f73b39da9a" containerName="kube-rbac-proxy" containerID="cri-o://da767f16a4777851e19d63505062029c1767530ea5d94ae2b4320ecf3d99027c" gracePeriod=30
Jul 28 14:17:07.463901 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:07.463867    2554 generic.go:356] "Generic (PLEG): container finished" podID="9409de62-685d-4284-bb5b-81f73b39da9a" containerID="da767f16a4777851e19d63505062029c1767530ea5d94ae2b4320ecf3d99027c" exitCode=2
Jul 28 14:17:07.464069 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:07.463946    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6" event={"ID":"9409de62-685d-4284-bb5b-81f73b39da9a","Type":"ContainerDied","Data":"da767f16a4777851e19d63505062029c1767530ea5d94ae2b4320ecf3d99027c"}
Jul 28 14:17:10.292023 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:10.291973    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6" podUID="9409de62-685d-4284-bb5b-81f73b39da9a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.60:8643/healthz\": dial tcp 10.134.0.60:8643: connect: connection refused"
Jul 28 14:17:10.298763 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:10.298734    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6" podUID="9409de62-685d-4284-bb5b-81f73b39da9a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.60:8080/v2/models/xgboost-v2-mlserver/ready\": dial tcp 10.134.0.60:8080: connect: connection refused"
Jul 28 14:17:12.195192 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:12.195157    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-stfc9"]
Jul 28 14:17:12.195702 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:12.195500    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1e67e6e-9932-4097-9820-68866162135b" containerName="kserve-container"
Jul 28 14:17:12.195702 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:12.195512    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1e67e6e-9932-4097-9820-68866162135b" containerName="kserve-container"
Jul 28 14:17:12.195702 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:12.195523    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1e67e6e-9932-4097-9820-68866162135b" containerName="storage-initializer"
Jul 28 14:17:12.195702 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:12.195529    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1e67e6e-9932-4097-9820-68866162135b" containerName="storage-initializer"
Jul 28 14:17:12.195702 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:12.195549    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1e67e6e-9932-4097-9820-68866162135b" containerName="kube-rbac-proxy"
Jul 28 14:17:12.195702 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:12.195555    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1e67e6e-9932-4097-9820-68866162135b" containerName="kube-rbac-proxy"
Jul 28 14:17:12.195702 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:12.195602    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1e67e6e-9932-4097-9820-68866162135b" containerName="kube-rbac-proxy"
Jul 28 14:17:12.195702 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:12.195613    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1e67e6e-9932-4097-9820-68866162135b" containerName="kserve-container"
Jul 28 14:17:12.198927 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:12.198908    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-stfc9"
Jul 28 14:17:12.201600 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:12.201574    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
Jul 28 14:17:12.201722 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:12.201596    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
Jul 28 14:17:12.214090 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:12.214063    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-stfc9"]
Jul 28 14:17:12.282760 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:12.282728    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4bcdc23a-f84e-401c-ac5e-3ece4f82eca4-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-stfc9\" (UID: \"4bcdc23a-f84e-401c-ac5e-3ece4f82eca4\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-stfc9"
Jul 28 14:17:12.282958 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:12.282765    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4bcdc23a-f84e-401c-ac5e-3ece4f82eca4-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-stfc9\" (UID: \"4bcdc23a-f84e-401c-ac5e-3ece4f82eca4\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-stfc9"
Jul 28 14:17:12.282958 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:12.282787    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfphm\" (UniqueName: \"kubernetes.io/projected/4bcdc23a-f84e-401c-ac5e-3ece4f82eca4-kube-api-access-lfphm\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-stfc9\" (UID: \"4bcdc23a-f84e-401c-ac5e-3ece4f82eca4\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-stfc9"
Jul 28 14:17:12.282958 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:12.282852    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bcdc23a-f84e-401c-ac5e-3ece4f82eca4-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-stfc9\" (UID: \"4bcdc23a-f84e-401c-ac5e-3ece4f82eca4\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-stfc9"
Jul 28 14:17:12.282958 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:12.282953    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4bcdc23a-f84e-401c-ac5e-3ece4f82eca4-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-stfc9\" (UID: \"4bcdc23a-f84e-401c-ac5e-3ece4f82eca4\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-stfc9"
Jul 28 14:17:12.384136 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:12.384084    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4bcdc23a-f84e-401c-ac5e-3ece4f82eca4-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-stfc9\" (UID: \"4bcdc23a-f84e-401c-ac5e-3ece4f82eca4\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-stfc9"
Jul 28 14:17:12.384371 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:12.384152    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4bcdc23a-f84e-401c-ac5e-3ece4f82eca4-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-stfc9\" (UID: \"4bcdc23a-f84e-401c-ac5e-3ece4f82eca4\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-stfc9"
Jul 28 14:17:12.384371 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:12.384188    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4bcdc23a-f84e-401c-ac5e-3ece4f82eca4-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-stfc9\" (UID: \"4bcdc23a-f84e-401c-ac5e-3ece4f82eca4\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-stfc9"
Jul 28 14:17:12.384371 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:12.384212    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfphm\" (UniqueName: \"kubernetes.io/projected/4bcdc23a-f84e-401c-ac5e-3ece4f82eca4-kube-api-access-lfphm\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-stfc9\" (UID: \"4bcdc23a-f84e-401c-ac5e-3ece4f82eca4\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-stfc9"
Jul 28 14:17:12.384371 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:12.384246    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bcdc23a-f84e-401c-ac5e-3ece4f82eca4-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-stfc9\" (UID: \"4bcdc23a-f84e-401c-ac5e-3ece4f82eca4\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-stfc9"
Jul 28 14:17:12.384584 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:17:12.384485    2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-serving-cert: secret "isvc-xgboost-runtime-predictor-serving-cert" not found
Jul 28 14:17:12.384625 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:17:12.384587    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bcdc23a-f84e-401c-ac5e-3ece4f82eca4-proxy-tls podName:4bcdc23a-f84e-401c-ac5e-3ece4f82eca4 nodeName:}" failed. No retries permitted until 2026-07-28 14:17:12.884563912 +0000 UTC m=+5029.670439021 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4bcdc23a-f84e-401c-ac5e-3ece4f82eca4-proxy-tls") pod "isvc-xgboost-runtime-predictor-54dd488c85-stfc9" (UID: "4bcdc23a-f84e-401c-ac5e-3ece4f82eca4") : secret "isvc-xgboost-runtime-predictor-serving-cert" not found
Jul 28 14:17:12.384687 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:12.384665    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4bcdc23a-f84e-401c-ac5e-3ece4f82eca4-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-stfc9\" (UID: \"4bcdc23a-f84e-401c-ac5e-3ece4f82eca4\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-stfc9"
Jul 28 14:17:12.384846 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:12.384827    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4bcdc23a-f84e-401c-ac5e-3ece4f82eca4-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-stfc9\" (UID: \"4bcdc23a-f84e-401c-ac5e-3ece4f82eca4\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-stfc9"
Jul 28 14:17:12.384938 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:12.384920    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4bcdc23a-f84e-401c-ac5e-3ece4f82eca4-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-stfc9\" (UID: \"4bcdc23a-f84e-401c-ac5e-3ece4f82eca4\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-stfc9"
Jul 28 14:17:12.393809 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:12.393782    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfphm\" (UniqueName: \"kubernetes.io/projected/4bcdc23a-f84e-401c-ac5e-3ece4f82eca4-kube-api-access-lfphm\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-stfc9\" (UID: \"4bcdc23a-f84e-401c-ac5e-3ece4f82eca4\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-stfc9"
Jul 28 14:17:12.888686 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:12.888643    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bcdc23a-f84e-401c-ac5e-3ece4f82eca4-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-stfc9\" (UID: \"4bcdc23a-f84e-401c-ac5e-3ece4f82eca4\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-stfc9"
Jul 28 14:17:12.891193 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:12.891167    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bcdc23a-f84e-401c-ac5e-3ece4f82eca4-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-stfc9\" (UID: \"4bcdc23a-f84e-401c-ac5e-3ece4f82eca4\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-stfc9"
Jul 28 14:17:13.108980 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:13.108947    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-stfc9"
Jul 28 14:17:13.245687 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:13.245659    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-stfc9"]
Jul 28 14:17:13.249394 ip-10-0-129-177 kubenswrapper[2554]: W0728 14:17:13.249364    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bcdc23a_f84e_401c_ac5e_3ece4f82eca4.slice/crio-4b31cb9e36ab11efc94fc385805baee8e2be9097b9c354be029b7f6c621f322d WatchSource:0}: Error finding container 4b31cb9e36ab11efc94fc385805baee8e2be9097b9c354be029b7f6c621f322d: Status 404 returned error can't find the container with id 4b31cb9e36ab11efc94fc385805baee8e2be9097b9c354be029b7f6c621f322d
Jul 28 14:17:13.354407 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:13.354374    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6"
Jul 28 14:17:13.393352 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:13.393323    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vh9zw\" (UniqueName: \"kubernetes.io/projected/9409de62-685d-4284-bb5b-81f73b39da9a-kube-api-access-vh9zw\") pod \"9409de62-685d-4284-bb5b-81f73b39da9a\" (UID: \"9409de62-685d-4284-bb5b-81f73b39da9a\") "
Jul 28 14:17:13.393520 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:13.393364    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9409de62-685d-4284-bb5b-81f73b39da9a-kube-rbac-proxy-sar-config\") pod \"9409de62-685d-4284-bb5b-81f73b39da9a\" (UID: \"9409de62-685d-4284-bb5b-81f73b39da9a\") "
Jul 28 14:17:13.393520 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:13.393415    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9409de62-685d-4284-bb5b-81f73b39da9a-proxy-tls\") pod \"9409de62-685d-4284-bb5b-81f73b39da9a\" (UID: \"9409de62-685d-4284-bb5b-81f73b39da9a\") "
Jul 28 14:17:13.393520 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:13.393475    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9409de62-685d-4284-bb5b-81f73b39da9a-cabundle-cert\") pod \"9409de62-685d-4284-bb5b-81f73b39da9a\" (UID: \"9409de62-685d-4284-bb5b-81f73b39da9a\") "
Jul 28 14:17:13.393520 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:13.393510    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9409de62-685d-4284-bb5b-81f73b39da9a-kserve-provision-location\") pod \"9409de62-685d-4284-bb5b-81f73b39da9a\" (UID: \"9409de62-685d-4284-bb5b-81f73b39da9a\") "
Jul 28 14:17:13.394043 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:13.394010    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9409de62-685d-4284-bb5b-81f73b39da9a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9409de62-685d-4284-bb5b-81f73b39da9a" (UID: "9409de62-685d-4284-bb5b-81f73b39da9a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:17:13.394216 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:13.394043    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9409de62-685d-4284-bb5b-81f73b39da9a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9409de62-685d-4284-bb5b-81f73b39da9a" (UID: "9409de62-685d-4284-bb5b-81f73b39da9a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:17:13.394420 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:13.394202    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9409de62-685d-4284-bb5b-81f73b39da9a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9409de62-685d-4284-bb5b-81f73b39da9a" (UID: "9409de62-685d-4284-bb5b-81f73b39da9a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:17:13.395842 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:13.395818    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9409de62-685d-4284-bb5b-81f73b39da9a-kube-api-access-vh9zw" (OuterVolumeSpecName: "kube-api-access-vh9zw") pod "9409de62-685d-4284-bb5b-81f73b39da9a" (UID: "9409de62-685d-4284-bb5b-81f73b39da9a"). InnerVolumeSpecName "kube-api-access-vh9zw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:17:13.396216 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:13.396200    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9409de62-685d-4284-bb5b-81f73b39da9a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9409de62-685d-4284-bb5b-81f73b39da9a" (UID: "9409de62-685d-4284-bb5b-81f73b39da9a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:17:13.494543 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:13.494508    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9409de62-685d-4284-bb5b-81f73b39da9a-cabundle-cert\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:17:13.494543 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:13.494532    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9409de62-685d-4284-bb5b-81f73b39da9a-kserve-provision-location\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:17:13.494543 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:13.494541    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh9zw\" (UniqueName: \"kubernetes.io/projected/9409de62-685d-4284-bb5b-81f73b39da9a-kube-api-access-vh9zw\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:17:13.494543 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:13.494549    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9409de62-685d-4284-bb5b-81f73b39da9a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:17:13.494543 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:13.494557    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9409de62-685d-4284-bb5b-81f73b39da9a-proxy-tls\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:17:13.497189 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:13.497164    2554 generic.go:356] "Generic (PLEG): container finished" podID="9409de62-685d-4284-bb5b-81f73b39da9a" containerID="b6235c421ebb235d6ae7037c930af9bef0e4c7c32ffd707c8a002c68f615c3b8" exitCode=0
Jul 28 14:17:13.497367 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:13.497220    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6"
Jul 28 14:17:13.497367 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:13.497251    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6" event={"ID":"9409de62-685d-4284-bb5b-81f73b39da9a","Type":"ContainerDied","Data":"b6235c421ebb235d6ae7037c930af9bef0e4c7c32ffd707c8a002c68f615c3b8"}
Jul 28 14:17:13.497367 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:13.497334    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6" event={"ID":"9409de62-685d-4284-bb5b-81f73b39da9a","Type":"ContainerDied","Data":"1bb679c2fabb18ee3f20c33adf938408e644e49cf08c2d6e2ac5b52572bbc970"}
Jul 28 14:17:13.497570 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:13.497372    2554 scope.go:117] "RemoveContainer" containerID="da767f16a4777851e19d63505062029c1767530ea5d94ae2b4320ecf3d99027c"
Jul 28 14:17:13.499017 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:13.498990    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-stfc9" event={"ID":"4bcdc23a-f84e-401c-ac5e-3ece4f82eca4","Type":"ContainerStarted","Data":"6f897dc2e4a40ea14c54b9f6b339a482b15a19beee9dcb71a242d7b4f41ccc2c"}
Jul 28 14:17:13.499117 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:13.499047    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-stfc9" event={"ID":"4bcdc23a-f84e-401c-ac5e-3ece4f82eca4","Type":"ContainerStarted","Data":"4b31cb9e36ab11efc94fc385805baee8e2be9097b9c354be029b7f6c621f322d"}
Jul 28 14:17:13.506816 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:13.506795    2554 scope.go:117] "RemoveContainer" containerID="b6235c421ebb235d6ae7037c930af9bef0e4c7c32ffd707c8a002c68f615c3b8"
Jul 28 14:17:13.514419 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:13.514401    2554 scope.go:117] "RemoveContainer" containerID="aa05fc9daa0c6e46b1c8774d49310bc003ccb9e1f8558ca4694c3af5f58b85b7"
Jul 28 14:17:13.524261 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:13.524241    2554 scope.go:117] "RemoveContainer" containerID="da767f16a4777851e19d63505062029c1767530ea5d94ae2b4320ecf3d99027c"
Jul 28 14:17:13.524556 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:17:13.524538    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da767f16a4777851e19d63505062029c1767530ea5d94ae2b4320ecf3d99027c\": container with ID starting with da767f16a4777851e19d63505062029c1767530ea5d94ae2b4320ecf3d99027c not found: ID does not exist" containerID="da767f16a4777851e19d63505062029c1767530ea5d94ae2b4320ecf3d99027c"
Jul 28 14:17:13.524614 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:13.524566    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da767f16a4777851e19d63505062029c1767530ea5d94ae2b4320ecf3d99027c"} err="failed to get container status \"da767f16a4777851e19d63505062029c1767530ea5d94ae2b4320ecf3d99027c\": rpc error: code = NotFound desc = could not find container \"da767f16a4777851e19d63505062029c1767530ea5d94ae2b4320ecf3d99027c\": container with ID starting with da767f16a4777851e19d63505062029c1767530ea5d94ae2b4320ecf3d99027c not found: ID does not exist"
Jul 28 14:17:13.524614 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:13.524584    2554 scope.go:117] "RemoveContainer" containerID="b6235c421ebb235d6ae7037c930af9bef0e4c7c32ffd707c8a002c68f615c3b8"
Jul 28 14:17:13.524858 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:17:13.524842    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6235c421ebb235d6ae7037c930af9bef0e4c7c32ffd707c8a002c68f615c3b8\": container with ID starting with b6235c421ebb235d6ae7037c930af9bef0e4c7c32ffd707c8a002c68f615c3b8 not found: ID does not exist" containerID="b6235c421ebb235d6ae7037c930af9bef0e4c7c32ffd707c8a002c68f615c3b8"
Jul 28 14:17:13.524914 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:13.524870    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6235c421ebb235d6ae7037c930af9bef0e4c7c32ffd707c8a002c68f615c3b8"} err="failed to get container status \"b6235c421ebb235d6ae7037c930af9bef0e4c7c32ffd707c8a002c68f615c3b8\": rpc error: code = NotFound desc = could not find container \"b6235c421ebb235d6ae7037c930af9bef0e4c7c32ffd707c8a002c68f615c3b8\": container with ID starting with b6235c421ebb235d6ae7037c930af9bef0e4c7c32ffd707c8a002c68f615c3b8 not found: ID does not exist"
Jul 28 14:17:13.524914 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:13.524883    2554 scope.go:117] "RemoveContainer" containerID="aa05fc9daa0c6e46b1c8774d49310bc003ccb9e1f8558ca4694c3af5f58b85b7"
Jul 28 14:17:13.525145 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:17:13.525087    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa05fc9daa0c6e46b1c8774d49310bc003ccb9e1f8558ca4694c3af5f58b85b7\": container with ID starting with aa05fc9daa0c6e46b1c8774d49310bc003ccb9e1f8558ca4694c3af5f58b85b7 not found: ID does not exist" containerID="aa05fc9daa0c6e46b1c8774d49310bc003ccb9e1f8558ca4694c3af5f58b85b7"
Jul 28 14:17:13.525145 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:13.525108    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa05fc9daa0c6e46b1c8774d49310bc003ccb9e1f8558ca4694c3af5f58b85b7"} err="failed to get container status \"aa05fc9daa0c6e46b1c8774d49310bc003ccb9e1f8558ca4694c3af5f58b85b7\": rpc error: code = NotFound desc = could not find container \"aa05fc9daa0c6e46b1c8774d49310bc003ccb9e1f8558ca4694c3af5f58b85b7\": container with ID starting with aa05fc9daa0c6e46b1c8774d49310bc003ccb9e1f8558ca4694c3af5f58b85b7 not found: ID does not exist"
Jul 28 14:17:13.541137 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:13.541100    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6"]
Jul 28 14:17:13.545132 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:13.545107    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-tb8t6"]
Jul 28 14:17:13.803258 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:13.803174    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9409de62-685d-4284-bb5b-81f73b39da9a" path="/var/lib/kubelet/pods/9409de62-685d-4284-bb5b-81f73b39da9a/volumes"
Jul 28 14:17:17.517669 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:17.517635    2554 generic.go:356] "Generic (PLEG): container finished" podID="4bcdc23a-f84e-401c-ac5e-3ece4f82eca4" containerID="6f897dc2e4a40ea14c54b9f6b339a482b15a19beee9dcb71a242d7b4f41ccc2c" exitCode=0
Jul 28 14:17:17.518050 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:17.517708    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-stfc9" event={"ID":"4bcdc23a-f84e-401c-ac5e-3ece4f82eca4","Type":"ContainerDied","Data":"6f897dc2e4a40ea14c54b9f6b339a482b15a19beee9dcb71a242d7b4f41ccc2c"}
Jul 28 14:17:18.524295 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:18.524240    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-stfc9" event={"ID":"4bcdc23a-f84e-401c-ac5e-3ece4f82eca4","Type":"ContainerStarted","Data":"10185faa97098ae73faa0f7d9e722279a65d8990a443bda6ebaab8c5b35a6046"}
Jul 28 14:17:18.524730 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:18.524325    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-stfc9" event={"ID":"4bcdc23a-f84e-401c-ac5e-3ece4f82eca4","Type":"ContainerStarted","Data":"5e3059ff20496c996dd59ee1a411406e514e5a8202ff88b542abc06b7e483ab4"}
Jul 28 14:17:18.524730 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:18.524580    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-stfc9"
Jul 28 14:17:18.524730 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:18.524636    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-stfc9"
Jul 28 14:17:18.525874 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:18.525848    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-stfc9" podUID="4bcdc23a-f84e-401c-ac5e-3ece4f82eca4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Jul 28 14:17:18.547343 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:18.547295    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-stfc9" podStartSLOduration=3.014333666 podStartE2EDuration="6.547261573s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.532927907s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:17:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:17:18.546181671 +0000 UTC m=+5035.332056782" watchObservedRunningTime="2026-07-28 14:17:18.547261573 +0000 UTC m=+5035.333136684"
Jul 28 14:17:19.527965 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:19.527927    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-stfc9" podUID="4bcdc23a-f84e-401c-ac5e-3ece4f82eca4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Jul 28 14:17:24.533077 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:24.533045    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-stfc9"
Jul 28 14:17:24.533621 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:24.533597    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-stfc9" podUID="4bcdc23a-f84e-401c-ac5e-3ece4f82eca4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Jul 28 14:17:34.534154 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:34.534116    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-stfc9" podUID="4bcdc23a-f84e-401c-ac5e-3ece4f82eca4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Jul 28 14:17:44.533751 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:44.533710    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-stfc9" podUID="4bcdc23a-f84e-401c-ac5e-3ece4f82eca4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Jul 28 14:17:54.533634 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:17:54.533596    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-stfc9" podUID="4bcdc23a-f84e-401c-ac5e-3ece4f82eca4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Jul 28 14:18:04.534069 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:04.534023    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-stfc9" podUID="4bcdc23a-f84e-401c-ac5e-3ece4f82eca4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Jul 28 14:18:14.534470 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:14.534444    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-stfc9"
Jul 28 14:18:22.284108 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:22.284076    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-stfc9"]
Jul 28 14:18:22.284591 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:22.284338    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-stfc9" podUID="4bcdc23a-f84e-401c-ac5e-3ece4f82eca4" containerName="kserve-container" containerID="cri-o://5e3059ff20496c996dd59ee1a411406e514e5a8202ff88b542abc06b7e483ab4" gracePeriod=30
Jul 28 14:18:22.284591 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:22.284411    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-stfc9" podUID="4bcdc23a-f84e-401c-ac5e-3ece4f82eca4" containerName="kube-rbac-proxy" containerID="cri-o://10185faa97098ae73faa0f7d9e722279a65d8990a443bda6ebaab8c5b35a6046" gracePeriod=30
Jul 28 14:18:22.786703 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:22.786674    2554 generic.go:356] "Generic (PLEG): container finished" podID="4bcdc23a-f84e-401c-ac5e-3ece4f82eca4" containerID="10185faa97098ae73faa0f7d9e722279a65d8990a443bda6ebaab8c5b35a6046" exitCode=2
Jul 28 14:18:22.786869 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:22.786732    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-stfc9" event={"ID":"4bcdc23a-f84e-401c-ac5e-3ece4f82eca4","Type":"ContainerDied","Data":"10185faa97098ae73faa0f7d9e722279a65d8990a443bda6ebaab8c5b35a6046"}
Jul 28 14:18:24.529093 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:24.529050    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-stfc9" podUID="4bcdc23a-f84e-401c-ac5e-3ece4f82eca4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.61:8643/healthz\": dial tcp 10.134.0.61:8643: connect: connection refused"
Jul 28 14:18:24.533717 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:24.533691    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-stfc9" podUID="4bcdc23a-f84e-401c-ac5e-3ece4f82eca4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Jul 28 14:18:25.801411 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:25.801374    2554 generic.go:356] "Generic (PLEG): container finished" podID="4bcdc23a-f84e-401c-ac5e-3ece4f82eca4" containerID="5e3059ff20496c996dd59ee1a411406e514e5a8202ff88b542abc06b7e483ab4" exitCode=0
Jul 28 14:18:25.801867 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:25.801846    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-stfc9" event={"ID":"4bcdc23a-f84e-401c-ac5e-3ece4f82eca4","Type":"ContainerDied","Data":"5e3059ff20496c996dd59ee1a411406e514e5a8202ff88b542abc06b7e483ab4"}
Jul 28 14:18:25.829054 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:25.829032    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-stfc9"
Jul 28 14:18:25.900737 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:25.900665    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4bcdc23a-f84e-401c-ac5e-3ece4f82eca4-kube-rbac-proxy-sar-config\") pod \"4bcdc23a-f84e-401c-ac5e-3ece4f82eca4\" (UID: \"4bcdc23a-f84e-401c-ac5e-3ece4f82eca4\") "
Jul 28 14:18:25.900737 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:25.900710    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4bcdc23a-f84e-401c-ac5e-3ece4f82eca4-kserve-provision-location\") pod \"4bcdc23a-f84e-401c-ac5e-3ece4f82eca4\" (UID: \"4bcdc23a-f84e-401c-ac5e-3ece4f82eca4\") "
Jul 28 14:18:25.900737 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:25.900736    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bcdc23a-f84e-401c-ac5e-3ece4f82eca4-proxy-tls\") pod \"4bcdc23a-f84e-401c-ac5e-3ece4f82eca4\" (UID: \"4bcdc23a-f84e-401c-ac5e-3ece4f82eca4\") "
Jul 28 14:18:25.900979 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:25.900756    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4bcdc23a-f84e-401c-ac5e-3ece4f82eca4-cabundle-cert\") pod \"4bcdc23a-f84e-401c-ac5e-3ece4f82eca4\" (UID: \"4bcdc23a-f84e-401c-ac5e-3ece4f82eca4\") "
Jul 28 14:18:25.900979 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:25.900789    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lfphm\" (UniqueName: \"kubernetes.io/projected/4bcdc23a-f84e-401c-ac5e-3ece4f82eca4-kube-api-access-lfphm\") pod \"4bcdc23a-f84e-401c-ac5e-3ece4f82eca4\" (UID: \"4bcdc23a-f84e-401c-ac5e-3ece4f82eca4\") "
Jul 28 14:18:25.901087 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:25.901037    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bcdc23a-f84e-401c-ac5e-3ece4f82eca4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4bcdc23a-f84e-401c-ac5e-3ece4f82eca4" (UID: "4bcdc23a-f84e-401c-ac5e-3ece4f82eca4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:18:25.901087 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:25.901052    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bcdc23a-f84e-401c-ac5e-3ece4f82eca4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4bcdc23a-f84e-401c-ac5e-3ece4f82eca4" (UID: "4bcdc23a-f84e-401c-ac5e-3ece4f82eca4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:18:25.901200 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:25.901123    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bcdc23a-f84e-401c-ac5e-3ece4f82eca4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4bcdc23a-f84e-401c-ac5e-3ece4f82eca4" (UID: "4bcdc23a-f84e-401c-ac5e-3ece4f82eca4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:18:25.901200 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:25.901184    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4bcdc23a-f84e-401c-ac5e-3ece4f82eca4-kserve-provision-location\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:18:25.903009 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:25.902981    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bcdc23a-f84e-401c-ac5e-3ece4f82eca4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4bcdc23a-f84e-401c-ac5e-3ece4f82eca4" (UID: "4bcdc23a-f84e-401c-ac5e-3ece4f82eca4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:18:25.903189 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:25.903167    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bcdc23a-f84e-401c-ac5e-3ece4f82eca4-kube-api-access-lfphm" (OuterVolumeSpecName: "kube-api-access-lfphm") pod "4bcdc23a-f84e-401c-ac5e-3ece4f82eca4" (UID: "4bcdc23a-f84e-401c-ac5e-3ece4f82eca4"). InnerVolumeSpecName "kube-api-access-lfphm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:18:26.002102 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:26.002067    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4bcdc23a-f84e-401c-ac5e-3ece4f82eca4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:18:26.002102 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:26.002100    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bcdc23a-f84e-401c-ac5e-3ece4f82eca4-proxy-tls\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:18:26.002102 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:26.002111    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4bcdc23a-f84e-401c-ac5e-3ece4f82eca4-cabundle-cert\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:18:26.002343 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:26.002119    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfphm\" (UniqueName: \"kubernetes.io/projected/4bcdc23a-f84e-401c-ac5e-3ece4f82eca4-kube-api-access-lfphm\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:18:26.806638 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:26.806608    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-stfc9"
Jul 28 14:18:26.806638 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:26.806630    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-stfc9" event={"ID":"4bcdc23a-f84e-401c-ac5e-3ece4f82eca4","Type":"ContainerDied","Data":"4b31cb9e36ab11efc94fc385805baee8e2be9097b9c354be029b7f6c621f322d"}
Jul 28 14:18:26.807152 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:26.806692    2554 scope.go:117] "RemoveContainer" containerID="10185faa97098ae73faa0f7d9e722279a65d8990a443bda6ebaab8c5b35a6046"
Jul 28 14:18:26.815157 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:26.815138    2554 scope.go:117] "RemoveContainer" containerID="5e3059ff20496c996dd59ee1a411406e514e5a8202ff88b542abc06b7e483ab4"
Jul 28 14:18:26.822745 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:26.822722    2554 scope.go:117] "RemoveContainer" containerID="6f897dc2e4a40ea14c54b9f6b339a482b15a19beee9dcb71a242d7b4f41ccc2c"
Jul 28 14:18:26.836807 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:26.836783    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-stfc9"]
Jul 28 14:18:26.840343 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:26.840322    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-stfc9"]
Jul 28 14:18:27.527443 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:27.527411    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj"]
Jul 28 14:18:27.527747 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:27.527734    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bcdc23a-f84e-401c-ac5e-3ece4f82eca4" containerName="storage-initializer"
Jul 28 14:18:27.527802 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:27.527749    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bcdc23a-f84e-401c-ac5e-3ece4f82eca4" containerName="storage-initializer"
Jul 28 14:18:27.527802 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:27.527760    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bcdc23a-f84e-401c-ac5e-3ece4f82eca4" containerName="kserve-container"
Jul 28 14:18:27.527802 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:27.527768    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bcdc23a-f84e-401c-ac5e-3ece4f82eca4" containerName="kserve-container"
Jul 28 14:18:27.527802 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:27.527776    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9409de62-685d-4284-bb5b-81f73b39da9a" containerName="storage-initializer"
Jul 28 14:18:27.527802 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:27.527782    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="9409de62-685d-4284-bb5b-81f73b39da9a" containerName="storage-initializer"
Jul 28 14:18:27.527802 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:27.527790    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bcdc23a-f84e-401c-ac5e-3ece4f82eca4" containerName="kube-rbac-proxy"
Jul 28 14:18:27.527802 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:27.527796    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bcdc23a-f84e-401c-ac5e-3ece4f82eca4" containerName="kube-rbac-proxy"
Jul 28 14:18:27.527802 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:27.527802    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9409de62-685d-4284-bb5b-81f73b39da9a" containerName="kserve-container"
Jul 28 14:18:27.528042 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:27.527808    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="9409de62-685d-4284-bb5b-81f73b39da9a" containerName="kserve-container"
Jul 28 14:18:27.528042 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:27.527815    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9409de62-685d-4284-bb5b-81f73b39da9a" containerName="kube-rbac-proxy"
Jul 28 14:18:27.528042 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:27.527820    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="9409de62-685d-4284-bb5b-81f73b39da9a" containerName="kube-rbac-proxy"
Jul 28 14:18:27.528042 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:27.527866    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bcdc23a-f84e-401c-ac5e-3ece4f82eca4" containerName="kserve-container"
Jul 28 14:18:27.528042 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:27.527874    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="9409de62-685d-4284-bb5b-81f73b39da9a" containerName="kserve-container"
Jul 28 14:18:27.528042 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:27.527880    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="9409de62-685d-4284-bb5b-81f73b39da9a" containerName="kube-rbac-proxy"
Jul 28 14:18:27.528042 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:27.527887    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bcdc23a-f84e-401c-ac5e-3ece4f82eca4" containerName="kube-rbac-proxy"
Jul 28 14:18:27.532709 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:27.532680    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj"
Jul 28 14:18:27.535179 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:27.535157    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rxfwd\""
Jul 28 14:18:27.535982 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:27.535951    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 28 14:18:27.535982 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:27.535958    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
Jul 28 14:18:27.536175 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:27.536004    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 14:18:27.536175 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:27.536100    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 14:18:27.536395 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:27.536379    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 14:18:27.536656 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:27.536639    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 14:18:27.544575 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:27.544553    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj"]
Jul 28 14:18:27.614886 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:27.614849    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7g4q\" (UniqueName: \"kubernetes.io/projected/841d9a72-daa8-45f4-8734-710eb1ec82c6-kube-api-access-b7g4q\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj\" (UID: \"841d9a72-daa8-45f4-8734-710eb1ec82c6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj"
Jul 28 14:18:27.615061 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:27.614891    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/841d9a72-daa8-45f4-8734-710eb1ec82c6-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj\" (UID: \"841d9a72-daa8-45f4-8734-710eb1ec82c6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj"
Jul 28 14:18:27.615061 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:27.614917    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/841d9a72-daa8-45f4-8734-710eb1ec82c6-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj\" (UID: \"841d9a72-daa8-45f4-8734-710eb1ec82c6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj"
Jul 28 14:18:27.615061 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:27.614943    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/841d9a72-daa8-45f4-8734-710eb1ec82c6-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj\" (UID: \"841d9a72-daa8-45f4-8734-710eb1ec82c6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj"
Jul 28 14:18:27.615061 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:27.614998    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/841d9a72-daa8-45f4-8734-710eb1ec82c6-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj\" (UID: \"841d9a72-daa8-45f4-8734-710eb1ec82c6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj"
Jul 28 14:18:27.716038 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:27.715991    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/841d9a72-daa8-45f4-8734-710eb1ec82c6-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj\" (UID: \"841d9a72-daa8-45f4-8734-710eb1ec82c6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj"
Jul 28 14:18:27.716038 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:27.716042    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/841d9a72-daa8-45f4-8734-710eb1ec82c6-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj\" (UID: \"841d9a72-daa8-45f4-8734-710eb1ec82c6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj"
Jul 28 14:18:27.716330 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:27.716070    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/841d9a72-daa8-45f4-8734-710eb1ec82c6-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj\" (UID: \"841d9a72-daa8-45f4-8734-710eb1ec82c6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj"
Jul 28 14:18:27.716330 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:27.716101    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/841d9a72-daa8-45f4-8734-710eb1ec82c6-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj\" (UID: \"841d9a72-daa8-45f4-8734-710eb1ec82c6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj"
Jul 28 14:18:27.716330 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:27.716194    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7g4q\" (UniqueName: \"kubernetes.io/projected/841d9a72-daa8-45f4-8734-710eb1ec82c6-kube-api-access-b7g4q\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj\" (UID: \"841d9a72-daa8-45f4-8734-710eb1ec82c6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj"
Jul 28 14:18:27.716526 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:27.716489    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/841d9a72-daa8-45f4-8734-710eb1ec82c6-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj\" (UID: \"841d9a72-daa8-45f4-8734-710eb1ec82c6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj"
Jul 28 14:18:27.716807 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:27.716790    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/841d9a72-daa8-45f4-8734-710eb1ec82c6-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj\" (UID: \"841d9a72-daa8-45f4-8734-710eb1ec82c6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj"
Jul 28 14:18:27.716973 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:27.716953    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/841d9a72-daa8-45f4-8734-710eb1ec82c6-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj\" (UID: \"841d9a72-daa8-45f4-8734-710eb1ec82c6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj"
Jul 28 14:18:27.718742 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:27.718722    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/841d9a72-daa8-45f4-8734-710eb1ec82c6-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj\" (UID: \"841d9a72-daa8-45f4-8734-710eb1ec82c6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj"
Jul 28 14:18:27.726140 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:27.726116    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7g4q\" (UniqueName: \"kubernetes.io/projected/841d9a72-daa8-45f4-8734-710eb1ec82c6-kube-api-access-b7g4q\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj\" (UID: \"841d9a72-daa8-45f4-8734-710eb1ec82c6\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj"
Jul 28 14:18:27.802077 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:27.802011    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4bcdc23a-f84e-401c-ac5e-3ece4f82eca4" path="/var/lib/kubelet/pods/4bcdc23a-f84e-401c-ac5e-3ece4f82eca4/volumes"
Jul 28 14:18:27.843765 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:27.843742    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj"
Jul 28 14:18:27.980489 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:27.980459    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj"]
Jul 28 14:18:27.983592 ip-10-0-129-177 kubenswrapper[2554]: W0728 14:18:27.983566    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod841d9a72_daa8_45f4_8734_710eb1ec82c6.slice/crio-7d7d43266173d694e501a3f52231469b19f1b763c18491a62dc421f1ddbb07d8 WatchSource:0}: Error finding container 7d7d43266173d694e501a3f52231469b19f1b763c18491a62dc421f1ddbb07d8: Status 404 returned error can't find the container with id 7d7d43266173d694e501a3f52231469b19f1b763c18491a62dc421f1ddbb07d8
Jul 28 14:18:28.819638 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:28.819601    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj" event={"ID":"841d9a72-daa8-45f4-8734-710eb1ec82c6","Type":"ContainerStarted","Data":"e8b1580fb0a56affed91d9ed85bdf5d21bc9f75861369f2dea337f0726011228"}
Jul 28 14:18:28.819790 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:28.819677    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj" event={"ID":"841d9a72-daa8-45f4-8734-710eb1ec82c6","Type":"ContainerStarted","Data":"7d7d43266173d694e501a3f52231469b19f1b763c18491a62dc421f1ddbb07d8"}
Jul 28 14:18:32.837445 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:32.837408    2554 generic.go:356] "Generic (PLEG): container finished" podID="841d9a72-daa8-45f4-8734-710eb1ec82c6" containerID="e8b1580fb0a56affed91d9ed85bdf5d21bc9f75861369f2dea337f0726011228" exitCode=0
Jul 28 14:18:32.837825 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:32.837481    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj" event={"ID":"841d9a72-daa8-45f4-8734-710eb1ec82c6","Type":"ContainerDied","Data":"e8b1580fb0a56affed91d9ed85bdf5d21bc9f75861369f2dea337f0726011228"}
Jul 28 14:18:33.843054 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:33.843020    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj" event={"ID":"841d9a72-daa8-45f4-8734-710eb1ec82c6","Type":"ContainerStarted","Data":"5861ebe5e16d4acde294127c37433f12aba37357e9f87eb4276fa80fbe84e341"}
Jul 28 14:18:33.843496 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:33.843083    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj" event={"ID":"841d9a72-daa8-45f4-8734-710eb1ec82c6","Type":"ContainerStarted","Data":"042c218550f7983d3abe148c9e13ba78f4ddf216f9e56451b5fd0e3519b893ee"}
Jul 28 14:18:33.843496 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:33.843408    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj"
Jul 28 14:18:33.843496 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:33.843453    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj"
Jul 28 14:18:33.867106 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:33.867063    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj" podStartSLOduration=2.807051366 podStartE2EDuration="6.86704783s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.059996464s" isStatefulPod=false podCreationTimestamp="2026-07-28 14:18:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:18:33.865381196 +0000 UTC m=+5110.651256320" watchObservedRunningTime="2026-07-28 14:18:33.86704783 +0000 UTC m=+5110.652922943"
Jul 28 14:18:39.852765 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:18:39.852734    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj"
Jul 28 14:19:09.890209 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:19:09.890166    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj" podUID="841d9a72-daa8-45f4-8734-710eb1ec82c6" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Jul 28 14:19:19.856147 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:19:19.856118    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj"
Jul 28 14:19:27.587918 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:19:27.587860    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj"]
Jul 28 14:19:27.588531 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:19:27.588136    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj" podUID="841d9a72-daa8-45f4-8734-710eb1ec82c6" containerName="kserve-container" containerID="cri-o://042c218550f7983d3abe148c9e13ba78f4ddf216f9e56451b5fd0e3519b893ee" gracePeriod=30
Jul 28 14:19:27.588531 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:19:27.588152    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj" podUID="841d9a72-daa8-45f4-8734-710eb1ec82c6" containerName="kube-rbac-proxy" containerID="cri-o://5861ebe5e16d4acde294127c37433f12aba37357e9f87eb4276fa80fbe84e341" gracePeriod=30
Jul 28 14:19:28.064878 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:19:28.064842    2554 generic.go:356] "Generic (PLEG): container finished" podID="841d9a72-daa8-45f4-8734-710eb1ec82c6" containerID="5861ebe5e16d4acde294127c37433f12aba37357e9f87eb4276fa80fbe84e341" exitCode=2
Jul 28 14:19:28.065060 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:19:28.064905    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj" event={"ID":"841d9a72-daa8-45f4-8734-710eb1ec82c6","Type":"ContainerDied","Data":"5861ebe5e16d4acde294127c37433f12aba37357e9f87eb4276fa80fbe84e341"}
Jul 28 14:19:29.848570 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:19:29.848529    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj" podUID="841d9a72-daa8-45f4-8734-710eb1ec82c6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.62:8643/healthz\": dial tcp 10.134.0.62:8643: connect: connection refused"
Jul 28 14:19:30.895472 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:19:30.895429    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj" podUID="841d9a72-daa8-45f4-8734-710eb1ec82c6" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.62:8080/v2/models/isvc-xgboost-v2-runtime/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 28 14:19:34.857927 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:19:34.857875    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj"
Jul 28 14:19:34.976102 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:19:34.976072    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/841d9a72-daa8-45f4-8734-710eb1ec82c6-kube-rbac-proxy-sar-config\") pod \"841d9a72-daa8-45f4-8734-710eb1ec82c6\" (UID: \"841d9a72-daa8-45f4-8734-710eb1ec82c6\") "
Jul 28 14:19:34.976305 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:19:34.976124    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/841d9a72-daa8-45f4-8734-710eb1ec82c6-proxy-tls\") pod \"841d9a72-daa8-45f4-8734-710eb1ec82c6\" (UID: \"841d9a72-daa8-45f4-8734-710eb1ec82c6\") "
Jul 28 14:19:34.976305 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:19:34.976156    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/841d9a72-daa8-45f4-8734-710eb1ec82c6-kserve-provision-location\") pod \"841d9a72-daa8-45f4-8734-710eb1ec82c6\" (UID: \"841d9a72-daa8-45f4-8734-710eb1ec82c6\") "
Jul 28 14:19:34.976305 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:19:34.976204    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7g4q\" (UniqueName: \"kubernetes.io/projected/841d9a72-daa8-45f4-8734-710eb1ec82c6-kube-api-access-b7g4q\") pod \"841d9a72-daa8-45f4-8734-710eb1ec82c6\" (UID: \"841d9a72-daa8-45f4-8734-710eb1ec82c6\") "
Jul 28 14:19:34.976305 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:19:34.976227    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/841d9a72-daa8-45f4-8734-710eb1ec82c6-cabundle-cert\") pod \"841d9a72-daa8-45f4-8734-710eb1ec82c6\" (UID: \"841d9a72-daa8-45f4-8734-710eb1ec82c6\") "
Jul 28 14:19:34.976578 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:19:34.976552    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/841d9a72-daa8-45f4-8734-710eb1ec82c6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "841d9a72-daa8-45f4-8734-710eb1ec82c6" (UID: "841d9a72-daa8-45f4-8734-710eb1ec82c6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:19:34.976632 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:19:34.976605    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/841d9a72-daa8-45f4-8734-710eb1ec82c6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "841d9a72-daa8-45f4-8734-710eb1ec82c6" (UID: "841d9a72-daa8-45f4-8734-710eb1ec82c6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:19:34.976683 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:19:34.976609    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/841d9a72-daa8-45f4-8734-710eb1ec82c6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "841d9a72-daa8-45f4-8734-710eb1ec82c6" (UID: "841d9a72-daa8-45f4-8734-710eb1ec82c6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:19:34.978549 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:19:34.978522    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/841d9a72-daa8-45f4-8734-710eb1ec82c6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "841d9a72-daa8-45f4-8734-710eb1ec82c6" (UID: "841d9a72-daa8-45f4-8734-710eb1ec82c6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:19:34.978652 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:19:34.978560    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/841d9a72-daa8-45f4-8734-710eb1ec82c6-kube-api-access-b7g4q" (OuterVolumeSpecName: "kube-api-access-b7g4q") pod "841d9a72-daa8-45f4-8734-710eb1ec82c6" (UID: "841d9a72-daa8-45f4-8734-710eb1ec82c6"). InnerVolumeSpecName "kube-api-access-b7g4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:19:35.077182 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:19:35.077153    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7g4q\" (UniqueName: \"kubernetes.io/projected/841d9a72-daa8-45f4-8734-710eb1ec82c6-kube-api-access-b7g4q\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:19:35.077182 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:19:35.077178    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/841d9a72-daa8-45f4-8734-710eb1ec82c6-cabundle-cert\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:19:35.077182 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:19:35.077187    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/841d9a72-daa8-45f4-8734-710eb1ec82c6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:19:35.077420 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:19:35.077197    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/841d9a72-daa8-45f4-8734-710eb1ec82c6-proxy-tls\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:19:35.077420 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:19:35.077205    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/841d9a72-daa8-45f4-8734-710eb1ec82c6-kserve-provision-location\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:19:35.094088 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:19:35.094056    2554 generic.go:356] "Generic (PLEG): container finished" podID="841d9a72-daa8-45f4-8734-710eb1ec82c6" containerID="042c218550f7983d3abe148c9e13ba78f4ddf216f9e56451b5fd0e3519b893ee" exitCode=0
Jul 28 14:19:35.094218 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:19:35.094117    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj"
Jul 28 14:19:35.094218 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:19:35.094114    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj" event={"ID":"841d9a72-daa8-45f4-8734-710eb1ec82c6","Type":"ContainerDied","Data":"042c218550f7983d3abe148c9e13ba78f4ddf216f9e56451b5fd0e3519b893ee"}
Jul 28 14:19:35.094338 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:19:35.094301    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj" event={"ID":"841d9a72-daa8-45f4-8734-710eb1ec82c6","Type":"ContainerDied","Data":"7d7d43266173d694e501a3f52231469b19f1b763c18491a62dc421f1ddbb07d8"}
Jul 28 14:19:35.094393 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:19:35.094332    2554 scope.go:117] "RemoveContainer" containerID="5861ebe5e16d4acde294127c37433f12aba37357e9f87eb4276fa80fbe84e341"
Jul 28 14:19:35.103226 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:19:35.103207    2554 scope.go:117] "RemoveContainer" containerID="042c218550f7983d3abe148c9e13ba78f4ddf216f9e56451b5fd0e3519b893ee"
Jul 28 14:19:35.111028 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:19:35.111000    2554 scope.go:117] "RemoveContainer" containerID="e8b1580fb0a56affed91d9ed85bdf5d21bc9f75861369f2dea337f0726011228"
Jul 28 14:19:35.118058 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:19:35.118043    2554 scope.go:117] "RemoveContainer" containerID="5861ebe5e16d4acde294127c37433f12aba37357e9f87eb4276fa80fbe84e341"
Jul 28 14:19:35.118333 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:19:35.118313    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5861ebe5e16d4acde294127c37433f12aba37357e9f87eb4276fa80fbe84e341\": container with ID starting with 5861ebe5e16d4acde294127c37433f12aba37357e9f87eb4276fa80fbe84e341 not found: ID does not exist" containerID="5861ebe5e16d4acde294127c37433f12aba37357e9f87eb4276fa80fbe84e341"
Jul 28 14:19:35.118404 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:19:35.118340    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5861ebe5e16d4acde294127c37433f12aba37357e9f87eb4276fa80fbe84e341"} err="failed to get container status \"5861ebe5e16d4acde294127c37433f12aba37357e9f87eb4276fa80fbe84e341\": rpc error: code = NotFound desc = could not find container \"5861ebe5e16d4acde294127c37433f12aba37357e9f87eb4276fa80fbe84e341\": container with ID starting with 5861ebe5e16d4acde294127c37433f12aba37357e9f87eb4276fa80fbe84e341 not found: ID does not exist"
Jul 28 14:19:35.118404 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:19:35.118356    2554 scope.go:117] "RemoveContainer" containerID="042c218550f7983d3abe148c9e13ba78f4ddf216f9e56451b5fd0e3519b893ee"
Jul 28 14:19:35.118596 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:19:35.118576    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"042c218550f7983d3abe148c9e13ba78f4ddf216f9e56451b5fd0e3519b893ee\": container with ID starting with 042c218550f7983d3abe148c9e13ba78f4ddf216f9e56451b5fd0e3519b893ee not found: ID does not exist" containerID="042c218550f7983d3abe148c9e13ba78f4ddf216f9e56451b5fd0e3519b893ee"
Jul 28 14:19:35.118634 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:19:35.118608    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"042c218550f7983d3abe148c9e13ba78f4ddf216f9e56451b5fd0e3519b893ee"} err="failed to get container status \"042c218550f7983d3abe148c9e13ba78f4ddf216f9e56451b5fd0e3519b893ee\": rpc error: code = NotFound desc = could not find container \"042c218550f7983d3abe148c9e13ba78f4ddf216f9e56451b5fd0e3519b893ee\": container with ID starting with 042c218550f7983d3abe148c9e13ba78f4ddf216f9e56451b5fd0e3519b893ee not found: ID does not exist"
Jul 28 14:19:35.118634 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:19:35.118624    2554 scope.go:117] "RemoveContainer" containerID="e8b1580fb0a56affed91d9ed85bdf5d21bc9f75861369f2dea337f0726011228"
Jul 28 14:19:35.118855 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:19:35.118838    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8b1580fb0a56affed91d9ed85bdf5d21bc9f75861369f2dea337f0726011228\": container with ID starting with e8b1580fb0a56affed91d9ed85bdf5d21bc9f75861369f2dea337f0726011228 not found: ID does not exist" containerID="e8b1580fb0a56affed91d9ed85bdf5d21bc9f75861369f2dea337f0726011228"
Jul 28 14:19:35.118895 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:19:35.118860    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8b1580fb0a56affed91d9ed85bdf5d21bc9f75861369f2dea337f0726011228"} err="failed to get container status \"e8b1580fb0a56affed91d9ed85bdf5d21bc9f75861369f2dea337f0726011228\": rpc error: code = NotFound desc = could not find container \"e8b1580fb0a56affed91d9ed85bdf5d21bc9f75861369f2dea337f0726011228\": container with ID starting with e8b1580fb0a56affed91d9ed85bdf5d21bc9f75861369f2dea337f0726011228 not found: ID does not exist"
Jul 28 14:19:35.150497 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:19:35.150472    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj"]
Jul 28 14:19:35.170428 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:19:35.170404    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj"]
Jul 28 14:19:35.802623 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:19:35.802588    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="841d9a72-daa8-45f4-8734-710eb1ec82c6" path="/var/lib/kubelet/pods/841d9a72-daa8-45f4-8734-710eb1ec82c6/volumes"
Jul 28 14:19:35.848003 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:19:35.847969    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-fbxbj" podUID="841d9a72-daa8-45f4-8734-710eb1ec82c6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.62:8643/healthz\": context deadline exceeded"
Jul 28 14:21:08.125975 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:21:08.125929    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-vwfjc"]
Jul 28 14:21:08.126516 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:21:08.126438    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="841d9a72-daa8-45f4-8734-710eb1ec82c6" containerName="kube-rbac-proxy"
Jul 28 14:21:08.126516 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:21:08.126457    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="841d9a72-daa8-45f4-8734-710eb1ec82c6" containerName="kube-rbac-proxy"
Jul 28 14:21:08.126516 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:21:08.126481    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="841d9a72-daa8-45f4-8734-710eb1ec82c6" containerName="storage-initializer"
Jul 28 14:21:08.126516 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:21:08.126490    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="841d9a72-daa8-45f4-8734-710eb1ec82c6" containerName="storage-initializer"
Jul 28 14:21:08.126516 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:21:08.126503    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="841d9a72-daa8-45f4-8734-710eb1ec82c6" containerName="kserve-container"
Jul 28 14:21:08.126516 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:21:08.126512    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="841d9a72-daa8-45f4-8734-710eb1ec82c6" containerName="kserve-container"
Jul 28 14:21:08.126842 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:21:08.126594    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="841d9a72-daa8-45f4-8734-710eb1ec82c6" containerName="kserve-container"
Jul 28 14:21:08.126842 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:21:08.126610    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="841d9a72-daa8-45f4-8734-710eb1ec82c6" containerName="kube-rbac-proxy"
Jul 28 14:21:08.130071 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:21:08.130051    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-vwfjc"
Jul 28 14:21:08.132701 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:21:08.132679    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
Jul 28 14:21:08.132905 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:21:08.132873    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Jul 28 14:21:08.132990 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:21:08.132926    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 14:21:08.133586 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:21:08.133568    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\""
Jul 28 14:21:08.133586 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:21:08.133581    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rxfwd\""
Jul 28 14:21:08.133704 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:21:08.133569    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 14:21:08.147592 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:21:08.147570    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-vwfjc"]
Jul 28 14:21:08.255388 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:21:08.255351    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2f2a824-b672-414b-8b32-d82adae2965b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-vwfjc\" (UID: \"a2f2a824-b672-414b-8b32-d82adae2965b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-vwfjc"
Jul 28 14:21:08.255561 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:21:08.255398    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tt7g\" (UniqueName: \"kubernetes.io/projected/a2f2a824-b672-414b-8b32-d82adae2965b-kube-api-access-8tt7g\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-vwfjc\" (UID: \"a2f2a824-b672-414b-8b32-d82adae2965b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-vwfjc"
Jul 28 14:21:08.255561 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:21:08.255446    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2f2a824-b672-414b-8b32-d82adae2965b-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-vwfjc\" (UID: \"a2f2a824-b672-414b-8b32-d82adae2965b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-vwfjc"
Jul 28 14:21:08.255561 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:21:08.255491    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2f2a824-b672-414b-8b32-d82adae2965b-proxy-tls\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-vwfjc\" (UID: \"a2f2a824-b672-414b-8b32-d82adae2965b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-vwfjc"
Jul 28 14:21:08.356394 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:21:08.356357    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2f2a824-b672-414b-8b32-d82adae2965b-proxy-tls\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-vwfjc\" (UID: \"a2f2a824-b672-414b-8b32-d82adae2965b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-vwfjc"
Jul 28 14:21:08.356592 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:21:08.356424    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2f2a824-b672-414b-8b32-d82adae2965b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-vwfjc\" (UID: \"a2f2a824-b672-414b-8b32-d82adae2965b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-vwfjc"
Jul 28 14:21:08.356592 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:21:08.356455    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tt7g\" (UniqueName: \"kubernetes.io/projected/a2f2a824-b672-414b-8b32-d82adae2965b-kube-api-access-8tt7g\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-vwfjc\" (UID: \"a2f2a824-b672-414b-8b32-d82adae2965b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-vwfjc"
Jul 28 14:21:08.356592 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:21:08.356476    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2f2a824-b672-414b-8b32-d82adae2965b-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-vwfjc\" (UID: \"a2f2a824-b672-414b-8b32-d82adae2965b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-vwfjc"
Jul 28 14:21:08.356849 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:21:08.356829    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2f2a824-b672-414b-8b32-d82adae2965b-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-vwfjc\" (UID: \"a2f2a824-b672-414b-8b32-d82adae2965b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-vwfjc"
Jul 28 14:21:08.357135 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:21:08.357116    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2f2a824-b672-414b-8b32-d82adae2965b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-vwfjc\" (UID: \"a2f2a824-b672-414b-8b32-d82adae2965b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-vwfjc"
Jul 28 14:21:08.359014 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:21:08.358996    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2f2a824-b672-414b-8b32-d82adae2965b-proxy-tls\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-vwfjc\" (UID: \"a2f2a824-b672-414b-8b32-d82adae2965b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-vwfjc"
Jul 28 14:21:08.366506 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:21:08.366484    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tt7g\" (UniqueName: \"kubernetes.io/projected/a2f2a824-b672-414b-8b32-d82adae2965b-kube-api-access-8tt7g\") pod \"isvc-sklearn-s3-predictor-dd446d6b9-vwfjc\" (UID: \"a2f2a824-b672-414b-8b32-d82adae2965b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-vwfjc"
Jul 28 14:21:08.440511 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:21:08.440429    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-vwfjc"
Jul 28 14:21:08.575677 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:21:08.575636    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-vwfjc"]
Jul 28 14:21:08.580046 ip-10-0-129-177 kubenswrapper[2554]: W0728 14:21:08.580017    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2f2a824_b672_414b_8b32_d82adae2965b.slice/crio-aaa9b2e22ecb67cd288fc21a5ee393e42e18cad146ff25cdfea24e1d01946953 WatchSource:0}: Error finding container aaa9b2e22ecb67cd288fc21a5ee393e42e18cad146ff25cdfea24e1d01946953: Status 404 returned error can't find the container with id aaa9b2e22ecb67cd288fc21a5ee393e42e18cad146ff25cdfea24e1d01946953
Jul 28 14:21:09.457717 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:21:09.457676    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-vwfjc" event={"ID":"a2f2a824-b672-414b-8b32-d82adae2965b","Type":"ContainerStarted","Data":"d0f45d4ebdeec7f5dadd717b978f8fa52c3f1935e1a4959c15721c6200af9e83"}
Jul 28 14:21:09.458145 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:21:09.457737    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-vwfjc" event={"ID":"a2f2a824-b672-414b-8b32-d82adae2965b","Type":"ContainerStarted","Data":"aaa9b2e22ecb67cd288fc21a5ee393e42e18cad146ff25cdfea24e1d01946953"}
Jul 28 14:21:10.462788 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:21:10.462755    2554 generic.go:356] "Generic (PLEG): container finished" podID="a2f2a824-b672-414b-8b32-d82adae2965b" containerID="d0f45d4ebdeec7f5dadd717b978f8fa52c3f1935e1a4959c15721c6200af9e83" exitCode=0
Jul 28 14:21:10.463179 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:21:10.462809    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-vwfjc" event={"ID":"a2f2a824-b672-414b-8b32-d82adae2965b","Type":"ContainerDied","Data":"d0f45d4ebdeec7f5dadd717b978f8fa52c3f1935e1a4959c15721c6200af9e83"}
Jul 28 14:21:11.468937 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:21:11.468904    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-vwfjc" event={"ID":"a2f2a824-b672-414b-8b32-d82adae2965b","Type":"ContainerStarted","Data":"39784efa0037a72df97d714352a7eae679ddbf7f1710624d0ce7b27c804c3b57"}
Jul 28 14:21:11.469344 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:21:11.468957    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-vwfjc" event={"ID":"a2f2a824-b672-414b-8b32-d82adae2965b","Type":"ContainerStarted","Data":"4bce62b6df527ae645a46255729f5485cf2e028d4ac1ed7da2c865ae02909f3a"}
Jul 28 14:21:11.469344 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:21:11.468983    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-vwfjc"
Jul 28 14:21:11.469344 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:21:11.469019    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-vwfjc"
Jul 28 14:21:11.470494 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:21:11.470469    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-vwfjc" podUID="a2f2a824-b672-414b-8b32-d82adae2965b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Jul 28 14:21:11.489487 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:21:11.489437    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-vwfjc" podStartSLOduration=2.588614022 podStartE2EDuration="3.489425543s" totalImagesPullingTime="0s" totalInitContainerRuntime="900.811521ms" isStatefulPod=false podCreationTimestamp="2026-07-28 14:21:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:21:11.488035807 +0000 UTC m=+5268.273910918" watchObservedRunningTime="2026-07-28 14:21:11.489425543 +0000 UTC m=+5268.275300654"
Jul 28 14:21:12.473033 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:21:12.472992    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-vwfjc" podUID="a2f2a824-b672-414b-8b32-d82adae2965b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Jul 28 14:21:17.477429 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:21:17.477402    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-vwfjc"
Jul 28 14:21:17.477981 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:21:17.477953    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-vwfjc" podUID="a2f2a824-b672-414b-8b32-d82adae2965b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Jul 28 14:21:27.478114 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:21:27.478079    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-vwfjc" podUID="a2f2a824-b672-414b-8b32-d82adae2965b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Jul 28 14:21:37.478927 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:21:37.478886    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-vwfjc" podUID="a2f2a824-b672-414b-8b32-d82adae2965b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Jul 28 14:21:47.477865 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:21:47.477781    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-vwfjc" podUID="a2f2a824-b672-414b-8b32-d82adae2965b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Jul 28 14:21:57.478369 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:21:57.478329    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-vwfjc" podUID="a2f2a824-b672-414b-8b32-d82adae2965b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Jul 28 14:22:07.477900 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:07.477861    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-vwfjc" podUID="a2f2a824-b672-414b-8b32-d82adae2965b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Jul 28 14:22:17.478031 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:17.477987    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-vwfjc" podUID="a2f2a824-b672-414b-8b32-d82adae2965b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Jul 28 14:22:24.797435 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:24.797396    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-vwfjc"
Jul 28 14:22:28.185465 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:28.185431    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-vwfjc"]
Jul 28 14:22:28.185955 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:28.185679    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-vwfjc" podUID="a2f2a824-b672-414b-8b32-d82adae2965b" containerName="kserve-container" containerID="cri-o://4bce62b6df527ae645a46255729f5485cf2e028d4ac1ed7da2c865ae02909f3a" gracePeriod=30
Jul 28 14:22:28.185955 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:28.185739    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-vwfjc" podUID="a2f2a824-b672-414b-8b32-d82adae2965b" containerName="kube-rbac-proxy" containerID="cri-o://39784efa0037a72df97d714352a7eae679ddbf7f1710624d0ce7b27c804c3b57" gracePeriod=30
Jul 28 14:22:28.367722 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:28.367688    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4"]
Jul 28 14:22:28.371218 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:28.371200    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4"
Jul 28 14:22:28.373868 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:28.373834    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
Jul 28 14:22:28.373984 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:28.373934    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 14:22:28.374042 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:28.373980    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
Jul 28 14:22:28.385359 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:28.385332    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4"]
Jul 28 14:22:28.432251 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:28.432214    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4hbb\" (UniqueName: \"kubernetes.io/projected/b70a5d7a-a4e4-4dd0-9d64-41146c19a28e-kube-api-access-f4hbb\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4\" (UID: \"b70a5d7a-a4e4-4dd0-9d64-41146c19a28e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4"
Jul 28 14:22:28.432442 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:28.432263    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b70a5d7a-a4e4-4dd0-9d64-41146c19a28e-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4\" (UID: \"b70a5d7a-a4e4-4dd0-9d64-41146c19a28e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4"
Jul 28 14:22:28.432442 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:28.432325    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b70a5d7a-a4e4-4dd0-9d64-41146c19a28e-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4\" (UID: \"b70a5d7a-a4e4-4dd0-9d64-41146c19a28e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4"
Jul 28 14:22:28.432442 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:28.432390    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b70a5d7a-a4e4-4dd0-9d64-41146c19a28e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4\" (UID: \"b70a5d7a-a4e4-4dd0-9d64-41146c19a28e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4"
Jul 28 14:22:28.432442 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:28.432428    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b70a5d7a-a4e4-4dd0-9d64-41146c19a28e-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4\" (UID: \"b70a5d7a-a4e4-4dd0-9d64-41146c19a28e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4"
Jul 28 14:22:28.533251 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:28.533218    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b70a5d7a-a4e4-4dd0-9d64-41146c19a28e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4\" (UID: \"b70a5d7a-a4e4-4dd0-9d64-41146c19a28e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4"
Jul 28 14:22:28.533453 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:28.533290    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b70a5d7a-a4e4-4dd0-9d64-41146c19a28e-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4\" (UID: \"b70a5d7a-a4e4-4dd0-9d64-41146c19a28e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4"
Jul 28 14:22:28.533453 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:28.533334    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4hbb\" (UniqueName: \"kubernetes.io/projected/b70a5d7a-a4e4-4dd0-9d64-41146c19a28e-kube-api-access-f4hbb\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4\" (UID: \"b70a5d7a-a4e4-4dd0-9d64-41146c19a28e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4"
Jul 28 14:22:28.533571 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:28.533469    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b70a5d7a-a4e4-4dd0-9d64-41146c19a28e-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4\" (UID: \"b70a5d7a-a4e4-4dd0-9d64-41146c19a28e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4"
Jul 28 14:22:28.533571 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:28.533517    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b70a5d7a-a4e4-4dd0-9d64-41146c19a28e-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4\" (UID: \"b70a5d7a-a4e4-4dd0-9d64-41146c19a28e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4"
Jul 28 14:22:28.533684 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:22:28.533639    2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-global-pass-predictor-serving-cert" not found
Jul 28 14:22:28.533740 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:22:28.533693    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b70a5d7a-a4e4-4dd0-9d64-41146c19a28e-proxy-tls podName:b70a5d7a-a4e4-4dd0-9d64-41146c19a28e nodeName:}" failed. No retries permitted until 2026-07-28 14:22:29.03367668 +0000 UTC m=+5345.819551769 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b70a5d7a-a4e4-4dd0-9d64-41146c19a28e-proxy-tls") pod "isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4" (UID: "b70a5d7a-a4e4-4dd0-9d64-41146c19a28e") : secret "isvc-sklearn-s3-tls-global-pass-predictor-serving-cert" not found
Jul 28 14:22:28.533895 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:28.533876    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b70a5d7a-a4e4-4dd0-9d64-41146c19a28e-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4\" (UID: \"b70a5d7a-a4e4-4dd0-9d64-41146c19a28e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4"
Jul 28 14:22:28.534031 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:28.534008    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b70a5d7a-a4e4-4dd0-9d64-41146c19a28e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4\" (UID: \"b70a5d7a-a4e4-4dd0-9d64-41146c19a28e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4"
Jul 28 14:22:28.534092 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:28.534064    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b70a5d7a-a4e4-4dd0-9d64-41146c19a28e-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4\" (UID: \"b70a5d7a-a4e4-4dd0-9d64-41146c19a28e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4"
Jul 28 14:22:28.550998 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:28.550976    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4hbb\" (UniqueName: \"kubernetes.io/projected/b70a5d7a-a4e4-4dd0-9d64-41146c19a28e-kube-api-access-f4hbb\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4\" (UID: \"b70a5d7a-a4e4-4dd0-9d64-41146c19a28e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4"
Jul 28 14:22:28.775406 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:28.775368    2554 generic.go:356] "Generic (PLEG): container finished" podID="a2f2a824-b672-414b-8b32-d82adae2965b" containerID="39784efa0037a72df97d714352a7eae679ddbf7f1710624d0ce7b27c804c3b57" exitCode=2
Jul 28 14:22:28.775406 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:28.775399    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-vwfjc" event={"ID":"a2f2a824-b672-414b-8b32-d82adae2965b","Type":"ContainerDied","Data":"39784efa0037a72df97d714352a7eae679ddbf7f1710624d0ce7b27c804c3b57"}
Jul 28 14:22:29.037945 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:29.037910    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b70a5d7a-a4e4-4dd0-9d64-41146c19a28e-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4\" (UID: \"b70a5d7a-a4e4-4dd0-9d64-41146c19a28e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4"
Jul 28 14:22:29.038146 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:22:29.038076    2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-global-pass-predictor-serving-cert" not found
Jul 28 14:22:29.038214 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:22:29.038151    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b70a5d7a-a4e4-4dd0-9d64-41146c19a28e-proxy-tls podName:b70a5d7a-a4e4-4dd0-9d64-41146c19a28e nodeName:}" failed. No retries permitted until 2026-07-28 14:22:30.03813095 +0000 UTC m=+5346.824006039 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b70a5d7a-a4e4-4dd0-9d64-41146c19a28e-proxy-tls") pod "isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4" (UID: "b70a5d7a-a4e4-4dd0-9d64-41146c19a28e") : secret "isvc-sklearn-s3-tls-global-pass-predictor-serving-cert" not found
Jul 28 14:22:30.046355 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:30.046323    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b70a5d7a-a4e4-4dd0-9d64-41146c19a28e-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4\" (UID: \"b70a5d7a-a4e4-4dd0-9d64-41146c19a28e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4"
Jul 28 14:22:30.048954 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:30.048930    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b70a5d7a-a4e4-4dd0-9d64-41146c19a28e-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4\" (UID: \"b70a5d7a-a4e4-4dd0-9d64-41146c19a28e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4"
Jul 28 14:22:30.181364 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:30.181331    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4"
Jul 28 14:22:30.319574 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:30.319498    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4"]
Jul 28 14:22:30.322837 ip-10-0-129-177 kubenswrapper[2554]: W0728 14:22:30.322796    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb70a5d7a_a4e4_4dd0_9d64_41146c19a28e.slice/crio-60a79fac0f6b0b2284f92be48d2e32047f7d04b218eaa92b707a715d3e3c412f WatchSource:0}: Error finding container 60a79fac0f6b0b2284f92be48d2e32047f7d04b218eaa92b707a715d3e3c412f: Status 404 returned error can't find the container with id 60a79fac0f6b0b2284f92be48d2e32047f7d04b218eaa92b707a715d3e3c412f
Jul 28 14:22:30.785264 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:30.785227    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4" event={"ID":"b70a5d7a-a4e4-4dd0-9d64-41146c19a28e","Type":"ContainerStarted","Data":"51a83e3a53b78eab7d5b63b3fb8067c3aba7d239ff6f7e4d451227e22e0c0fca"}
Jul 28 14:22:30.785506 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:30.785322    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4" event={"ID":"b70a5d7a-a4e4-4dd0-9d64-41146c19a28e","Type":"ContainerStarted","Data":"60a79fac0f6b0b2284f92be48d2e32047f7d04b218eaa92b707a715d3e3c412f"}
Jul 28 14:22:31.791922 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:31.791883    2554 generic.go:356] "Generic (PLEG): container finished" podID="b70a5d7a-a4e4-4dd0-9d64-41146c19a28e" containerID="51a83e3a53b78eab7d5b63b3fb8067c3aba7d239ff6f7e4d451227e22e0c0fca" exitCode=0
Jul 28 14:22:31.792421 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:31.791965    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4" event={"ID":"b70a5d7a-a4e4-4dd0-9d64-41146c19a28e","Type":"ContainerDied","Data":"51a83e3a53b78eab7d5b63b3fb8067c3aba7d239ff6f7e4d451227e22e0c0fca"}
Jul 28 14:22:32.473777 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:32.473737    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-vwfjc" podUID="a2f2a824-b672-414b-8b32-d82adae2965b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.63:8643/healthz\": dial tcp 10.134.0.63:8643: connect: connection refused"
Jul 28 14:22:32.800560 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:32.800533    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4" event={"ID":"b70a5d7a-a4e4-4dd0-9d64-41146c19a28e","Type":"ContainerStarted","Data":"443bc2b24d83311042a015801123d5349c4d8303f2d6ec88eccbaf27496b3d57"}
Jul 28 14:22:32.800851 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:32.800593    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4" event={"ID":"b70a5d7a-a4e4-4dd0-9d64-41146c19a28e","Type":"ContainerStarted","Data":"35a04b815f9ce5f2d6a9bad2ab927b4035d8dd46ff89584e84ccfbbcaa17ceda"}
Jul 28 14:22:32.800851 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:32.800657    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4"
Jul 28 14:22:32.821899 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:32.821847    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4" podStartSLOduration=3.891179844 podStartE2EDuration="4.821828715s" totalImagesPullingTime="0s" totalInitContainerRuntime="930.648871ms" isStatefulPod=false podCreationTimestamp="2026-07-28 14:22:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:22:32.819498857 +0000 UTC m=+5349.605374015" watchObservedRunningTime="2026-07-28 14:22:32.821828715 +0000 UTC m=+5349.607703828"
Jul 28 14:22:32.955772 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:32.955749    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-vwfjc"
Jul 28 14:22:32.973241 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:32.973214    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tt7g\" (UniqueName: \"kubernetes.io/projected/a2f2a824-b672-414b-8b32-d82adae2965b-kube-api-access-8tt7g\") pod \"a2f2a824-b672-414b-8b32-d82adae2965b\" (UID: \"a2f2a824-b672-414b-8b32-d82adae2965b\") "
Jul 28 14:22:32.973373 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:32.973306    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2f2a824-b672-414b-8b32-d82adae2965b-kserve-provision-location\") pod \"a2f2a824-b672-414b-8b32-d82adae2965b\" (UID: \"a2f2a824-b672-414b-8b32-d82adae2965b\") "
Jul 28 14:22:32.973419 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:32.973396    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2f2a824-b672-414b-8b32-d82adae2965b-kube-rbac-proxy-sar-config\") pod \"a2f2a824-b672-414b-8b32-d82adae2965b\" (UID: \"a2f2a824-b672-414b-8b32-d82adae2965b\") "
Jul 28 14:22:32.973453 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:32.973442    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2f2a824-b672-414b-8b32-d82adae2965b-proxy-tls\") pod \"a2f2a824-b672-414b-8b32-d82adae2965b\" (UID: \"a2f2a824-b672-414b-8b32-d82adae2965b\") "
Jul 28 14:22:32.973661 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:32.973571    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2f2a824-b672-414b-8b32-d82adae2965b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a2f2a824-b672-414b-8b32-d82adae2965b" (UID: "a2f2a824-b672-414b-8b32-d82adae2965b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:22:32.973782 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:32.973713    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2f2a824-b672-414b-8b32-d82adae2965b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a2f2a824-b672-414b-8b32-d82adae2965b" (UID: "a2f2a824-b672-414b-8b32-d82adae2965b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:22:32.975878 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:32.975857    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2f2a824-b672-414b-8b32-d82adae2965b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a2f2a824-b672-414b-8b32-d82adae2965b" (UID: "a2f2a824-b672-414b-8b32-d82adae2965b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:22:32.976120 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:32.976103    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2f2a824-b672-414b-8b32-d82adae2965b-kube-api-access-8tt7g" (OuterVolumeSpecName: "kube-api-access-8tt7g") pod "a2f2a824-b672-414b-8b32-d82adae2965b" (UID: "a2f2a824-b672-414b-8b32-d82adae2965b"). InnerVolumeSpecName "kube-api-access-8tt7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:22:33.074715 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:33.074625    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tt7g\" (UniqueName: \"kubernetes.io/projected/a2f2a824-b672-414b-8b32-d82adae2965b-kube-api-access-8tt7g\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:22:33.074715 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:33.074656    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2f2a824-b672-414b-8b32-d82adae2965b-kserve-provision-location\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:22:33.074715 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:33.074669    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2f2a824-b672-414b-8b32-d82adae2965b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:22:33.074715 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:33.074680    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2f2a824-b672-414b-8b32-d82adae2965b-proxy-tls\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:22:33.806652 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:33.806615    2554 generic.go:356] "Generic (PLEG): container finished" podID="a2f2a824-b672-414b-8b32-d82adae2965b" containerID="4bce62b6df527ae645a46255729f5485cf2e028d4ac1ed7da2c865ae02909f3a" exitCode=0
Jul 28 14:22:33.807145 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:33.806669    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-vwfjc"
Jul 28 14:22:33.807145 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:33.806705    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-vwfjc" event={"ID":"a2f2a824-b672-414b-8b32-d82adae2965b","Type":"ContainerDied","Data":"4bce62b6df527ae645a46255729f5485cf2e028d4ac1ed7da2c865ae02909f3a"}
Jul 28 14:22:33.807145 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:33.806772    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-vwfjc" event={"ID":"a2f2a824-b672-414b-8b32-d82adae2965b","Type":"ContainerDied","Data":"aaa9b2e22ecb67cd288fc21a5ee393e42e18cad146ff25cdfea24e1d01946953"}
Jul 28 14:22:33.807145 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:33.806815    2554 scope.go:117] "RemoveContainer" containerID="39784efa0037a72df97d714352a7eae679ddbf7f1710624d0ce7b27c804c3b57"
Jul 28 14:22:33.807438 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:33.807413    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4"
Jul 28 14:22:33.809070 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:33.809034    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4" podUID="b70a5d7a-a4e4-4dd0-9d64-41146c19a28e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Jul 28 14:22:33.816058 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:33.815936    2554 scope.go:117] "RemoveContainer" containerID="4bce62b6df527ae645a46255729f5485cf2e028d4ac1ed7da2c865ae02909f3a"
Jul 28 14:22:33.823473 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:33.823457    2554 scope.go:117] "RemoveContainer" containerID="d0f45d4ebdeec7f5dadd717b978f8fa52c3f1935e1a4959c15721c6200af9e83"
Jul 28 14:22:33.826795 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:33.826769    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-vwfjc"]
Jul 28 14:22:33.830782 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:33.830759    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-dd446d6b9-vwfjc"]
Jul 28 14:22:33.831452 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:33.831428    2554 scope.go:117] "RemoveContainer" containerID="39784efa0037a72df97d714352a7eae679ddbf7f1710624d0ce7b27c804c3b57"
Jul 28 14:22:33.831742 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:22:33.831725    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39784efa0037a72df97d714352a7eae679ddbf7f1710624d0ce7b27c804c3b57\": container with ID starting with 39784efa0037a72df97d714352a7eae679ddbf7f1710624d0ce7b27c804c3b57 not found: ID does not exist" containerID="39784efa0037a72df97d714352a7eae679ddbf7f1710624d0ce7b27c804c3b57"
Jul 28 14:22:33.831791 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:33.831751    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39784efa0037a72df97d714352a7eae679ddbf7f1710624d0ce7b27c804c3b57"} err="failed to get container status \"39784efa0037a72df97d714352a7eae679ddbf7f1710624d0ce7b27c804c3b57\": rpc error: code = NotFound desc = could not find container \"39784efa0037a72df97d714352a7eae679ddbf7f1710624d0ce7b27c804c3b57\": container with ID starting with 39784efa0037a72df97d714352a7eae679ddbf7f1710624d0ce7b27c804c3b57 not found: ID does not exist"
Jul 28 14:22:33.831791 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:33.831769    2554 scope.go:117] "RemoveContainer" containerID="4bce62b6df527ae645a46255729f5485cf2e028d4ac1ed7da2c865ae02909f3a"
Jul 28 14:22:33.832041 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:22:33.832024    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bce62b6df527ae645a46255729f5485cf2e028d4ac1ed7da2c865ae02909f3a\": container with ID starting with 4bce62b6df527ae645a46255729f5485cf2e028d4ac1ed7da2c865ae02909f3a not found: ID does not exist" containerID="4bce62b6df527ae645a46255729f5485cf2e028d4ac1ed7da2c865ae02909f3a"
Jul 28 14:22:33.832094 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:33.832049    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bce62b6df527ae645a46255729f5485cf2e028d4ac1ed7da2c865ae02909f3a"} err="failed to get container status \"4bce62b6df527ae645a46255729f5485cf2e028d4ac1ed7da2c865ae02909f3a\": rpc error: code = NotFound desc = could not find container \"4bce62b6df527ae645a46255729f5485cf2e028d4ac1ed7da2c865ae02909f3a\": container with ID starting with 4bce62b6df527ae645a46255729f5485cf2e028d4ac1ed7da2c865ae02909f3a not found: ID does not exist"
Jul 28 14:22:33.832094 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:33.832064    2554 scope.go:117] "RemoveContainer" containerID="d0f45d4ebdeec7f5dadd717b978f8fa52c3f1935e1a4959c15721c6200af9e83"
Jul 28 14:22:33.832317 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:22:33.832299    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0f45d4ebdeec7f5dadd717b978f8fa52c3f1935e1a4959c15721c6200af9e83\": container with ID starting with d0f45d4ebdeec7f5dadd717b978f8fa52c3f1935e1a4959c15721c6200af9e83 not found: ID does not exist" containerID="d0f45d4ebdeec7f5dadd717b978f8fa52c3f1935e1a4959c15721c6200af9e83"
Jul 28 14:22:33.832424 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:33.832326    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0f45d4ebdeec7f5dadd717b978f8fa52c3f1935e1a4959c15721c6200af9e83"} err="failed to get container status \"d0f45d4ebdeec7f5dadd717b978f8fa52c3f1935e1a4959c15721c6200af9e83\": rpc error: code = NotFound desc = could not find container \"d0f45d4ebdeec7f5dadd717b978f8fa52c3f1935e1a4959c15721c6200af9e83\": container with ID starting with d0f45d4ebdeec7f5dadd717b978f8fa52c3f1935e1a4959c15721c6200af9e83 not found: ID does not exist"
Jul 28 14:22:34.811858 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:34.811820    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4" podUID="b70a5d7a-a4e4-4dd0-9d64-41146c19a28e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Jul 28 14:22:35.801855 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:35.801822    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2f2a824-b672-414b-8b32-d82adae2965b" path="/var/lib/kubelet/pods/a2f2a824-b672-414b-8b32-d82adae2965b/volumes"
Jul 28 14:22:39.816137 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:39.816109    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4"
Jul 28 14:22:39.816634 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:39.816610    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4" podUID="b70a5d7a-a4e4-4dd0-9d64-41146c19a28e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Jul 28 14:22:49.816730 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:49.816692    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4" podUID="b70a5d7a-a4e4-4dd0-9d64-41146c19a28e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Jul 28 14:22:59.816872 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:22:59.816835    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4" podUID="b70a5d7a-a4e4-4dd0-9d64-41146c19a28e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Jul 28 14:23:09.816589 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:23:09.816548    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4" podUID="b70a5d7a-a4e4-4dd0-9d64-41146c19a28e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Jul 28 14:23:19.817238 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:23:19.817145    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4" podUID="b70a5d7a-a4e4-4dd0-9d64-41146c19a28e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Jul 28 14:23:29.816655 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:23:29.816616    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4" podUID="b70a5d7a-a4e4-4dd0-9d64-41146c19a28e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Jul 28 14:23:39.817129 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:23:39.817090    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4" podUID="b70a5d7a-a4e4-4dd0-9d64-41146c19a28e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Jul 28 14:23:49.801710 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:23:49.801682    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4"
Jul 28 14:23:58.450399 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:23:58.450364    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4"]
Jul 28 14:23:58.450913 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:23:58.450608    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4" podUID="b70a5d7a-a4e4-4dd0-9d64-41146c19a28e" containerName="kserve-container" containerID="cri-o://35a04b815f9ce5f2d6a9bad2ab927b4035d8dd46ff89584e84ccfbbcaa17ceda" gracePeriod=30
Jul 28 14:23:58.450913 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:23:58.450692    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4" podUID="b70a5d7a-a4e4-4dd0-9d64-41146c19a28e" containerName="kube-rbac-proxy" containerID="cri-o://443bc2b24d83311042a015801123d5349c4d8303f2d6ec88eccbaf27496b3d57" gracePeriod=30
Jul 28 14:23:59.173134 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:23:59.173099    2554 generic.go:356] "Generic (PLEG): container finished" podID="b70a5d7a-a4e4-4dd0-9d64-41146c19a28e" containerID="443bc2b24d83311042a015801123d5349c4d8303f2d6ec88eccbaf27496b3d57" exitCode=2
Jul 28 14:23:59.173330 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:23:59.173175    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4" event={"ID":"b70a5d7a-a4e4-4dd0-9d64-41146c19a28e","Type":"ContainerDied","Data":"443bc2b24d83311042a015801123d5349c4d8303f2d6ec88eccbaf27496b3d57"}
Jul 28 14:23:59.518399 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:23:59.518365    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-pdngf"]
Jul 28 14:23:59.518876 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:23:59.518860    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2f2a824-b672-414b-8b32-d82adae2965b" containerName="storage-initializer"
Jul 28 14:23:59.518949 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:23:59.518879    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2f2a824-b672-414b-8b32-d82adae2965b" containerName="storage-initializer"
Jul 28 14:23:59.518949 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:23:59.518900    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2f2a824-b672-414b-8b32-d82adae2965b" containerName="kserve-container"
Jul 28 14:23:59.518949 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:23:59.518911    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2f2a824-b672-414b-8b32-d82adae2965b" containerName="kserve-container"
Jul 28 14:23:59.518949 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:23:59.518924    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2f2a824-b672-414b-8b32-d82adae2965b" containerName="kube-rbac-proxy"
Jul 28 14:23:59.518949 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:23:59.518935    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2f2a824-b672-414b-8b32-d82adae2965b" containerName="kube-rbac-proxy"
Jul 28 14:23:59.519204 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:23:59.519028    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2f2a824-b672-414b-8b32-d82adae2965b" containerName="kube-rbac-proxy"
Jul 28 14:23:59.519204 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:23:59.519040    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2f2a824-b672-414b-8b32-d82adae2965b" containerName="kserve-container"
Jul 28 14:23:59.522405 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:23:59.522383    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-pdngf"
Jul 28 14:23:59.524837 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:23:59.524819    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
Jul 28 14:23:59.524955 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:23:59.524858    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
Jul 28 14:23:59.530534 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:23:59.530505    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-pdngf"]
Jul 28 14:23:59.598658 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:23:59.598627    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bad7a490-8b5a-4e9e-be80-61e3579fcf89-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-pdngf\" (UID: \"bad7a490-8b5a-4e9e-be80-61e3579fcf89\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-pdngf"
Jul 28 14:23:59.598814 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:23:59.598664    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bad7a490-8b5a-4e9e-be80-61e3579fcf89-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-pdngf\" (UID: \"bad7a490-8b5a-4e9e-be80-61e3579fcf89\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-pdngf"
Jul 28 14:23:59.598814 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:23:59.598689    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bad7a490-8b5a-4e9e-be80-61e3579fcf89-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-pdngf\" (UID: \"bad7a490-8b5a-4e9e-be80-61e3579fcf89\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-pdngf"
Jul 28 14:23:59.598814 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:23:59.598757    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx5wk\" (UniqueName: \"kubernetes.io/projected/bad7a490-8b5a-4e9e-be80-61e3579fcf89-kube-api-access-nx5wk\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-pdngf\" (UID: \"bad7a490-8b5a-4e9e-be80-61e3579fcf89\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-pdngf"
Jul 28 14:23:59.699500 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:23:59.699468    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bad7a490-8b5a-4e9e-be80-61e3579fcf89-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-pdngf\" (UID: \"bad7a490-8b5a-4e9e-be80-61e3579fcf89\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-pdngf"
Jul 28 14:23:59.699500 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:23:59.699504    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bad7a490-8b5a-4e9e-be80-61e3579fcf89-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-pdngf\" (UID: \"bad7a490-8b5a-4e9e-be80-61e3579fcf89\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-pdngf"
Jul 28 14:23:59.699770 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:23:59.699532    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bad7a490-8b5a-4e9e-be80-61e3579fcf89-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-pdngf\" (UID: \"bad7a490-8b5a-4e9e-be80-61e3579fcf89\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-pdngf"
Jul 28 14:23:59.699770 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:23:59.699566    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx5wk\" (UniqueName: \"kubernetes.io/projected/bad7a490-8b5a-4e9e-be80-61e3579fcf89-kube-api-access-nx5wk\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-pdngf\" (UID: \"bad7a490-8b5a-4e9e-be80-61e3579fcf89\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-pdngf"
Jul 28 14:23:59.699770 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:23:59.699647    2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
Jul 28 14:23:59.699770 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:23:59.699726    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bad7a490-8b5a-4e9e-be80-61e3579fcf89-proxy-tls podName:bad7a490-8b5a-4e9e-be80-61e3579fcf89 nodeName:}" failed. No retries permitted until 2026-07-28 14:24:00.199706772 +0000 UTC m=+5436.985581875 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bad7a490-8b5a-4e9e-be80-61e3579fcf89-proxy-tls") pod "isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-pdngf" (UID: "bad7a490-8b5a-4e9e-be80-61e3579fcf89") : secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
Jul 28 14:23:59.699997 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:23:59.699983    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bad7a490-8b5a-4e9e-be80-61e3579fcf89-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-pdngf\" (UID: \"bad7a490-8b5a-4e9e-be80-61e3579fcf89\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-pdngf"
Jul 28 14:23:59.700289 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:23:59.700249    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bad7a490-8b5a-4e9e-be80-61e3579fcf89-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-pdngf\" (UID: \"bad7a490-8b5a-4e9e-be80-61e3579fcf89\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-pdngf"
Jul 28 14:23:59.708911 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:23:59.708887    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx5wk\" (UniqueName: \"kubernetes.io/projected/bad7a490-8b5a-4e9e-be80-61e3579fcf89-kube-api-access-nx5wk\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-pdngf\" (UID: \"bad7a490-8b5a-4e9e-be80-61e3579fcf89\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-pdngf"
Jul 28 14:23:59.797067 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:23:59.796986    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4" podUID="b70a5d7a-a4e4-4dd0-9d64-41146c19a28e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Jul 28 14:23:59.812550 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:23:59.812520    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4" podUID="b70a5d7a-a4e4-4dd0-9d64-41146c19a28e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.64:8643/healthz\": dial tcp 10.134.0.64:8643: connect: connection refused"
Jul 28 14:24:00.203208 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:00.203172    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bad7a490-8b5a-4e9e-be80-61e3579fcf89-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-pdngf\" (UID: \"bad7a490-8b5a-4e9e-be80-61e3579fcf89\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-pdngf"
Jul 28 14:24:00.205730 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:00.205710    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bad7a490-8b5a-4e9e-be80-61e3579fcf89-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-pdngf\" (UID: \"bad7a490-8b5a-4e9e-be80-61e3579fcf89\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-pdngf"
Jul 28 14:24:00.435150 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:00.435112    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-pdngf"
Jul 28 14:24:00.564906 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:00.564876    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-pdngf"]
Jul 28 14:24:00.568291 ip-10-0-129-177 kubenswrapper[2554]: W0728 14:24:00.568247    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbad7a490_8b5a_4e9e_be80_61e3579fcf89.slice/crio-a6cfb738d07c3e9cf1a45507079d7039fc21ef6d34de5c4e915ce05557f4dcb8 WatchSource:0}: Error finding container a6cfb738d07c3e9cf1a45507079d7039fc21ef6d34de5c4e915ce05557f4dcb8: Status 404 returned error can't find the container with id a6cfb738d07c3e9cf1a45507079d7039fc21ef6d34de5c4e915ce05557f4dcb8
Jul 28 14:24:01.184227 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:01.184190    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-pdngf" event={"ID":"bad7a490-8b5a-4e9e-be80-61e3579fcf89","Type":"ContainerStarted","Data":"ed8467d1608da61868354d4fb25d9689de72b9e7fc2b34bf85c06b70cbcf8488"}
Jul 28 14:24:01.184470 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:01.184246    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-pdngf" event={"ID":"bad7a490-8b5a-4e9e-be80-61e3579fcf89","Type":"ContainerStarted","Data":"a6cfb738d07c3e9cf1a45507079d7039fc21ef6d34de5c4e915ce05557f4dcb8"}
Jul 28 14:24:03.109854 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:03.109831    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4"
Jul 28 14:24:03.194898 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:03.194869    2554 generic.go:356] "Generic (PLEG): container finished" podID="b70a5d7a-a4e4-4dd0-9d64-41146c19a28e" containerID="35a04b815f9ce5f2d6a9bad2ab927b4035d8dd46ff89584e84ccfbbcaa17ceda" exitCode=0
Jul 28 14:24:03.195060 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:03.194928    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4"
Jul 28 14:24:03.195060 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:03.194949    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4" event={"ID":"b70a5d7a-a4e4-4dd0-9d64-41146c19a28e","Type":"ContainerDied","Data":"35a04b815f9ce5f2d6a9bad2ab927b4035d8dd46ff89584e84ccfbbcaa17ceda"}
Jul 28 14:24:03.195060 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:03.195007    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4" event={"ID":"b70a5d7a-a4e4-4dd0-9d64-41146c19a28e","Type":"ContainerDied","Data":"60a79fac0f6b0b2284f92be48d2e32047f7d04b218eaa92b707a715d3e3c412f"}
Jul 28 14:24:03.195060 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:03.195033    2554 scope.go:117] "RemoveContainer" containerID="443bc2b24d83311042a015801123d5349c4d8303f2d6ec88eccbaf27496b3d57"
Jul 28 14:24:03.203502 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:03.203486    2554 scope.go:117] "RemoveContainer" containerID="35a04b815f9ce5f2d6a9bad2ab927b4035d8dd46ff89584e84ccfbbcaa17ceda"
Jul 28 14:24:03.210732 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:03.210716    2554 scope.go:117] "RemoveContainer" containerID="51a83e3a53b78eab7d5b63b3fb8067c3aba7d239ff6f7e4d451227e22e0c0fca"
Jul 28 14:24:03.217771 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:03.217756    2554 scope.go:117] "RemoveContainer" containerID="443bc2b24d83311042a015801123d5349c4d8303f2d6ec88eccbaf27496b3d57"
Jul 28 14:24:03.218018 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:24:03.218000    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"443bc2b24d83311042a015801123d5349c4d8303f2d6ec88eccbaf27496b3d57\": container with ID starting with 443bc2b24d83311042a015801123d5349c4d8303f2d6ec88eccbaf27496b3d57 not found: ID does not exist" containerID="443bc2b24d83311042a015801123d5349c4d8303f2d6ec88eccbaf27496b3d57"
Jul 28 14:24:03.218088 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:03.218031    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"443bc2b24d83311042a015801123d5349c4d8303f2d6ec88eccbaf27496b3d57"} err="failed to get container status \"443bc2b24d83311042a015801123d5349c4d8303f2d6ec88eccbaf27496b3d57\": rpc error: code = NotFound desc = could not find container \"443bc2b24d83311042a015801123d5349c4d8303f2d6ec88eccbaf27496b3d57\": container with ID starting with 443bc2b24d83311042a015801123d5349c4d8303f2d6ec88eccbaf27496b3d57 not found: ID does not exist"
Jul 28 14:24:03.218088 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:03.218057    2554 scope.go:117] "RemoveContainer" containerID="35a04b815f9ce5f2d6a9bad2ab927b4035d8dd46ff89584e84ccfbbcaa17ceda"
Jul 28 14:24:03.218309 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:24:03.218290    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35a04b815f9ce5f2d6a9bad2ab927b4035d8dd46ff89584e84ccfbbcaa17ceda\": container with ID starting with 35a04b815f9ce5f2d6a9bad2ab927b4035d8dd46ff89584e84ccfbbcaa17ceda not found: ID does not exist" containerID="35a04b815f9ce5f2d6a9bad2ab927b4035d8dd46ff89584e84ccfbbcaa17ceda"
Jul 28 14:24:03.218358 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:03.218316    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35a04b815f9ce5f2d6a9bad2ab927b4035d8dd46ff89584e84ccfbbcaa17ceda"} err="failed to get container status \"35a04b815f9ce5f2d6a9bad2ab927b4035d8dd46ff89584e84ccfbbcaa17ceda\": rpc error: code = NotFound desc = could not find container \"35a04b815f9ce5f2d6a9bad2ab927b4035d8dd46ff89584e84ccfbbcaa17ceda\": container with ID starting with 35a04b815f9ce5f2d6a9bad2ab927b4035d8dd46ff89584e84ccfbbcaa17ceda not found: ID does not exist"
Jul 28 14:24:03.218358 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:03.218331    2554 scope.go:117] "RemoveContainer" containerID="51a83e3a53b78eab7d5b63b3fb8067c3aba7d239ff6f7e4d451227e22e0c0fca"
Jul 28 14:24:03.218559 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:24:03.218542    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51a83e3a53b78eab7d5b63b3fb8067c3aba7d239ff6f7e4d451227e22e0c0fca\": container with ID starting with 51a83e3a53b78eab7d5b63b3fb8067c3aba7d239ff6f7e4d451227e22e0c0fca not found: ID does not exist" containerID="51a83e3a53b78eab7d5b63b3fb8067c3aba7d239ff6f7e4d451227e22e0c0fca"
Jul 28 14:24:03.218613 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:03.218566    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51a83e3a53b78eab7d5b63b3fb8067c3aba7d239ff6f7e4d451227e22e0c0fca"} err="failed to get container status \"51a83e3a53b78eab7d5b63b3fb8067c3aba7d239ff6f7e4d451227e22e0c0fca\": rpc error: code = NotFound desc = could not find container \"51a83e3a53b78eab7d5b63b3fb8067c3aba7d239ff6f7e4d451227e22e0c0fca\": container with ID starting with 51a83e3a53b78eab7d5b63b3fb8067c3aba7d239ff6f7e4d451227e22e0c0fca not found: ID does not exist"
Jul 28 14:24:03.227842 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:03.227824    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b70a5d7a-a4e4-4dd0-9d64-41146c19a28e-proxy-tls\") pod \"b70a5d7a-a4e4-4dd0-9d64-41146c19a28e\" (UID: \"b70a5d7a-a4e4-4dd0-9d64-41146c19a28e\") "
Jul 28 14:24:03.227908 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:03.227886    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4hbb\" (UniqueName: \"kubernetes.io/projected/b70a5d7a-a4e4-4dd0-9d64-41146c19a28e-kube-api-access-f4hbb\") pod \"b70a5d7a-a4e4-4dd0-9d64-41146c19a28e\" (UID: \"b70a5d7a-a4e4-4dd0-9d64-41146c19a28e\") "
Jul 28 14:24:03.227945 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:03.227939    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b70a5d7a-a4e4-4dd0-9d64-41146c19a28e-cabundle-cert\") pod \"b70a5d7a-a4e4-4dd0-9d64-41146c19a28e\" (UID: \"b70a5d7a-a4e4-4dd0-9d64-41146c19a28e\") "
Jul 28 14:24:03.227993 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:03.227962    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b70a5d7a-a4e4-4dd0-9d64-41146c19a28e-kube-rbac-proxy-sar-config\") pod \"b70a5d7a-a4e4-4dd0-9d64-41146c19a28e\" (UID: \"b70a5d7a-a4e4-4dd0-9d64-41146c19a28e\") "
Jul 28 14:24:03.228050 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:03.228025    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b70a5d7a-a4e4-4dd0-9d64-41146c19a28e-kserve-provision-location\") pod \"b70a5d7a-a4e4-4dd0-9d64-41146c19a28e\" (UID: \"b70a5d7a-a4e4-4dd0-9d64-41146c19a28e\") "
Jul 28 14:24:03.228353 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:03.228327    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b70a5d7a-a4e4-4dd0-9d64-41146c19a28e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b70a5d7a-a4e4-4dd0-9d64-41146c19a28e" (UID: "b70a5d7a-a4e4-4dd0-9d64-41146c19a28e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:24:03.228683 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:03.228357    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b70a5d7a-a4e4-4dd0-9d64-41146c19a28e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b70a5d7a-a4e4-4dd0-9d64-41146c19a28e" (UID: "b70a5d7a-a4e4-4dd0-9d64-41146c19a28e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:24:03.228683 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:03.228415    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b70a5d7a-a4e4-4dd0-9d64-41146c19a28e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b70a5d7a-a4e4-4dd0-9d64-41146c19a28e" (UID: "b70a5d7a-a4e4-4dd0-9d64-41146c19a28e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:24:03.230014 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:03.229994    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b70a5d7a-a4e4-4dd0-9d64-41146c19a28e-kube-api-access-f4hbb" (OuterVolumeSpecName: "kube-api-access-f4hbb") pod "b70a5d7a-a4e4-4dd0-9d64-41146c19a28e" (UID: "b70a5d7a-a4e4-4dd0-9d64-41146c19a28e"). InnerVolumeSpecName "kube-api-access-f4hbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:24:03.230091 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:03.230069    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b70a5d7a-a4e4-4dd0-9d64-41146c19a28e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b70a5d7a-a4e4-4dd0-9d64-41146c19a28e" (UID: "b70a5d7a-a4e4-4dd0-9d64-41146c19a28e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:24:03.328920 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:03.328882    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b70a5d7a-a4e4-4dd0-9d64-41146c19a28e-proxy-tls\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:24:03.328920 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:03.328915    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4hbb\" (UniqueName: \"kubernetes.io/projected/b70a5d7a-a4e4-4dd0-9d64-41146c19a28e-kube-api-access-f4hbb\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:24:03.328920 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:03.328925    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b70a5d7a-a4e4-4dd0-9d64-41146c19a28e-cabundle-cert\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:24:03.329121 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:03.328934    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b70a5d7a-a4e4-4dd0-9d64-41146c19a28e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:24:03.329121 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:03.328944    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b70a5d7a-a4e4-4dd0-9d64-41146c19a28e-kserve-provision-location\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:24:03.523044 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:03.523015    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4"]
Jul 28 14:24:03.525318 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:03.525295    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-665d8df985-28mv4"]
Jul 28 14:24:03.802119 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:03.802046    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b70a5d7a-a4e4-4dd0-9d64-41146c19a28e" path="/var/lib/kubelet/pods/b70a5d7a-a4e4-4dd0-9d64-41146c19a28e/volumes"
Jul 28 14:24:07.213244 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:07.213213    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-pdngf_bad7a490-8b5a-4e9e-be80-61e3579fcf89/storage-initializer/0.log"
Jul 28 14:24:07.213640 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:07.213253    2554 generic.go:356] "Generic (PLEG): container finished" podID="bad7a490-8b5a-4e9e-be80-61e3579fcf89" containerID="ed8467d1608da61868354d4fb25d9689de72b9e7fc2b34bf85c06b70cbcf8488" exitCode=1
Jul 28 14:24:07.213640 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:07.213313    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-pdngf" event={"ID":"bad7a490-8b5a-4e9e-be80-61e3579fcf89","Type":"ContainerDied","Data":"ed8467d1608da61868354d4fb25d9689de72b9e7fc2b34bf85c06b70cbcf8488"}
Jul 28 14:24:08.220465 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:08.220433    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-pdngf_bad7a490-8b5a-4e9e-be80-61e3579fcf89/storage-initializer/0.log"
Jul 28 14:24:08.220945 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:08.220525    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-pdngf" event={"ID":"bad7a490-8b5a-4e9e-be80-61e3579fcf89","Type":"ContainerStarted","Data":"308f0b85c1ffa24533720a52cf1dad2b5a1d087e3e76c8ca6e0b83bbcc2c9fca"}
Jul 28 14:24:09.530755 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:09.530723    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-pdngf"]
Jul 28 14:24:09.531154 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:09.530942    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-pdngf" podUID="bad7a490-8b5a-4e9e-be80-61e3579fcf89" containerName="storage-initializer" containerID="cri-o://308f0b85c1ffa24533720a52cf1dad2b5a1d087e3e76c8ca6e0b83bbcc2c9fca" gracePeriod=30
Jul 28 14:24:10.670341 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:10.670291    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl"]
Jul 28 14:24:10.670737 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:10.670628    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b70a5d7a-a4e4-4dd0-9d64-41146c19a28e" containerName="kserve-container"
Jul 28 14:24:10.670737 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:10.670639    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="b70a5d7a-a4e4-4dd0-9d64-41146c19a28e" containerName="kserve-container"
Jul 28 14:24:10.670737 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:10.670648    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b70a5d7a-a4e4-4dd0-9d64-41146c19a28e" containerName="kube-rbac-proxy"
Jul 28 14:24:10.670737 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:10.670653    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="b70a5d7a-a4e4-4dd0-9d64-41146c19a28e" containerName="kube-rbac-proxy"
Jul 28 14:24:10.670737 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:10.670668    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b70a5d7a-a4e4-4dd0-9d64-41146c19a28e" containerName="storage-initializer"
Jul 28 14:24:10.670737 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:10.670673    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="b70a5d7a-a4e4-4dd0-9d64-41146c19a28e" containerName="storage-initializer"
Jul 28 14:24:10.670737 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:10.670729    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="b70a5d7a-a4e4-4dd0-9d64-41146c19a28e" containerName="kserve-container"
Jul 28 14:24:10.670737 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:10.670738    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="b70a5d7a-a4e4-4dd0-9d64-41146c19a28e" containerName="kube-rbac-proxy"
Jul 28 14:24:10.674048 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:10.674029    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl"
Jul 28 14:24:10.678258 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:10.678227    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 14:24:10.678373 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:10.678233    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
Jul 28 14:24:10.678691 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:10.678675    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
Jul 28 14:24:10.688411 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:10.688382    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl"]
Jul 28 14:24:10.794074 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:10.794036    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1b00cce-fd85-41f4-a0bb-d5e86a4057e6-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl\" (UID: \"a1b00cce-fd85-41f4-a0bb-d5e86a4057e6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl"
Jul 28 14:24:10.794225 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:10.794107    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgb88\" (UniqueName: \"kubernetes.io/projected/a1b00cce-fd85-41f4-a0bb-d5e86a4057e6-kube-api-access-lgb88\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl\" (UID: \"a1b00cce-fd85-41f4-a0bb-d5e86a4057e6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl"
Jul 28 14:24:10.794225 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:10.794150    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1b00cce-fd85-41f4-a0bb-d5e86a4057e6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl\" (UID: \"a1b00cce-fd85-41f4-a0bb-d5e86a4057e6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl"
Jul 28 14:24:10.794225 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:10.794210    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1b00cce-fd85-41f4-a0bb-d5e86a4057e6-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl\" (UID: \"a1b00cce-fd85-41f4-a0bb-d5e86a4057e6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl"
Jul 28 14:24:10.794361 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:10.794285    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1b00cce-fd85-41f4-a0bb-d5e86a4057e6-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl\" (UID: \"a1b00cce-fd85-41f4-a0bb-d5e86a4057e6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl"
Jul 28 14:24:10.894982 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:10.894947    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1b00cce-fd85-41f4-a0bb-d5e86a4057e6-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl\" (UID: \"a1b00cce-fd85-41f4-a0bb-d5e86a4057e6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl"
Jul 28 14:24:10.895169 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:10.895011    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1b00cce-fd85-41f4-a0bb-d5e86a4057e6-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl\" (UID: \"a1b00cce-fd85-41f4-a0bb-d5e86a4057e6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl"
Jul 28 14:24:10.895169 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:10.895064    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgb88\" (UniqueName: \"kubernetes.io/projected/a1b00cce-fd85-41f4-a0bb-d5e86a4057e6-kube-api-access-lgb88\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl\" (UID: \"a1b00cce-fd85-41f4-a0bb-d5e86a4057e6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl"
Jul 28 14:24:10.895169 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:10.895099    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1b00cce-fd85-41f4-a0bb-d5e86a4057e6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl\" (UID: \"a1b00cce-fd85-41f4-a0bb-d5e86a4057e6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl"
Jul 28 14:24:10.895169 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:24:10.895103    2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found
Jul 28 14:24:10.895169 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:10.895151    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1b00cce-fd85-41f4-a0bb-d5e86a4057e6-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl\" (UID: \"a1b00cce-fd85-41f4-a0bb-d5e86a4057e6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl"
Jul 28 14:24:10.895488 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:24:10.895184    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1b00cce-fd85-41f4-a0bb-d5e86a4057e6-proxy-tls podName:a1b00cce-fd85-41f4-a0bb-d5e86a4057e6 nodeName:}" failed. No retries permitted until 2026-07-28 14:24:11.39516412 +0000 UTC m=+5448.181039229 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a1b00cce-fd85-41f4-a0bb-d5e86a4057e6-proxy-tls") pod "isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl" (UID: "a1b00cce-fd85-41f4-a0bb-d5e86a4057e6") : secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found
Jul 28 14:24:10.895629 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:10.895609    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1b00cce-fd85-41f4-a0bb-d5e86a4057e6-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl\" (UID: \"a1b00cce-fd85-41f4-a0bb-d5e86a4057e6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl"
Jul 28 14:24:10.895878 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:10.895857    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1b00cce-fd85-41f4-a0bb-d5e86a4057e6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl\" (UID: \"a1b00cce-fd85-41f4-a0bb-d5e86a4057e6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl"
Jul 28 14:24:10.895954 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:10.895863    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1b00cce-fd85-41f4-a0bb-d5e86a4057e6-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl\" (UID: \"a1b00cce-fd85-41f4-a0bb-d5e86a4057e6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl"
Jul 28 14:24:10.904307 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:10.904259    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgb88\" (UniqueName: \"kubernetes.io/projected/a1b00cce-fd85-41f4-a0bb-d5e86a4057e6-kube-api-access-lgb88\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl\" (UID: \"a1b00cce-fd85-41f4-a0bb-d5e86a4057e6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl"
Jul 28 14:24:11.401573 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:11.401540    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1b00cce-fd85-41f4-a0bb-d5e86a4057e6-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl\" (UID: \"a1b00cce-fd85-41f4-a0bb-d5e86a4057e6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl"
Jul 28 14:24:11.404026 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:11.404005    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1b00cce-fd85-41f4-a0bb-d5e86a4057e6-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl\" (UID: \"a1b00cce-fd85-41f4-a0bb-d5e86a4057e6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl"
Jul 28 14:24:11.584945 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:11.584907    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl"
Jul 28 14:24:11.718678 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:11.718644    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl"]
Jul 28 14:24:11.721961 ip-10-0-129-177 kubenswrapper[2554]: W0728 14:24:11.721935    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1b00cce_fd85_41f4_a0bb_d5e86a4057e6.slice/crio-0f04e0c6da5f40fb7fe308612600f2d5101cd150262269f8c8fa5b9feb5427ba WatchSource:0}: Error finding container 0f04e0c6da5f40fb7fe308612600f2d5101cd150262269f8c8fa5b9feb5427ba: Status 404 returned error can't find the container with id 0f04e0c6da5f40fb7fe308612600f2d5101cd150262269f8c8fa5b9feb5427ba
Jul 28 14:24:12.240530 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:12.240495    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl" event={"ID":"a1b00cce-fd85-41f4-a0bb-d5e86a4057e6","Type":"ContainerStarted","Data":"edd289238208f3c573aa4e4fec4d3c29e6aa002ebb51be0acad94562fe5f1e70"}
Jul 28 14:24:12.240739 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:12.240562    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl" event={"ID":"a1b00cce-fd85-41f4-a0bb-d5e86a4057e6","Type":"ContainerStarted","Data":"0f04e0c6da5f40fb7fe308612600f2d5101cd150262269f8c8fa5b9feb5427ba"}
Jul 28 14:24:12.572894 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:12.572872    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-pdngf_bad7a490-8b5a-4e9e-be80-61e3579fcf89/storage-initializer/1.log"
Jul 28 14:24:12.573284 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:12.573257    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-pdngf_bad7a490-8b5a-4e9e-be80-61e3579fcf89/storage-initializer/0.log"
Jul 28 14:24:12.575599 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:12.575585    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-pdngf"
Jul 28 14:24:12.613356 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:12.613328    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bad7a490-8b5a-4e9e-be80-61e3579fcf89-kserve-provision-location\") pod \"bad7a490-8b5a-4e9e-be80-61e3579fcf89\" (UID: \"bad7a490-8b5a-4e9e-be80-61e3579fcf89\") "
Jul 28 14:24:12.613515 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:12.613373    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bad7a490-8b5a-4e9e-be80-61e3579fcf89-kube-rbac-proxy-sar-config\") pod \"bad7a490-8b5a-4e9e-be80-61e3579fcf89\" (UID: \"bad7a490-8b5a-4e9e-be80-61e3579fcf89\") "
Jul 28 14:24:12.613515 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:12.613403    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nx5wk\" (UniqueName: \"kubernetes.io/projected/bad7a490-8b5a-4e9e-be80-61e3579fcf89-kube-api-access-nx5wk\") pod \"bad7a490-8b5a-4e9e-be80-61e3579fcf89\" (UID: \"bad7a490-8b5a-4e9e-be80-61e3579fcf89\") "
Jul 28 14:24:12.613515 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:12.613439    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bad7a490-8b5a-4e9e-be80-61e3579fcf89-proxy-tls\") pod \"bad7a490-8b5a-4e9e-be80-61e3579fcf89\" (UID: \"bad7a490-8b5a-4e9e-be80-61e3579fcf89\") "
Jul 28 14:24:12.613689 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:12.613594    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bad7a490-8b5a-4e9e-be80-61e3579fcf89-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bad7a490-8b5a-4e9e-be80-61e3579fcf89" (UID: "bad7a490-8b5a-4e9e-be80-61e3579fcf89"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:24:12.613822 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:12.613797    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bad7a490-8b5a-4e9e-be80-61e3579fcf89-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bad7a490-8b5a-4e9e-be80-61e3579fcf89" (UID: "bad7a490-8b5a-4e9e-be80-61e3579fcf89"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:24:12.615704 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:12.615675    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bad7a490-8b5a-4e9e-be80-61e3579fcf89-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bad7a490-8b5a-4e9e-be80-61e3579fcf89" (UID: "bad7a490-8b5a-4e9e-be80-61e3579fcf89"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:24:12.615813 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:12.615747    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bad7a490-8b5a-4e9e-be80-61e3579fcf89-kube-api-access-nx5wk" (OuterVolumeSpecName: "kube-api-access-nx5wk") pod "bad7a490-8b5a-4e9e-be80-61e3579fcf89" (UID: "bad7a490-8b5a-4e9e-be80-61e3579fcf89"). InnerVolumeSpecName "kube-api-access-nx5wk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:24:12.713982 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:12.713941    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nx5wk\" (UniqueName: \"kubernetes.io/projected/bad7a490-8b5a-4e9e-be80-61e3579fcf89-kube-api-access-nx5wk\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:24:12.713982 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:12.713973    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bad7a490-8b5a-4e9e-be80-61e3579fcf89-proxy-tls\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:24:12.713982 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:12.713982    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bad7a490-8b5a-4e9e-be80-61e3579fcf89-kserve-provision-location\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:24:12.713982 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:12.713992    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bad7a490-8b5a-4e9e-be80-61e3579fcf89-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:24:13.245973 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:13.245944    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-pdngf_bad7a490-8b5a-4e9e-be80-61e3579fcf89/storage-initializer/1.log"
Jul 28 14:24:13.246410 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:13.246394    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-pdngf_bad7a490-8b5a-4e9e-be80-61e3579fcf89/storage-initializer/0.log"
Jul 28 14:24:13.246464 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:13.246425    2554 generic.go:356] "Generic (PLEG): container finished" podID="bad7a490-8b5a-4e9e-be80-61e3579fcf89" containerID="308f0b85c1ffa24533720a52cf1dad2b5a1d087e3e76c8ca6e0b83bbcc2c9fca" exitCode=1
Jul 28 14:24:13.246505 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:13.246477    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-pdngf"
Jul 28 14:24:13.246554 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:13.246509    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-pdngf" event={"ID":"bad7a490-8b5a-4e9e-be80-61e3579fcf89","Type":"ContainerDied","Data":"308f0b85c1ffa24533720a52cf1dad2b5a1d087e3e76c8ca6e0b83bbcc2c9fca"}
Jul 28 14:24:13.246612 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:13.246576    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-pdngf" event={"ID":"bad7a490-8b5a-4e9e-be80-61e3579fcf89","Type":"ContainerDied","Data":"a6cfb738d07c3e9cf1a45507079d7039fc21ef6d34de5c4e915ce05557f4dcb8"}
Jul 28 14:24:13.246669 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:13.246614    2554 scope.go:117] "RemoveContainer" containerID="308f0b85c1ffa24533720a52cf1dad2b5a1d087e3e76c8ca6e0b83bbcc2c9fca"
Jul 28 14:24:13.248291 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:13.248252    2554 generic.go:356] "Generic (PLEG): container finished" podID="a1b00cce-fd85-41f4-a0bb-d5e86a4057e6" containerID="edd289238208f3c573aa4e4fec4d3c29e6aa002ebb51be0acad94562fe5f1e70" exitCode=0
Jul 28 14:24:13.248419 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:13.248306    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl" event={"ID":"a1b00cce-fd85-41f4-a0bb-d5e86a4057e6","Type":"ContainerDied","Data":"edd289238208f3c573aa4e4fec4d3c29e6aa002ebb51be0acad94562fe5f1e70"}
Jul 28 14:24:13.255832 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:13.255810    2554 scope.go:117] "RemoveContainer" containerID="ed8467d1608da61868354d4fb25d9689de72b9e7fc2b34bf85c06b70cbcf8488"
Jul 28 14:24:13.263637 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:13.263615    2554 scope.go:117] "RemoveContainer" containerID="308f0b85c1ffa24533720a52cf1dad2b5a1d087e3e76c8ca6e0b83bbcc2c9fca"
Jul 28 14:24:13.263919 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:24:13.263894    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"308f0b85c1ffa24533720a52cf1dad2b5a1d087e3e76c8ca6e0b83bbcc2c9fca\": container with ID starting with 308f0b85c1ffa24533720a52cf1dad2b5a1d087e3e76c8ca6e0b83bbcc2c9fca not found: ID does not exist" containerID="308f0b85c1ffa24533720a52cf1dad2b5a1d087e3e76c8ca6e0b83bbcc2c9fca"
Jul 28 14:24:13.264006 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:13.263929    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"308f0b85c1ffa24533720a52cf1dad2b5a1d087e3e76c8ca6e0b83bbcc2c9fca"} err="failed to get container status \"308f0b85c1ffa24533720a52cf1dad2b5a1d087e3e76c8ca6e0b83bbcc2c9fca\": rpc error: code = NotFound desc = could not find container \"308f0b85c1ffa24533720a52cf1dad2b5a1d087e3e76c8ca6e0b83bbcc2c9fca\": container with ID starting with 308f0b85c1ffa24533720a52cf1dad2b5a1d087e3e76c8ca6e0b83bbcc2c9fca not found: ID does not exist"
Jul 28 14:24:13.264006 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:13.263952    2554 scope.go:117] "RemoveContainer" containerID="ed8467d1608da61868354d4fb25d9689de72b9e7fc2b34bf85c06b70cbcf8488"
Jul 28 14:24:13.264210 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:24:13.264190    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed8467d1608da61868354d4fb25d9689de72b9e7fc2b34bf85c06b70cbcf8488\": container with ID starting with ed8467d1608da61868354d4fb25d9689de72b9e7fc2b34bf85c06b70cbcf8488 not found: ID does not exist" containerID="ed8467d1608da61868354d4fb25d9689de72b9e7fc2b34bf85c06b70cbcf8488"
Jul 28 14:24:13.264256 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:13.264222    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed8467d1608da61868354d4fb25d9689de72b9e7fc2b34bf85c06b70cbcf8488"} err="failed to get container status \"ed8467d1608da61868354d4fb25d9689de72b9e7fc2b34bf85c06b70cbcf8488\": rpc error: code = NotFound desc = could not find container \"ed8467d1608da61868354d4fb25d9689de72b9e7fc2b34bf85c06b70cbcf8488\": container with ID starting with ed8467d1608da61868354d4fb25d9689de72b9e7fc2b34bf85c06b70cbcf8488 not found: ID does not exist"
Jul 28 14:24:13.299041 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:13.299012    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-pdngf"]
Jul 28 14:24:13.303665 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:13.303635    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7d56b5c8fc-pdngf"]
Jul 28 14:24:13.802343 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:13.802308    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bad7a490-8b5a-4e9e-be80-61e3579fcf89" path="/var/lib/kubelet/pods/bad7a490-8b5a-4e9e-be80-61e3579fcf89/volumes"
Jul 28 14:24:14.254552 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:14.254519    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl" event={"ID":"a1b00cce-fd85-41f4-a0bb-d5e86a4057e6","Type":"ContainerStarted","Data":"705d62614b86d9d928cef2cceccf248c1f833a8f6212e8ac268a0467b882c7dc"}
Jul 28 14:24:14.254987 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:14.254579    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl" event={"ID":"a1b00cce-fd85-41f4-a0bb-d5e86a4057e6","Type":"ContainerStarted","Data":"2e18ccf4940e76fb1794af04147d1e980334be4492b98c9f345bb3fa9b0977d7"}
Jul 28 14:24:14.254987 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:14.254646    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl"
Jul 28 14:24:14.278248 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:14.278204    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl" podStartSLOduration=3.303932793 podStartE2EDuration="4.278186766s" totalImagesPullingTime="0s" totalInitContainerRuntime="974.253973ms" isStatefulPod=false podCreationTimestamp="2026-07-28 14:24:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:24:14.27681674 +0000 UTC m=+5451.062691851" watchObservedRunningTime="2026-07-28 14:24:14.278186766 +0000 UTC m=+5451.064061877"
Jul 28 14:24:15.260231 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:15.260198    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl"
Jul 28 14:24:15.261590 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:15.261562    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl" podUID="a1b00cce-fd85-41f4-a0bb-d5e86a4057e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Jul 28 14:24:16.264778 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:16.264734    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl" podUID="a1b00cce-fd85-41f4-a0bb-d5e86a4057e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Jul 28 14:24:21.271158 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:21.271131    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl"
Jul 28 14:24:21.271639 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:21.271615    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl" podUID="a1b00cce-fd85-41f4-a0bb-d5e86a4057e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Jul 28 14:24:31.272387 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:31.272348    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl" podUID="a1b00cce-fd85-41f4-a0bb-d5e86a4057e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Jul 28 14:24:41.271817 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:41.271778    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl" podUID="a1b00cce-fd85-41f4-a0bb-d5e86a4057e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Jul 28 14:24:51.272436 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:24:51.272335    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl" podUID="a1b00cce-fd85-41f4-a0bb-d5e86a4057e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Jul 28 14:25:01.271525 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:01.271488    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl" podUID="a1b00cce-fd85-41f4-a0bb-d5e86a4057e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Jul 28 14:25:11.271995 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:11.271951    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl" podUID="a1b00cce-fd85-41f4-a0bb-d5e86a4057e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Jul 28 14:25:21.271905 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:21.271868    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl" podUID="a1b00cce-fd85-41f4-a0bb-d5e86a4057e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Jul 28 14:25:31.272913 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:31.272879    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl"
Jul 28 14:25:40.734157 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:40.734125    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl"]
Jul 28 14:25:40.734733 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:40.734435    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl" podUID="a1b00cce-fd85-41f4-a0bb-d5e86a4057e6" containerName="kserve-container" containerID="cri-o://2e18ccf4940e76fb1794af04147d1e980334be4492b98c9f345bb3fa9b0977d7" gracePeriod=30
Jul 28 14:25:40.734733 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:40.734495    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl" podUID="a1b00cce-fd85-41f4-a0bb-d5e86a4057e6" containerName="kube-rbac-proxy" containerID="cri-o://705d62614b86d9d928cef2cceccf248c1f833a8f6212e8ac268a0467b882c7dc" gracePeriod=30
Jul 28 14:25:41.264946 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:41.264908    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl" podUID="a1b00cce-fd85-41f4-a0bb-d5e86a4057e6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.66:8643/healthz\": dial tcp 10.134.0.66:8643: connect: connection refused"
Jul 28 14:25:41.272363 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:41.272340    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl" podUID="a1b00cce-fd85-41f4-a0bb-d5e86a4057e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Jul 28 14:25:41.612142 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:41.612055    2554 generic.go:356] "Generic (PLEG): container finished" podID="a1b00cce-fd85-41f4-a0bb-d5e86a4057e6" containerID="705d62614b86d9d928cef2cceccf248c1f833a8f6212e8ac268a0467b882c7dc" exitCode=2
Jul 28 14:25:41.612142 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:41.612099    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl" event={"ID":"a1b00cce-fd85-41f4-a0bb-d5e86a4057e6","Type":"ContainerDied","Data":"705d62614b86d9d928cef2cceccf248c1f833a8f6212e8ac268a0467b882c7dc"}
Jul 28 14:25:41.866180 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:41.866100    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-dvhfc"]
Jul 28 14:25:41.866625 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:41.866609    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bad7a490-8b5a-4e9e-be80-61e3579fcf89" containerName="storage-initializer"
Jul 28 14:25:41.866671 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:41.866628    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="bad7a490-8b5a-4e9e-be80-61e3579fcf89" containerName="storage-initializer"
Jul 28 14:25:41.866739 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:41.866729    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="bad7a490-8b5a-4e9e-be80-61e3579fcf89" containerName="storage-initializer"
Jul 28 14:25:41.866819 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:41.866809    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bad7a490-8b5a-4e9e-be80-61e3579fcf89" containerName="storage-initializer"
Jul 28 14:25:41.866859 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:41.866821    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="bad7a490-8b5a-4e9e-be80-61e3579fcf89" containerName="storage-initializer"
Jul 28 14:25:41.866915 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:41.866905    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="bad7a490-8b5a-4e9e-be80-61e3579fcf89" containerName="storage-initializer"
Jul 28 14:25:41.870313 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:41.870297    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-dvhfc"
Jul 28 14:25:41.872853 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:41.872822    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
Jul 28 14:25:41.872968 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:41.872934    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
Jul 28 14:25:41.880507 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:41.880488    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-dvhfc"]
Jul 28 14:25:41.950138 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:41.950112    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1231148-2413-4247-b787-0fc1bf914811-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-dvhfc\" (UID: \"f1231148-2413-4247-b787-0fc1bf914811\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-dvhfc"
Jul 28 14:25:41.950306 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:41.950162    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl684\" (UniqueName: \"kubernetes.io/projected/f1231148-2413-4247-b787-0fc1bf914811-kube-api-access-cl684\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-dvhfc\" (UID: \"f1231148-2413-4247-b787-0fc1bf914811\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-dvhfc"
Jul 28 14:25:41.950306 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:41.950192    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f1231148-2413-4247-b787-0fc1bf914811-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-dvhfc\" (UID: \"f1231148-2413-4247-b787-0fc1bf914811\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-dvhfc"
Jul 28 14:25:41.950306 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:41.950223    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1231148-2413-4247-b787-0fc1bf914811-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-dvhfc\" (UID: \"f1231148-2413-4247-b787-0fc1bf914811\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-dvhfc"
Jul 28 14:25:42.050771 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:42.050738    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl684\" (UniqueName: \"kubernetes.io/projected/f1231148-2413-4247-b787-0fc1bf914811-kube-api-access-cl684\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-dvhfc\" (UID: \"f1231148-2413-4247-b787-0fc1bf914811\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-dvhfc"
Jul 28 14:25:42.050771 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:42.050777    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f1231148-2413-4247-b787-0fc1bf914811-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-dvhfc\" (UID: \"f1231148-2413-4247-b787-0fc1bf914811\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-dvhfc"
Jul 28 14:25:42.050977 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:42.050820    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1231148-2413-4247-b787-0fc1bf914811-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-dvhfc\" (UID: \"f1231148-2413-4247-b787-0fc1bf914811\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-dvhfc"
Jul 28 14:25:42.050977 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:42.050882    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1231148-2413-4247-b787-0fc1bf914811-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-dvhfc\" (UID: \"f1231148-2413-4247-b787-0fc1bf914811\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-dvhfc"
Jul 28 14:25:42.051262 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:42.051226    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1231148-2413-4247-b787-0fc1bf914811-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-dvhfc\" (UID: \"f1231148-2413-4247-b787-0fc1bf914811\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-dvhfc"
Jul 28 14:25:42.051528 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:42.051507    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f1231148-2413-4247-b787-0fc1bf914811-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-dvhfc\" (UID: \"f1231148-2413-4247-b787-0fc1bf914811\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-dvhfc"
Jul 28 14:25:42.053422 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:42.053404    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1231148-2413-4247-b787-0fc1bf914811-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-dvhfc\" (UID: \"f1231148-2413-4247-b787-0fc1bf914811\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-dvhfc"
Jul 28 14:25:42.078795 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:42.078773    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl684\" (UniqueName: \"kubernetes.io/projected/f1231148-2413-4247-b787-0fc1bf914811-kube-api-access-cl684\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-dvhfc\" (UID: \"f1231148-2413-4247-b787-0fc1bf914811\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-dvhfc"
Jul 28 14:25:42.181136 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:42.181047    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-dvhfc"
Jul 28 14:25:42.312601 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:42.312577    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-dvhfc"]
Jul 28 14:25:42.315251 ip-10-0-129-177 kubenswrapper[2554]: W0728 14:25:42.315219    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1231148_2413_4247_b787_0fc1bf914811.slice/crio-4e10b3c2b30bb47ae15a569defe3466056232b03d48e86dc76094de8bbe2bc3b WatchSource:0}: Error finding container 4e10b3c2b30bb47ae15a569defe3466056232b03d48e86dc76094de8bbe2bc3b: Status 404 returned error can't find the container with id 4e10b3c2b30bb47ae15a569defe3466056232b03d48e86dc76094de8bbe2bc3b
Jul 28 14:25:42.618596 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:42.618555    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-dvhfc" event={"ID":"f1231148-2413-4247-b787-0fc1bf914811","Type":"ContainerStarted","Data":"f7e3f59bf1e8c4fe4b9dc925fd9608cc25cb7ce8b66bb7b469349e5d99e67e5a"}
Jul 28 14:25:42.618776 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:42.618629    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-dvhfc" event={"ID":"f1231148-2413-4247-b787-0fc1bf914811","Type":"ContainerStarted","Data":"4e10b3c2b30bb47ae15a569defe3466056232b03d48e86dc76094de8bbe2bc3b"}
Jul 28 14:25:45.386101 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:45.386073    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl"
Jul 28 14:25:45.478106 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:45.478073    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1b00cce-fd85-41f4-a0bb-d5e86a4057e6-kube-rbac-proxy-sar-config\") pod \"a1b00cce-fd85-41f4-a0bb-d5e86a4057e6\" (UID: \"a1b00cce-fd85-41f4-a0bb-d5e86a4057e6\") "
Jul 28 14:25:45.478296 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:45.478118    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1b00cce-fd85-41f4-a0bb-d5e86a4057e6-kserve-provision-location\") pod \"a1b00cce-fd85-41f4-a0bb-d5e86a4057e6\" (UID: \"a1b00cce-fd85-41f4-a0bb-d5e86a4057e6\") "
Jul 28 14:25:45.478296 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:45.478142    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgb88\" (UniqueName: \"kubernetes.io/projected/a1b00cce-fd85-41f4-a0bb-d5e86a4057e6-kube-api-access-lgb88\") pod \"a1b00cce-fd85-41f4-a0bb-d5e86a4057e6\" (UID: \"a1b00cce-fd85-41f4-a0bb-d5e86a4057e6\") "
Jul 28 14:25:45.478296 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:45.478199    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1b00cce-fd85-41f4-a0bb-d5e86a4057e6-cabundle-cert\") pod \"a1b00cce-fd85-41f4-a0bb-d5e86a4057e6\" (UID: \"a1b00cce-fd85-41f4-a0bb-d5e86a4057e6\") "
Jul 28 14:25:45.478296 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:45.478249    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1b00cce-fd85-41f4-a0bb-d5e86a4057e6-proxy-tls\") pod \"a1b00cce-fd85-41f4-a0bb-d5e86a4057e6\" (UID: \"a1b00cce-fd85-41f4-a0bb-d5e86a4057e6\") "
Jul 28 14:25:45.478590 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:45.478530    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1b00cce-fd85-41f4-a0bb-d5e86a4057e6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a1b00cce-fd85-41f4-a0bb-d5e86a4057e6" (UID: "a1b00cce-fd85-41f4-a0bb-d5e86a4057e6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:25:45.478590 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:45.478557    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1b00cce-fd85-41f4-a0bb-d5e86a4057e6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a1b00cce-fd85-41f4-a0bb-d5e86a4057e6" (UID: "a1b00cce-fd85-41f4-a0bb-d5e86a4057e6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:25:45.478674 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:45.478618    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1b00cce-fd85-41f4-a0bb-d5e86a4057e6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a1b00cce-fd85-41f4-a0bb-d5e86a4057e6" (UID: "a1b00cce-fd85-41f4-a0bb-d5e86a4057e6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:25:45.480520 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:45.480497    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1b00cce-fd85-41f4-a0bb-d5e86a4057e6-kube-api-access-lgb88" (OuterVolumeSpecName: "kube-api-access-lgb88") pod "a1b00cce-fd85-41f4-a0bb-d5e86a4057e6" (UID: "a1b00cce-fd85-41f4-a0bb-d5e86a4057e6"). InnerVolumeSpecName "kube-api-access-lgb88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:25:45.480608 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:45.480540    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1b00cce-fd85-41f4-a0bb-d5e86a4057e6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a1b00cce-fd85-41f4-a0bb-d5e86a4057e6" (UID: "a1b00cce-fd85-41f4-a0bb-d5e86a4057e6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:25:45.578811 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:45.578784    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1b00cce-fd85-41f4-a0bb-d5e86a4057e6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:25:45.578811 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:45.578810    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1b00cce-fd85-41f4-a0bb-d5e86a4057e6-kserve-provision-location\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:25:45.578982 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:45.578820    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgb88\" (UniqueName: \"kubernetes.io/projected/a1b00cce-fd85-41f4-a0bb-d5e86a4057e6-kube-api-access-lgb88\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:25:45.578982 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:45.578829    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1b00cce-fd85-41f4-a0bb-d5e86a4057e6-cabundle-cert\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:25:45.578982 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:45.578838    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1b00cce-fd85-41f4-a0bb-d5e86a4057e6-proxy-tls\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:25:45.633174 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:45.633147    2554 generic.go:356] "Generic (PLEG): container finished" podID="a1b00cce-fd85-41f4-a0bb-d5e86a4057e6" containerID="2e18ccf4940e76fb1794af04147d1e980334be4492b98c9f345bb3fa9b0977d7" exitCode=0
Jul 28 14:25:45.633352 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:45.633189    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl" event={"ID":"a1b00cce-fd85-41f4-a0bb-d5e86a4057e6","Type":"ContainerDied","Data":"2e18ccf4940e76fb1794af04147d1e980334be4492b98c9f345bb3fa9b0977d7"}
Jul 28 14:25:45.633352 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:45.633230    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl"
Jul 28 14:25:45.633352 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:45.633258    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl" event={"ID":"a1b00cce-fd85-41f4-a0bb-d5e86a4057e6","Type":"ContainerDied","Data":"0f04e0c6da5f40fb7fe308612600f2d5101cd150262269f8c8fa5b9feb5427ba"}
Jul 28 14:25:45.633352 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:45.633306    2554 scope.go:117] "RemoveContainer" containerID="705d62614b86d9d928cef2cceccf248c1f833a8f6212e8ac268a0467b882c7dc"
Jul 28 14:25:45.635062 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:45.635047    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-dvhfc_f1231148-2413-4247-b787-0fc1bf914811/storage-initializer/0.log"
Jul 28 14:25:45.635172 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:45.635074    2554 generic.go:356] "Generic (PLEG): container finished" podID="f1231148-2413-4247-b787-0fc1bf914811" containerID="f7e3f59bf1e8c4fe4b9dc925fd9608cc25cb7ce8b66bb7b469349e5d99e67e5a" exitCode=1
Jul 28 14:25:45.635172 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:45.635094    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-dvhfc" event={"ID":"f1231148-2413-4247-b787-0fc1bf914811","Type":"ContainerDied","Data":"f7e3f59bf1e8c4fe4b9dc925fd9608cc25cb7ce8b66bb7b469349e5d99e67e5a"}
Jul 28 14:25:45.642325 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:45.642303    2554 scope.go:117] "RemoveContainer" containerID="2e18ccf4940e76fb1794af04147d1e980334be4492b98c9f345bb3fa9b0977d7"
Jul 28 14:25:45.649763 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:45.649743    2554 scope.go:117] "RemoveContainer" containerID="edd289238208f3c573aa4e4fec4d3c29e6aa002ebb51be0acad94562fe5f1e70"
Jul 28 14:25:45.658235 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:45.658215    2554 scope.go:117] "RemoveContainer" containerID="705d62614b86d9d928cef2cceccf248c1f833a8f6212e8ac268a0467b882c7dc"
Jul 28 14:25:45.658582 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:25:45.658562    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"705d62614b86d9d928cef2cceccf248c1f833a8f6212e8ac268a0467b882c7dc\": container with ID starting with 705d62614b86d9d928cef2cceccf248c1f833a8f6212e8ac268a0467b882c7dc not found: ID does not exist" containerID="705d62614b86d9d928cef2cceccf248c1f833a8f6212e8ac268a0467b882c7dc"
Jul 28 14:25:45.658647 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:45.658592    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"705d62614b86d9d928cef2cceccf248c1f833a8f6212e8ac268a0467b882c7dc"} err="failed to get container status \"705d62614b86d9d928cef2cceccf248c1f833a8f6212e8ac268a0467b882c7dc\": rpc error: code = NotFound desc = could not find container \"705d62614b86d9d928cef2cceccf248c1f833a8f6212e8ac268a0467b882c7dc\": container with ID starting with 705d62614b86d9d928cef2cceccf248c1f833a8f6212e8ac268a0467b882c7dc not found: ID does not exist"
Jul 28 14:25:45.658647 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:45.658608    2554 scope.go:117] "RemoveContainer" containerID="2e18ccf4940e76fb1794af04147d1e980334be4492b98c9f345bb3fa9b0977d7"
Jul 28 14:25:45.658840 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:25:45.658822    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e18ccf4940e76fb1794af04147d1e980334be4492b98c9f345bb3fa9b0977d7\": container with ID starting with 2e18ccf4940e76fb1794af04147d1e980334be4492b98c9f345bb3fa9b0977d7 not found: ID does not exist" containerID="2e18ccf4940e76fb1794af04147d1e980334be4492b98c9f345bb3fa9b0977d7"
Jul 28 14:25:45.658915 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:45.658851    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e18ccf4940e76fb1794af04147d1e980334be4492b98c9f345bb3fa9b0977d7"} err="failed to get container status \"2e18ccf4940e76fb1794af04147d1e980334be4492b98c9f345bb3fa9b0977d7\": rpc error: code = NotFound desc = could not find container \"2e18ccf4940e76fb1794af04147d1e980334be4492b98c9f345bb3fa9b0977d7\": container with ID starting with 2e18ccf4940e76fb1794af04147d1e980334be4492b98c9f345bb3fa9b0977d7 not found: ID does not exist"
Jul 28 14:25:45.658915 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:45.658872    2554 scope.go:117] "RemoveContainer" containerID="edd289238208f3c573aa4e4fec4d3c29e6aa002ebb51be0acad94562fe5f1e70"
Jul 28 14:25:45.659107 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:25:45.659090    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edd289238208f3c573aa4e4fec4d3c29e6aa002ebb51be0acad94562fe5f1e70\": container with ID starting with edd289238208f3c573aa4e4fec4d3c29e6aa002ebb51be0acad94562fe5f1e70 not found: ID does not exist" containerID="edd289238208f3c573aa4e4fec4d3c29e6aa002ebb51be0acad94562fe5f1e70"
Jul 28 14:25:45.659152 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:45.659110    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edd289238208f3c573aa4e4fec4d3c29e6aa002ebb51be0acad94562fe5f1e70"} err="failed to get container status \"edd289238208f3c573aa4e4fec4d3c29e6aa002ebb51be0acad94562fe5f1e70\": rpc error: code = NotFound desc = could not find container \"edd289238208f3c573aa4e4fec4d3c29e6aa002ebb51be0acad94562fe5f1e70\": container with ID starting with edd289238208f3c573aa4e4fec4d3c29e6aa002ebb51be0acad94562fe5f1e70 not found: ID does not exist"
Jul 28 14:25:45.670155 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:45.670131    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl"]
Jul 28 14:25:45.675561 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:45.675541    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-mhqgl"]
Jul 28 14:25:45.801804 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:45.801729    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1b00cce-fd85-41f4-a0bb-d5e86a4057e6" path="/var/lib/kubelet/pods/a1b00cce-fd85-41f4-a0bb-d5e86a4057e6/volumes"
Jul 28 14:25:46.642644 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:46.642617    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-dvhfc_f1231148-2413-4247-b787-0fc1bf914811/storage-initializer/0.log"
Jul 28 14:25:46.643105 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:46.642691    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-dvhfc" event={"ID":"f1231148-2413-4247-b787-0fc1bf914811","Type":"ContainerStarted","Data":"58fd5bb63a65374e626bd8cc2c714f5346429be141ec897f358db903c49a09c1"}
Jul 28 14:25:49.656458 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:49.656428    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-dvhfc_f1231148-2413-4247-b787-0fc1bf914811/storage-initializer/1.log"
Jul 28 14:25:49.656903 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:49.656822    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-dvhfc_f1231148-2413-4247-b787-0fc1bf914811/storage-initializer/0.log"
Jul 28 14:25:49.656903 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:49.656853    2554 generic.go:356] "Generic (PLEG): container finished" podID="f1231148-2413-4247-b787-0fc1bf914811" containerID="58fd5bb63a65374e626bd8cc2c714f5346429be141ec897f358db903c49a09c1" exitCode=1
Jul 28 14:25:49.656982 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:49.656909    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-dvhfc" event={"ID":"f1231148-2413-4247-b787-0fc1bf914811","Type":"ContainerDied","Data":"58fd5bb63a65374e626bd8cc2c714f5346429be141ec897f358db903c49a09c1"}
Jul 28 14:25:49.657018 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:49.656990    2554 scope.go:117] "RemoveContainer" containerID="f7e3f59bf1e8c4fe4b9dc925fd9608cc25cb7ce8b66bb7b469349e5d99e67e5a"
Jul 28 14:25:49.657348 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:49.657326    2554 scope.go:117] "RemoveContainer" containerID="f7e3f59bf1e8c4fe4b9dc925fd9608cc25cb7ce8b66bb7b469349e5d99e67e5a"
Jul 28 14:25:49.667915 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:25:49.667893    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-dvhfc_kserve-ci-e2e-test(f1231148-2413-4247-b787-0fc1bf914811)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-dvhfc" podUID="f1231148-2413-4247-b787-0fc1bf914811"
Jul 28 14:25:49.668049 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:25:49.668018    2554 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-dvhfc_kserve-ci-e2e-test_f1231148-2413-4247-b787-0fc1bf914811_0 in pod sandbox 4e10b3c2b30bb47ae15a569defe3466056232b03d48e86dc76094de8bbe2bc3b from index: no such id: 'f7e3f59bf1e8c4fe4b9dc925fd9608cc25cb7ce8b66bb7b469349e5d99e67e5a'" containerID="f7e3f59bf1e8c4fe4b9dc925fd9608cc25cb7ce8b66bb7b469349e5d99e67e5a"
Jul 28 14:25:49.668097 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:49.668059    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7e3f59bf1e8c4fe4b9dc925fd9608cc25cb7ce8b66bb7b469349e5d99e67e5a"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-dvhfc_kserve-ci-e2e-test_f1231148-2413-4247-b787-0fc1bf914811_0 in pod sandbox 4e10b3c2b30bb47ae15a569defe3466056232b03d48e86dc76094de8bbe2bc3b from index: no such id: 'f7e3f59bf1e8c4fe4b9dc925fd9608cc25cb7ce8b66bb7b469349e5d99e67e5a'"
Jul 28 14:25:50.662986 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:50.662954    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-dvhfc_f1231148-2413-4247-b787-0fc1bf914811/storage-initializer/1.log"
Jul 28 14:25:51.792159 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:51.792129    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-dvhfc"]
Jul 28 14:25:51.924720 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:51.924697    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-dvhfc_f1231148-2413-4247-b787-0fc1bf914811/storage-initializer/1.log"
Jul 28 14:25:51.927003 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:51.926982    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-dvhfc"
Jul 28 14:25:52.035560 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:52.035531    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1231148-2413-4247-b787-0fc1bf914811-proxy-tls\") pod \"f1231148-2413-4247-b787-0fc1bf914811\" (UID: \"f1231148-2413-4247-b787-0fc1bf914811\") "
Jul 28 14:25:52.035741 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:52.035596    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cl684\" (UniqueName: \"kubernetes.io/projected/f1231148-2413-4247-b787-0fc1bf914811-kube-api-access-cl684\") pod \"f1231148-2413-4247-b787-0fc1bf914811\" (UID: \"f1231148-2413-4247-b787-0fc1bf914811\") "
Jul 28 14:25:52.035741 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:52.035624    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1231148-2413-4247-b787-0fc1bf914811-kserve-provision-location\") pod \"f1231148-2413-4247-b787-0fc1bf914811\" (UID: \"f1231148-2413-4247-b787-0fc1bf914811\") "
Jul 28 14:25:52.035741 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:52.035685    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f1231148-2413-4247-b787-0fc1bf914811-kube-rbac-proxy-sar-config\") pod \"f1231148-2413-4247-b787-0fc1bf914811\" (UID: \"f1231148-2413-4247-b787-0fc1bf914811\") "
Jul 28 14:25:52.035929 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:52.035864    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1231148-2413-4247-b787-0fc1bf914811-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f1231148-2413-4247-b787-0fc1bf914811" (UID: "f1231148-2413-4247-b787-0fc1bf914811"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:25:52.036022 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:52.036000    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1231148-2413-4247-b787-0fc1bf914811-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f1231148-2413-4247-b787-0fc1bf914811" (UID: "f1231148-2413-4247-b787-0fc1bf914811"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:25:52.037838 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:52.037815    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1231148-2413-4247-b787-0fc1bf914811-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f1231148-2413-4247-b787-0fc1bf914811" (UID: "f1231148-2413-4247-b787-0fc1bf914811"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:25:52.037937 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:52.037882    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1231148-2413-4247-b787-0fc1bf914811-kube-api-access-cl684" (OuterVolumeSpecName: "kube-api-access-cl684") pod "f1231148-2413-4247-b787-0fc1bf914811" (UID: "f1231148-2413-4247-b787-0fc1bf914811"). InnerVolumeSpecName "kube-api-access-cl684". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:25:52.136440 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:52.136393    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f1231148-2413-4247-b787-0fc1bf914811-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:25:52.136440 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:52.136437    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1231148-2413-4247-b787-0fc1bf914811-proxy-tls\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:25:52.136440 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:52.136449    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl684\" (UniqueName: \"kubernetes.io/projected/f1231148-2413-4247-b787-0fc1bf914811-kube-api-access-cl684\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:25:52.136440 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:52.136457    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1231148-2413-4247-b787-0fc1bf914811-kserve-provision-location\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:25:52.672554 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:52.672524    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-dvhfc_f1231148-2413-4247-b787-0fc1bf914811/storage-initializer/1.log"
Jul 28 14:25:52.672731 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:52.672615    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-dvhfc" event={"ID":"f1231148-2413-4247-b787-0fc1bf914811","Type":"ContainerDied","Data":"4e10b3c2b30bb47ae15a569defe3466056232b03d48e86dc76094de8bbe2bc3b"}
Jul 28 14:25:52.672731 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:52.672626    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-dvhfc"
Jul 28 14:25:52.672731 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:52.672682    2554 scope.go:117] "RemoveContainer" containerID="58fd5bb63a65374e626bd8cc2c714f5346429be141ec897f358db903c49a09c1"
Jul 28 14:25:52.713403 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:52.713375    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-dvhfc"]
Jul 28 14:25:52.717947 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:52.717922    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-749df86c69-dvhfc"]
Jul 28 14:25:52.942770 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:52.942736    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745"]
Jul 28 14:25:52.943243 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:52.943222    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1231148-2413-4247-b787-0fc1bf914811" containerName="storage-initializer"
Jul 28 14:25:52.943352 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:52.943247    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1231148-2413-4247-b787-0fc1bf914811" containerName="storage-initializer"
Jul 28 14:25:52.943352 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:52.943261    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1b00cce-fd85-41f4-a0bb-d5e86a4057e6" containerName="storage-initializer"
Jul 28 14:25:52.943352 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:52.943287    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1b00cce-fd85-41f4-a0bb-d5e86a4057e6" containerName="storage-initializer"
Jul 28 14:25:52.943352 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:52.943301    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1231148-2413-4247-b787-0fc1bf914811" containerName="storage-initializer"
Jul 28 14:25:52.943352 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:52.943309    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1231148-2413-4247-b787-0fc1bf914811" containerName="storage-initializer"
Jul 28 14:25:52.943352 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:52.943323    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1b00cce-fd85-41f4-a0bb-d5e86a4057e6" containerName="kube-rbac-proxy"
Jul 28 14:25:52.943352 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:52.943331    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1b00cce-fd85-41f4-a0bb-d5e86a4057e6" containerName="kube-rbac-proxy"
Jul 28 14:25:52.943352 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:52.943345    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1b00cce-fd85-41f4-a0bb-d5e86a4057e6" containerName="kserve-container"
Jul 28 14:25:52.943352 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:52.943355    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1b00cce-fd85-41f4-a0bb-d5e86a4057e6" containerName="kserve-container"
Jul 28 14:25:52.943840 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:52.943436    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1231148-2413-4247-b787-0fc1bf914811" containerName="storage-initializer"
Jul 28 14:25:52.943840 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:52.943451    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1231148-2413-4247-b787-0fc1bf914811" containerName="storage-initializer"
Jul 28 14:25:52.943840 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:52.943460    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1b00cce-fd85-41f4-a0bb-d5e86a4057e6" containerName="kube-rbac-proxy"
Jul 28 14:25:52.943840 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:52.943470    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1b00cce-fd85-41f4-a0bb-d5e86a4057e6" containerName="kserve-container"
Jul 28 14:25:52.948037 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:52.948013    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745"
Jul 28 14:25:52.951037 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:52.950765    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 14:25:52.951037 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:52.950771    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rxfwd\""
Jul 28 14:25:52.951037 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:52.950770    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Jul 28 14:25:52.951037 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:52.950775    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 14:25:52.951512 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:52.951485    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Jul 28 14:25:52.951705 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:52.951687    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 14:25:52.951863 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:52.951741    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Jul 28 14:25:52.957010 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:52.956988    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745"]
Jul 28 14:25:53.042547 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:53.042518    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17282308-47d0-4284-8e8d-29872d332940-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745\" (UID: \"17282308-47d0-4284-8e8d-29872d332940\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745"
Jul 28 14:25:53.042547 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:53.042552    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17282308-47d0-4284-8e8d-29872d332940-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745\" (UID: \"17282308-47d0-4284-8e8d-29872d332940\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745"
Jul 28 14:25:53.042806 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:53.042582    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7clvj\" (UniqueName: \"kubernetes.io/projected/17282308-47d0-4284-8e8d-29872d332940-kube-api-access-7clvj\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745\" (UID: \"17282308-47d0-4284-8e8d-29872d332940\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745"
Jul 28 14:25:53.042806 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:53.042679    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17282308-47d0-4284-8e8d-29872d332940-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745\" (UID: \"17282308-47d0-4284-8e8d-29872d332940\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745"
Jul 28 14:25:53.042806 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:53.042727    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17282308-47d0-4284-8e8d-29872d332940-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745\" (UID: \"17282308-47d0-4284-8e8d-29872d332940\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745"
Jul 28 14:25:53.144159 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:53.144128    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17282308-47d0-4284-8e8d-29872d332940-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745\" (UID: \"17282308-47d0-4284-8e8d-29872d332940\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745"
Jul 28 14:25:53.144376 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:53.144168    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17282308-47d0-4284-8e8d-29872d332940-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745\" (UID: \"17282308-47d0-4284-8e8d-29872d332940\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745"
Jul 28 14:25:53.144437 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:53.144396    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17282308-47d0-4284-8e8d-29872d332940-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745\" (UID: \"17282308-47d0-4284-8e8d-29872d332940\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745"
Jul 28 14:25:53.144494 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:53.144432    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17282308-47d0-4284-8e8d-29872d332940-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745\" (UID: \"17282308-47d0-4284-8e8d-29872d332940\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745"
Jul 28 14:25:53.144547 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:53.144491    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7clvj\" (UniqueName: \"kubernetes.io/projected/17282308-47d0-4284-8e8d-29872d332940-kube-api-access-7clvj\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745\" (UID: \"17282308-47d0-4284-8e8d-29872d332940\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745"
Jul 28 14:25:53.144600 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:25:53.144554    2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
Jul 28 14:25:53.144658 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:25:53.144634    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17282308-47d0-4284-8e8d-29872d332940-proxy-tls podName:17282308-47d0-4284-8e8d-29872d332940 nodeName:}" failed. No retries permitted until 2026-07-28 14:25:53.644611414 +0000 UTC m=+5550.430486505 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/17282308-47d0-4284-8e8d-29872d332940-proxy-tls") pod "isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745" (UID: "17282308-47d0-4284-8e8d-29872d332940") : secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
Jul 28 14:25:53.144866 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:53.144843    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17282308-47d0-4284-8e8d-29872d332940-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745\" (UID: \"17282308-47d0-4284-8e8d-29872d332940\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745"
Jul 28 14:25:53.144975 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:53.144954    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17282308-47d0-4284-8e8d-29872d332940-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745\" (UID: \"17282308-47d0-4284-8e8d-29872d332940\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745"
Jul 28 14:25:53.145013 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:53.144963    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17282308-47d0-4284-8e8d-29872d332940-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745\" (UID: \"17282308-47d0-4284-8e8d-29872d332940\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745"
Jul 28 14:25:53.153564 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:53.153541    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7clvj\" (UniqueName: \"kubernetes.io/projected/17282308-47d0-4284-8e8d-29872d332940-kube-api-access-7clvj\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745\" (UID: \"17282308-47d0-4284-8e8d-29872d332940\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745"
Jul 28 14:25:53.650347 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:53.650313    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17282308-47d0-4284-8e8d-29872d332940-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745\" (UID: \"17282308-47d0-4284-8e8d-29872d332940\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745"
Jul 28 14:25:53.652901 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:53.652870    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17282308-47d0-4284-8e8d-29872d332940-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745\" (UID: \"17282308-47d0-4284-8e8d-29872d332940\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745"
Jul 28 14:25:53.802517 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:53.802484    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1231148-2413-4247-b787-0fc1bf914811" path="/var/lib/kubelet/pods/f1231148-2413-4247-b787-0fc1bf914811/volumes"
Jul 28 14:25:53.860238 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:53.860194    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745"
Jul 28 14:25:53.991862 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:53.991839    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745"]
Jul 28 14:25:53.994664 ip-10-0-129-177 kubenswrapper[2554]: W0728 14:25:53.994633    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17282308_47d0_4284_8e8d_29872d332940.slice/crio-33a559bc527b6cd71da1bc622c8309bb92d3046d7d12483273854f8424e208ec WatchSource:0}: Error finding container 33a559bc527b6cd71da1bc622c8309bb92d3046d7d12483273854f8424e208ec: Status 404 returned error can't find the container with id 33a559bc527b6cd71da1bc622c8309bb92d3046d7d12483273854f8424e208ec
Jul 28 14:25:54.683383 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:54.683347    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745" event={"ID":"17282308-47d0-4284-8e8d-29872d332940","Type":"ContainerStarted","Data":"188b0647a86667e40600d08c4e91e9c98ab85f00d5a4b67b7b81341e053c6663"}
Jul 28 14:25:54.683565 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:54.683437    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745" event={"ID":"17282308-47d0-4284-8e8d-29872d332940","Type":"ContainerStarted","Data":"33a559bc527b6cd71da1bc622c8309bb92d3046d7d12483273854f8424e208ec"}
Jul 28 14:25:55.689736 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:55.689698    2554 generic.go:356] "Generic (PLEG): container finished" podID="17282308-47d0-4284-8e8d-29872d332940" containerID="188b0647a86667e40600d08c4e91e9c98ab85f00d5a4b67b7b81341e053c6663" exitCode=0
Jul 28 14:25:55.690198 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:55.689788    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745" event={"ID":"17282308-47d0-4284-8e8d-29872d332940","Type":"ContainerDied","Data":"188b0647a86667e40600d08c4e91e9c98ab85f00d5a4b67b7b81341e053c6663"}
Jul 28 14:25:56.697114 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:56.697078    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745" event={"ID":"17282308-47d0-4284-8e8d-29872d332940","Type":"ContainerStarted","Data":"43e57e18d13fbe2446f298bcc1c800f53fc0235e701d210ba589bcb46b17ea94"}
Jul 28 14:25:56.697589 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:56.697136    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745" event={"ID":"17282308-47d0-4284-8e8d-29872d332940","Type":"ContainerStarted","Data":"303be7bf1c667209e0283f4d4afb5d447bfd842f8c33430e8b9a5a32a9490bd6"}
Jul 28 14:25:56.697589 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:56.697376    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745"
Jul 28 14:25:56.697589 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:56.697429    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745"
Jul 28 14:25:56.698728 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:56.698703    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745" podUID="17282308-47d0-4284-8e8d-29872d332940" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Jul 28 14:25:56.717847 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:56.717807    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745" podStartSLOduration=3.806395385 podStartE2EDuration="4.717794303s" totalImagesPullingTime="0s" totalInitContainerRuntime="911.398918ms" isStatefulPod=false podCreationTimestamp="2026-07-28 14:25:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:25:56.716903086 +0000 UTC m=+5553.502778193" watchObservedRunningTime="2026-07-28 14:25:56.717794303 +0000 UTC m=+5553.503669821"
Jul 28 14:25:57.701584 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:25:57.701541    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745" podUID="17282308-47d0-4284-8e8d-29872d332940" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Jul 28 14:26:02.706323 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:26:02.706290    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745"
Jul 28 14:26:02.706798 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:26:02.706773    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745" podUID="17282308-47d0-4284-8e8d-29872d332940" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Jul 28 14:26:12.707447 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:26:12.707407    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745" podUID="17282308-47d0-4284-8e8d-29872d332940" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Jul 28 14:26:22.707411 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:26:22.707332    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745" podUID="17282308-47d0-4284-8e8d-29872d332940" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Jul 28 14:26:32.707078 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:26:32.707040    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745" podUID="17282308-47d0-4284-8e8d-29872d332940" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Jul 28 14:26:42.706849 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:26:42.706807    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745" podUID="17282308-47d0-4284-8e8d-29872d332940" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Jul 28 14:26:52.706857 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:26:52.706817    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745" podUID="17282308-47d0-4284-8e8d-29872d332940" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Jul 28 14:27:02.707394 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:02.707352    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745" podUID="17282308-47d0-4284-8e8d-29872d332940" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Jul 28 14:27:06.797372 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:06.797337    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745"
Jul 28 14:27:12.978913 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:12.978878    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745"]
Jul 28 14:27:12.979423 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:12.979117    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745" podUID="17282308-47d0-4284-8e8d-29872d332940" containerName="kserve-container" containerID="cri-o://303be7bf1c667209e0283f4d4afb5d447bfd842f8c33430e8b9a5a32a9490bd6" gracePeriod=30
Jul 28 14:27:12.979423 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:12.979174    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745" podUID="17282308-47d0-4284-8e8d-29872d332940" containerName="kube-rbac-proxy" containerID="cri-o://43e57e18d13fbe2446f298bcc1c800f53fc0235e701d210ba589bcb46b17ea94" gracePeriod=30
Jul 28 14:27:14.014363 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:14.014327    2554 generic.go:356] "Generic (PLEG): container finished" podID="17282308-47d0-4284-8e8d-29872d332940" containerID="43e57e18d13fbe2446f298bcc1c800f53fc0235e701d210ba589bcb46b17ea94" exitCode=2
Jul 28 14:27:14.014828 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:14.014391    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745" event={"ID":"17282308-47d0-4284-8e8d-29872d332940","Type":"ContainerDied","Data":"43e57e18d13fbe2446f298bcc1c800f53fc0235e701d210ba589bcb46b17ea94"}
Jul 28 14:27:14.135724 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:14.135686    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-8j5xk"]
Jul 28 14:27:14.139858 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:14.139840    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-8j5xk"
Jul 28 14:27:14.142496 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:14.142474    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
Jul 28 14:27:14.142615 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:14.142478    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
Jul 28 14:27:14.150421 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:14.150401    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-8j5xk"]
Jul 28 14:27:14.220223 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:14.220187    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0816f9f-634a-4a39-92cf-5966ea9ca9eb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-8j5xk\" (UID: \"b0816f9f-634a-4a39-92cf-5966ea9ca9eb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-8j5xk"
Jul 28 14:27:14.220383 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:14.220237    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0816f9f-634a-4a39-92cf-5966ea9ca9eb-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-8j5xk\" (UID: \"b0816f9f-634a-4a39-92cf-5966ea9ca9eb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-8j5xk"
Jul 28 14:27:14.220383 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:14.220328    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5rqw\" (UniqueName: \"kubernetes.io/projected/b0816f9f-634a-4a39-92cf-5966ea9ca9eb-kube-api-access-r5rqw\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-8j5xk\" (UID: \"b0816f9f-634a-4a39-92cf-5966ea9ca9eb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-8j5xk"
Jul 28 14:27:14.220477 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:14.220379    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0816f9f-634a-4a39-92cf-5966ea9ca9eb-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-8j5xk\" (UID: \"b0816f9f-634a-4a39-92cf-5966ea9ca9eb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-8j5xk"
Jul 28 14:27:14.320797 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:14.320710    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0816f9f-634a-4a39-92cf-5966ea9ca9eb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-8j5xk\" (UID: \"b0816f9f-634a-4a39-92cf-5966ea9ca9eb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-8j5xk"
Jul 28 14:27:14.320797 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:14.320750    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0816f9f-634a-4a39-92cf-5966ea9ca9eb-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-8j5xk\" (UID: \"b0816f9f-634a-4a39-92cf-5966ea9ca9eb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-8j5xk"
Jul 28 14:27:14.320797 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:14.320788    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5rqw\" (UniqueName: \"kubernetes.io/projected/b0816f9f-634a-4a39-92cf-5966ea9ca9eb-kube-api-access-r5rqw\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-8j5xk\" (UID: \"b0816f9f-634a-4a39-92cf-5966ea9ca9eb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-8j5xk"
Jul 28 14:27:14.321107 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:14.320825    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0816f9f-634a-4a39-92cf-5966ea9ca9eb-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-8j5xk\" (UID: \"b0816f9f-634a-4a39-92cf-5966ea9ca9eb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-8j5xk"
Jul 28 14:27:14.321176 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:14.321155    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0816f9f-634a-4a39-92cf-5966ea9ca9eb-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-8j5xk\" (UID: \"b0816f9f-634a-4a39-92cf-5966ea9ca9eb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-8j5xk"
Jul 28 14:27:14.321502 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:14.321476    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0816f9f-634a-4a39-92cf-5966ea9ca9eb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-8j5xk\" (UID: \"b0816f9f-634a-4a39-92cf-5966ea9ca9eb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-8j5xk"
Jul 28 14:27:14.323370 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:14.323353    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0816f9f-634a-4a39-92cf-5966ea9ca9eb-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-8j5xk\" (UID: \"b0816f9f-634a-4a39-92cf-5966ea9ca9eb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-8j5xk"
Jul 28 14:27:14.330287 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:14.330247    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5rqw\" (UniqueName: \"kubernetes.io/projected/b0816f9f-634a-4a39-92cf-5966ea9ca9eb-kube-api-access-r5rqw\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-8j5xk\" (UID: \"b0816f9f-634a-4a39-92cf-5966ea9ca9eb\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-8j5xk"
Jul 28 14:27:14.451897 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:14.451852    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-8j5xk"
Jul 28 14:27:14.587659 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:14.587625    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-8j5xk"]
Jul 28 14:27:14.590183 ip-10-0-129-177 kubenswrapper[2554]: W0728 14:27:14.590156    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0816f9f_634a_4a39_92cf_5966ea9ca9eb.slice/crio-9c7cb7c38e1433abfc7363717f1a5d44088ae2b1884eb835e05bc212ca05549e WatchSource:0}: Error finding container 9c7cb7c38e1433abfc7363717f1a5d44088ae2b1884eb835e05bc212ca05549e: Status 404 returned error can't find the container with id 9c7cb7c38e1433abfc7363717f1a5d44088ae2b1884eb835e05bc212ca05549e
Jul 28 14:27:15.020334 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:15.020297    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-8j5xk" event={"ID":"b0816f9f-634a-4a39-92cf-5966ea9ca9eb","Type":"ContainerStarted","Data":"bb2bac443d3bdcf0f12b54d6778189fa03e5935c4a20556ce697c094d8179122"}
Jul 28 14:27:15.020711 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:15.020388    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-8j5xk" event={"ID":"b0816f9f-634a-4a39-92cf-5966ea9ca9eb","Type":"ContainerStarted","Data":"9c7cb7c38e1433abfc7363717f1a5d44088ae2b1884eb835e05bc212ca05549e"}
Jul 28 14:27:16.797004 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:16.796958    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745" podUID="17282308-47d0-4284-8e8d-29872d332940" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Jul 28 14:27:17.723964 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:17.723940    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745"
Jul 28 14:27:17.751577 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:17.751506    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17282308-47d0-4284-8e8d-29872d332940-kube-rbac-proxy-sar-config\") pod \"17282308-47d0-4284-8e8d-29872d332940\" (UID: \"17282308-47d0-4284-8e8d-29872d332940\") "
Jul 28 14:27:17.751577 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:17.751545    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17282308-47d0-4284-8e8d-29872d332940-cabundle-cert\") pod \"17282308-47d0-4284-8e8d-29872d332940\" (UID: \"17282308-47d0-4284-8e8d-29872d332940\") "
Jul 28 14:27:17.751793 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:17.751596    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17282308-47d0-4284-8e8d-29872d332940-proxy-tls\") pod \"17282308-47d0-4284-8e8d-29872d332940\" (UID: \"17282308-47d0-4284-8e8d-29872d332940\") "
Jul 28 14:27:17.751793 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:17.751627    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7clvj\" (UniqueName: \"kubernetes.io/projected/17282308-47d0-4284-8e8d-29872d332940-kube-api-access-7clvj\") pod \"17282308-47d0-4284-8e8d-29872d332940\" (UID: \"17282308-47d0-4284-8e8d-29872d332940\") "
Jul 28 14:27:17.751793 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:17.751666    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17282308-47d0-4284-8e8d-29872d332940-kserve-provision-location\") pod \"17282308-47d0-4284-8e8d-29872d332940\" (UID: \"17282308-47d0-4284-8e8d-29872d332940\") "
Jul 28 14:27:17.752063 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:17.751902    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17282308-47d0-4284-8e8d-29872d332940-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "17282308-47d0-4284-8e8d-29872d332940" (UID: "17282308-47d0-4284-8e8d-29872d332940"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:27:17.752063 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:17.751983    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17282308-47d0-4284-8e8d-29872d332940-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "17282308-47d0-4284-8e8d-29872d332940" (UID: "17282308-47d0-4284-8e8d-29872d332940"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:27:17.752236 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:17.752215    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17282308-47d0-4284-8e8d-29872d332940-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "17282308-47d0-4284-8e8d-29872d332940" (UID: "17282308-47d0-4284-8e8d-29872d332940"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:27:17.753738 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:17.753716    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17282308-47d0-4284-8e8d-29872d332940-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "17282308-47d0-4284-8e8d-29872d332940" (UID: "17282308-47d0-4284-8e8d-29872d332940"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:27:17.753946 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:17.753923    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17282308-47d0-4284-8e8d-29872d332940-kube-api-access-7clvj" (OuterVolumeSpecName: "kube-api-access-7clvj") pod "17282308-47d0-4284-8e8d-29872d332940" (UID: "17282308-47d0-4284-8e8d-29872d332940"). InnerVolumeSpecName "kube-api-access-7clvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:27:17.852452 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:17.852417    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17282308-47d0-4284-8e8d-29872d332940-cabundle-cert\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:27:17.852452 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:17.852452    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17282308-47d0-4284-8e8d-29872d332940-proxy-tls\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:27:17.852803 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:17.852468    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7clvj\" (UniqueName: \"kubernetes.io/projected/17282308-47d0-4284-8e8d-29872d332940-kube-api-access-7clvj\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:27:17.852803 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:17.852483    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17282308-47d0-4284-8e8d-29872d332940-kserve-provision-location\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:27:17.852803 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:17.852498    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17282308-47d0-4284-8e8d-29872d332940-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:27:18.033633 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:18.033556    2554 generic.go:356] "Generic (PLEG): container finished" podID="17282308-47d0-4284-8e8d-29872d332940" containerID="303be7bf1c667209e0283f4d4afb5d447bfd842f8c33430e8b9a5a32a9490bd6" exitCode=0
Jul 28 14:27:18.033633 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:18.033612    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745"
Jul 28 14:27:18.033813 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:18.033638    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745" event={"ID":"17282308-47d0-4284-8e8d-29872d332940","Type":"ContainerDied","Data":"303be7bf1c667209e0283f4d4afb5d447bfd842f8c33430e8b9a5a32a9490bd6"}
Jul 28 14:27:18.033813 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:18.033698    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745" event={"ID":"17282308-47d0-4284-8e8d-29872d332940","Type":"ContainerDied","Data":"33a559bc527b6cd71da1bc622c8309bb92d3046d7d12483273854f8424e208ec"}
Jul 28 14:27:18.033813 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:18.033725    2554 scope.go:117] "RemoveContainer" containerID="43e57e18d13fbe2446f298bcc1c800f53fc0235e701d210ba589bcb46b17ea94"
Jul 28 14:27:18.041895 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:18.041878    2554 scope.go:117] "RemoveContainer" containerID="303be7bf1c667209e0283f4d4afb5d447bfd842f8c33430e8b9a5a32a9490bd6"
Jul 28 14:27:18.049700 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:18.049312    2554 scope.go:117] "RemoveContainer" containerID="188b0647a86667e40600d08c4e91e9c98ab85f00d5a4b67b7b81341e053c6663"
Jul 28 14:27:18.051550 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:18.051529    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745"]
Jul 28 14:27:18.055491 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:18.055469    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745"]
Jul 28 14:27:18.058194 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:18.058178    2554 scope.go:117] "RemoveContainer" containerID="43e57e18d13fbe2446f298bcc1c800f53fc0235e701d210ba589bcb46b17ea94"
Jul 28 14:27:18.058470 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:27:18.058452    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43e57e18d13fbe2446f298bcc1c800f53fc0235e701d210ba589bcb46b17ea94\": container with ID starting with 43e57e18d13fbe2446f298bcc1c800f53fc0235e701d210ba589bcb46b17ea94 not found: ID does not exist" containerID="43e57e18d13fbe2446f298bcc1c800f53fc0235e701d210ba589bcb46b17ea94"
Jul 28 14:27:18.058543 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:18.058482    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43e57e18d13fbe2446f298bcc1c800f53fc0235e701d210ba589bcb46b17ea94"} err="failed to get container status \"43e57e18d13fbe2446f298bcc1c800f53fc0235e701d210ba589bcb46b17ea94\": rpc error: code = NotFound desc = could not find container \"43e57e18d13fbe2446f298bcc1c800f53fc0235e701d210ba589bcb46b17ea94\": container with ID starting with 43e57e18d13fbe2446f298bcc1c800f53fc0235e701d210ba589bcb46b17ea94 not found: ID does not exist"
Jul 28 14:27:18.058543 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:18.058501    2554 scope.go:117] "RemoveContainer" containerID="303be7bf1c667209e0283f4d4afb5d447bfd842f8c33430e8b9a5a32a9490bd6"
Jul 28 14:27:18.058747 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:27:18.058730    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"303be7bf1c667209e0283f4d4afb5d447bfd842f8c33430e8b9a5a32a9490bd6\": container with ID starting with 303be7bf1c667209e0283f4d4afb5d447bfd842f8c33430e8b9a5a32a9490bd6 not found: ID does not exist" containerID="303be7bf1c667209e0283f4d4afb5d447bfd842f8c33430e8b9a5a32a9490bd6"
Jul 28 14:27:18.058793 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:18.058753    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"303be7bf1c667209e0283f4d4afb5d447bfd842f8c33430e8b9a5a32a9490bd6"} err="failed to get container status \"303be7bf1c667209e0283f4d4afb5d447bfd842f8c33430e8b9a5a32a9490bd6\": rpc error: code = NotFound desc = could not find container \"303be7bf1c667209e0283f4d4afb5d447bfd842f8c33430e8b9a5a32a9490bd6\": container with ID starting with 303be7bf1c667209e0283f4d4afb5d447bfd842f8c33430e8b9a5a32a9490bd6 not found: ID does not exist"
Jul 28 14:27:18.058793 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:18.058769    2554 scope.go:117] "RemoveContainer" containerID="188b0647a86667e40600d08c4e91e9c98ab85f00d5a4b67b7b81341e053c6663"
Jul 28 14:27:18.059005 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:27:18.058986    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"188b0647a86667e40600d08c4e91e9c98ab85f00d5a4b67b7b81341e053c6663\": container with ID starting with 188b0647a86667e40600d08c4e91e9c98ab85f00d5a4b67b7b81341e053c6663 not found: ID does not exist" containerID="188b0647a86667e40600d08c4e91e9c98ab85f00d5a4b67b7b81341e053c6663"
Jul 28 14:27:18.059046 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:18.059009    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"188b0647a86667e40600d08c4e91e9c98ab85f00d5a4b67b7b81341e053c6663"} err="failed to get container status \"188b0647a86667e40600d08c4e91e9c98ab85f00d5a4b67b7b81341e053c6663\": rpc error: code = NotFound desc = could not find container \"188b0647a86667e40600d08c4e91e9c98ab85f00d5a4b67b7b81341e053c6663\": container with ID starting with 188b0647a86667e40600d08c4e91e9c98ab85f00d5a4b67b7b81341e053c6663 not found: ID does not exist"
Jul 28 14:27:18.703379 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:18.703330    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6fccdccf8b-kr745" podUID="17282308-47d0-4284-8e8d-29872d332940" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.68:8643/healthz\": context deadline exceeded"
Jul 28 14:27:19.801863 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:19.801830    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17282308-47d0-4284-8e8d-29872d332940" path="/var/lib/kubelet/pods/17282308-47d0-4284-8e8d-29872d332940/volumes"
Jul 28 14:27:22.051433 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:22.051403    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-8j5xk_b0816f9f-634a-4a39-92cf-5966ea9ca9eb/storage-initializer/0.log"
Jul 28 14:27:22.051832 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:22.051441    2554 generic.go:356] "Generic (PLEG): container finished" podID="b0816f9f-634a-4a39-92cf-5966ea9ca9eb" containerID="bb2bac443d3bdcf0f12b54d6778189fa03e5935c4a20556ce697c094d8179122" exitCode=1
Jul 28 14:27:22.051832 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:22.051485    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-8j5xk" event={"ID":"b0816f9f-634a-4a39-92cf-5966ea9ca9eb","Type":"ContainerDied","Data":"bb2bac443d3bdcf0f12b54d6778189fa03e5935c4a20556ce697c094d8179122"}
Jul 28 14:27:23.057641 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:23.057609    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-8j5xk_b0816f9f-634a-4a39-92cf-5966ea9ca9eb/storage-initializer/0.log"
Jul 28 14:27:23.058032 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:23.057674    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-8j5xk" event={"ID":"b0816f9f-634a-4a39-92cf-5966ea9ca9eb","Type":"ContainerStarted","Data":"a190aedb1bbb290536301cf0c30783e675f0b7077232cc9f78dfdb3ee59d8199"}
Jul 28 14:27:24.040624 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:24.040562    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-8j5xk"]
Jul 28 14:27:24.062261 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:24.062225    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-8j5xk" podUID="b0816f9f-634a-4a39-92cf-5966ea9ca9eb" containerName="storage-initializer" containerID="cri-o://a190aedb1bbb290536301cf0c30783e675f0b7077232cc9f78dfdb3ee59d8199" gracePeriod=30
Jul 28 14:27:27.712379 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:27.712357    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-8j5xk_b0816f9f-634a-4a39-92cf-5966ea9ca9eb/storage-initializer/1.log"
Jul 28 14:27:27.712777 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:27.712761    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-8j5xk_b0816f9f-634a-4a39-92cf-5966ea9ca9eb/storage-initializer/0.log"
Jul 28 14:27:27.715061 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:27.715046    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-8j5xk"
Jul 28 14:27:27.833290 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:27.833182    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5rqw\" (UniqueName: \"kubernetes.io/projected/b0816f9f-634a-4a39-92cf-5966ea9ca9eb-kube-api-access-r5rqw\") pod \"b0816f9f-634a-4a39-92cf-5966ea9ca9eb\" (UID: \"b0816f9f-634a-4a39-92cf-5966ea9ca9eb\") "
Jul 28 14:27:27.833290 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:27.833254    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0816f9f-634a-4a39-92cf-5966ea9ca9eb-kube-rbac-proxy-sar-config\") pod \"b0816f9f-634a-4a39-92cf-5966ea9ca9eb\" (UID: \"b0816f9f-634a-4a39-92cf-5966ea9ca9eb\") "
Jul 28 14:27:27.833523 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:27.833308    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0816f9f-634a-4a39-92cf-5966ea9ca9eb-kserve-provision-location\") pod \"b0816f9f-634a-4a39-92cf-5966ea9ca9eb\" (UID: \"b0816f9f-634a-4a39-92cf-5966ea9ca9eb\") "
Jul 28 14:27:27.833523 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:27.833344    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0816f9f-634a-4a39-92cf-5966ea9ca9eb-proxy-tls\") pod \"b0816f9f-634a-4a39-92cf-5966ea9ca9eb\" (UID: \"b0816f9f-634a-4a39-92cf-5966ea9ca9eb\") "
Jul 28 14:27:27.833634 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:27.833512    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0816f9f-634a-4a39-92cf-5966ea9ca9eb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b0816f9f-634a-4a39-92cf-5966ea9ca9eb" (UID: "b0816f9f-634a-4a39-92cf-5966ea9ca9eb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 14:27:27.833634 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:27.833594    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0816f9f-634a-4a39-92cf-5966ea9ca9eb-kserve-provision-location\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:27:27.833746 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:27.833628    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0816f9f-634a-4a39-92cf-5966ea9ca9eb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b0816f9f-634a-4a39-92cf-5966ea9ca9eb" (UID: "b0816f9f-634a-4a39-92cf-5966ea9ca9eb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 14:27:27.835546 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:27.835523    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0816f9f-634a-4a39-92cf-5966ea9ca9eb-kube-api-access-r5rqw" (OuterVolumeSpecName: "kube-api-access-r5rqw") pod "b0816f9f-634a-4a39-92cf-5966ea9ca9eb" (UID: "b0816f9f-634a-4a39-92cf-5966ea9ca9eb"). InnerVolumeSpecName "kube-api-access-r5rqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 14:27:27.835665 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:27.835573    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0816f9f-634a-4a39-92cf-5966ea9ca9eb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b0816f9f-634a-4a39-92cf-5966ea9ca9eb" (UID: "b0816f9f-634a-4a39-92cf-5966ea9ca9eb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 14:27:27.934268 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:27.934239    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0816f9f-634a-4a39-92cf-5966ea9ca9eb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:27:27.934268 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:27.934264    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0816f9f-634a-4a39-92cf-5966ea9ca9eb-proxy-tls\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:27:27.934268 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:27.934291    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5rqw\" (UniqueName: \"kubernetes.io/projected/b0816f9f-634a-4a39-92cf-5966ea9ca9eb-kube-api-access-r5rqw\") on node \"ip-10-0-129-177.ec2.internal\" DevicePath \"\""
Jul 28 14:27:28.078805 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:28.078775    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-8j5xk_b0816f9f-634a-4a39-92cf-5966ea9ca9eb/storage-initializer/1.log"
Jul 28 14:27:28.079187 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:28.079171    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-8j5xk_b0816f9f-634a-4a39-92cf-5966ea9ca9eb/storage-initializer/0.log"
Jul 28 14:27:28.079257 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:28.079208    2554 generic.go:356] "Generic (PLEG): container finished" podID="b0816f9f-634a-4a39-92cf-5966ea9ca9eb" containerID="a190aedb1bbb290536301cf0c30783e675f0b7077232cc9f78dfdb3ee59d8199" exitCode=1
Jul 28 14:27:28.079257 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:28.079237    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-8j5xk" event={"ID":"b0816f9f-634a-4a39-92cf-5966ea9ca9eb","Type":"ContainerDied","Data":"a190aedb1bbb290536301cf0c30783e675f0b7077232cc9f78dfdb3ee59d8199"}
Jul 28 14:27:28.079359 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:28.079265    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-8j5xk"
Jul 28 14:27:28.079359 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:28.079303    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-8j5xk" event={"ID":"b0816f9f-634a-4a39-92cf-5966ea9ca9eb","Type":"ContainerDied","Data":"9c7cb7c38e1433abfc7363717f1a5d44088ae2b1884eb835e05bc212ca05549e"}
Jul 28 14:27:28.079359 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:28.079330    2554 scope.go:117] "RemoveContainer" containerID="a190aedb1bbb290536301cf0c30783e675f0b7077232cc9f78dfdb3ee59d8199"
Jul 28 14:27:28.087803 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:28.087781    2554 scope.go:117] "RemoveContainer" containerID="bb2bac443d3bdcf0f12b54d6778189fa03e5935c4a20556ce697c094d8179122"
Jul 28 14:27:28.095140 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:28.095123    2554 scope.go:117] "RemoveContainer" containerID="a190aedb1bbb290536301cf0c30783e675f0b7077232cc9f78dfdb3ee59d8199"
Jul 28 14:27:28.095392 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:27:28.095375    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a190aedb1bbb290536301cf0c30783e675f0b7077232cc9f78dfdb3ee59d8199\": container with ID starting with a190aedb1bbb290536301cf0c30783e675f0b7077232cc9f78dfdb3ee59d8199 not found: ID does not exist" containerID="a190aedb1bbb290536301cf0c30783e675f0b7077232cc9f78dfdb3ee59d8199"
Jul 28 14:27:28.095440 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:28.095399    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a190aedb1bbb290536301cf0c30783e675f0b7077232cc9f78dfdb3ee59d8199"} err="failed to get container status \"a190aedb1bbb290536301cf0c30783e675f0b7077232cc9f78dfdb3ee59d8199\": rpc error: code = NotFound desc = could not find container \"a190aedb1bbb290536301cf0c30783e675f0b7077232cc9f78dfdb3ee59d8199\": container with ID starting with a190aedb1bbb290536301cf0c30783e675f0b7077232cc9f78dfdb3ee59d8199 not found: ID does not exist"
Jul 28 14:27:28.095440 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:28.095417    2554 scope.go:117] "RemoveContainer" containerID="bb2bac443d3bdcf0f12b54d6778189fa03e5935c4a20556ce697c094d8179122"
Jul 28 14:27:28.095627 ip-10-0-129-177 kubenswrapper[2554]: E0728 14:27:28.095609    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb2bac443d3bdcf0f12b54d6778189fa03e5935c4a20556ce697c094d8179122\": container with ID starting with bb2bac443d3bdcf0f12b54d6778189fa03e5935c4a20556ce697c094d8179122 not found: ID does not exist" containerID="bb2bac443d3bdcf0f12b54d6778189fa03e5935c4a20556ce697c094d8179122"
Jul 28 14:27:28.095671 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:28.095631    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb2bac443d3bdcf0f12b54d6778189fa03e5935c4a20556ce697c094d8179122"} err="failed to get container status \"bb2bac443d3bdcf0f12b54d6778189fa03e5935c4a20556ce697c094d8179122\": rpc error: code = NotFound desc = could not find container \"bb2bac443d3bdcf0f12b54d6778189fa03e5935c4a20556ce697c094d8179122\": container with ID starting with bb2bac443d3bdcf0f12b54d6778189fa03e5935c4a20556ce697c094d8179122 not found: ID does not exist"
Jul 28 14:27:28.113685 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:28.113648    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-8j5xk"]
Jul 28 14:27:28.117163 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:28.117141    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-599448cc44-8j5xk"]
Jul 28 14:27:29.801601 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:27:29.801567    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0816f9f-634a-4a39-92cf-5966ea9ca9eb" path="/var/lib/kubelet/pods/b0816f9f-634a-4a39-92cf-5966ea9ca9eb/volumes"
Jul 28 14:28:04.172426 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:04.172391    2554 ???:1] "http: TLS handshake error from 10.0.129.177:40920: EOF"
Jul 28 14:28:04.182117 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:04.182091    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jcccx_9f6f2eb9-22ea-4119-b474-75c3d6ec08eb/global-pull-secret-syncer/0.log"
Jul 28 14:28:04.389867 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:04.389835    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-x88hp_4afbaf34-0b91-4704-9e53-39f24049c3c7/konnectivity-agent/0.log"
Jul 28 14:28:04.410784 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:04.410756    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-177.ec2.internal_571f960b5c36dc73834816c1792a750a/haproxy/0.log"
Jul 28 14:28:08.042472 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:08.042439    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-kc5t6_1a00af5e-6526-4eb0-951d-1ac554676d9e/cluster-monitoring-operator/0.log"
Jul 28 14:28:08.387911 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:08.387832    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lwp6v_b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6/node-exporter/0.log"
Jul 28 14:28:08.443415 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:08.443372    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lwp6v_b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6/kube-rbac-proxy/0.log"
Jul 28 14:28:08.489040 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:08.489008    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lwp6v_b0640ec3-f2c9-407d-80b2-c9e7b4b30aa6/init-textfile/0.log"
Jul 28 14:28:09.250832 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:09.250805    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-b5g48_90416dc6-1ef1-4e61-91fa-f5aaaddc09e5/prometheus-operator-admission-webhook/0.log"
Jul 28 14:28:09.508781 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:09.508699    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-665d6c9587-cjqhj_66a5dfa8-fc4f-4fae-810f-3074802421a2/thanos-query/0.log"
Jul 28 14:28:09.543917 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:09.543890    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-665d6c9587-cjqhj_66a5dfa8-fc4f-4fae-810f-3074802421a2/kube-rbac-proxy-web/0.log"
Jul 28 14:28:09.569429 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:09.569404    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-665d6c9587-cjqhj_66a5dfa8-fc4f-4fae-810f-3074802421a2/kube-rbac-proxy/0.log"
Jul 28 14:28:09.594133 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:09.594109    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-665d6c9587-cjqhj_66a5dfa8-fc4f-4fae-810f-3074802421a2/prom-label-proxy/0.log"
Jul 28 14:28:09.681131 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:09.681108    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-665d6c9587-cjqhj_66a5dfa8-fc4f-4fae-810f-3074802421a2/kube-rbac-proxy-rules/0.log"
Jul 28 14:28:09.738239 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:09.738211    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-665d6c9587-cjqhj_66a5dfa8-fc4f-4fae-810f-3074802421a2/kube-rbac-proxy-metrics/0.log"
Jul 28 14:28:11.958731 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:11.958704    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-f599fd97c-ndwrb_b504eb3b-e209-4ed9-b348-fa7cbbea7681/console/0.log"
Jul 28 14:28:12.060593 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:12.060556    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-j8v88/perf-node-gather-daemonset-4q9nz"]
Jul 28 14:28:12.060936 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:12.060917    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17282308-47d0-4284-8e8d-29872d332940" containerName="kube-rbac-proxy"
Jul 28 14:28:12.061025 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:12.060939    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="17282308-47d0-4284-8e8d-29872d332940" containerName="kube-rbac-proxy"
Jul 28 14:28:12.061025 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:12.060955    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17282308-47d0-4284-8e8d-29872d332940" containerName="kserve-container"
Jul 28 14:28:12.061025 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:12.060964    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="17282308-47d0-4284-8e8d-29872d332940" containerName="kserve-container"
Jul 28 14:28:12.061025 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:12.060982    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17282308-47d0-4284-8e8d-29872d332940" containerName="storage-initializer"
Jul 28 14:28:12.061025 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:12.060990    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="17282308-47d0-4284-8e8d-29872d332940" containerName="storage-initializer"
Jul 28 14:28:12.061025 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:12.061007    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0816f9f-634a-4a39-92cf-5966ea9ca9eb" containerName="storage-initializer"
Jul 28 14:28:12.061025 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:12.061016    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0816f9f-634a-4a39-92cf-5966ea9ca9eb" containerName="storage-initializer"
Jul 28 14:28:12.061401 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:12.061028    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0816f9f-634a-4a39-92cf-5966ea9ca9eb" containerName="storage-initializer"
Jul 28 14:28:12.061401 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:12.061036    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0816f9f-634a-4a39-92cf-5966ea9ca9eb" containerName="storage-initializer"
Jul 28 14:28:12.061401 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:12.061122    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0816f9f-634a-4a39-92cf-5966ea9ca9eb" containerName="storage-initializer"
Jul 28 14:28:12.061401 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:12.061137    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="17282308-47d0-4284-8e8d-29872d332940" containerName="kube-rbac-proxy"
Jul 28 14:28:12.061401 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:12.061147    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0816f9f-634a-4a39-92cf-5966ea9ca9eb" containerName="storage-initializer"
Jul 28 14:28:12.061401 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:12.061155    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="17282308-47d0-4284-8e8d-29872d332940" containerName="kserve-container"
Jul 28 14:28:12.064365 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:12.064335    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j8v88/perf-node-gather-daemonset-4q9nz"
Jul 28 14:28:12.067026 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:12.067001    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j8v88\"/\"openshift-service-ca.crt\""
Jul 28 14:28:12.067137 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:12.067103    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j8v88\"/\"kube-root-ca.crt\""
Jul 28 14:28:12.067967 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:12.067949    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-j8v88\"/\"default-dockercfg-hswfh\""
Jul 28 14:28:12.072478 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:12.072459    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j8v88/perf-node-gather-daemonset-4q9nz"]
Jul 28 14:28:12.190601 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:12.190567    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cvsk\" (UniqueName: \"kubernetes.io/projected/9769494f-e3a3-4faa-b43c-06c694125a19-kube-api-access-7cvsk\") pod \"perf-node-gather-daemonset-4q9nz\" (UID: \"9769494f-e3a3-4faa-b43c-06c694125a19\") " pod="openshift-must-gather-j8v88/perf-node-gather-daemonset-4q9nz"
Jul 28 14:28:12.190601 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:12.190605    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9769494f-e3a3-4faa-b43c-06c694125a19-sys\") pod \"perf-node-gather-daemonset-4q9nz\" (UID: \"9769494f-e3a3-4faa-b43c-06c694125a19\") " pod="openshift-must-gather-j8v88/perf-node-gather-daemonset-4q9nz"
Jul 28 14:28:12.190823 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:12.190677    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9769494f-e3a3-4faa-b43c-06c694125a19-podres\") pod \"perf-node-gather-daemonset-4q9nz\" (UID: \"9769494f-e3a3-4faa-b43c-06c694125a19\") " pod="openshift-must-gather-j8v88/perf-node-gather-daemonset-4q9nz"
Jul 28 14:28:12.190823 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:12.190712    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9769494f-e3a3-4faa-b43c-06c694125a19-proc\") pod \"perf-node-gather-daemonset-4q9nz\" (UID: \"9769494f-e3a3-4faa-b43c-06c694125a19\") " pod="openshift-must-gather-j8v88/perf-node-gather-daemonset-4q9nz"
Jul 28 14:28:12.190823 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:12.190753    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9769494f-e3a3-4faa-b43c-06c694125a19-lib-modules\") pod \"perf-node-gather-daemonset-4q9nz\" (UID: \"9769494f-e3a3-4faa-b43c-06c694125a19\") " pod="openshift-must-gather-j8v88/perf-node-gather-daemonset-4q9nz"
Jul 28 14:28:12.291409 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:12.291332    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9769494f-e3a3-4faa-b43c-06c694125a19-lib-modules\") pod \"perf-node-gather-daemonset-4q9nz\" (UID: \"9769494f-e3a3-4faa-b43c-06c694125a19\") " pod="openshift-must-gather-j8v88/perf-node-gather-daemonset-4q9nz"
Jul 28 14:28:12.291409 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:12.291392    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cvsk\" (UniqueName: \"kubernetes.io/projected/9769494f-e3a3-4faa-b43c-06c694125a19-kube-api-access-7cvsk\") pod \"perf-node-gather-daemonset-4q9nz\" (UID: \"9769494f-e3a3-4faa-b43c-06c694125a19\") " pod="openshift-must-gather-j8v88/perf-node-gather-daemonset-4q9nz"
Jul 28 14:28:12.291614 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:12.291416    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9769494f-e3a3-4faa-b43c-06c694125a19-sys\") pod \"perf-node-gather-daemonset-4q9nz\" (UID: \"9769494f-e3a3-4faa-b43c-06c694125a19\") " pod="openshift-must-gather-j8v88/perf-node-gather-daemonset-4q9nz"
Jul 28 14:28:12.291614 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:12.291456    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9769494f-e3a3-4faa-b43c-06c694125a19-podres\") pod \"perf-node-gather-daemonset-4q9nz\" (UID: \"9769494f-e3a3-4faa-b43c-06c694125a19\") " pod="openshift-must-gather-j8v88/perf-node-gather-daemonset-4q9nz"
Jul 28 14:28:12.291614 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:12.291475    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9769494f-e3a3-4faa-b43c-06c694125a19-proc\") pod \"perf-node-gather-daemonset-4q9nz\" (UID: \"9769494f-e3a3-4faa-b43c-06c694125a19\") " pod="openshift-must-gather-j8v88/perf-node-gather-daemonset-4q9nz"
Jul 28 14:28:12.291614 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:12.291525    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9769494f-e3a3-4faa-b43c-06c694125a19-lib-modules\") pod \"perf-node-gather-daemonset-4q9nz\" (UID: \"9769494f-e3a3-4faa-b43c-06c694125a19\") " pod="openshift-must-gather-j8v88/perf-node-gather-daemonset-4q9nz"
Jul 28 14:28:12.291614 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:12.291551    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9769494f-e3a3-4faa-b43c-06c694125a19-sys\") pod \"perf-node-gather-daemonset-4q9nz\" (UID: \"9769494f-e3a3-4faa-b43c-06c694125a19\") " pod="openshift-must-gather-j8v88/perf-node-gather-daemonset-4q9nz"
Jul 28 14:28:12.291614 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:12.291556    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9769494f-e3a3-4faa-b43c-06c694125a19-proc\") pod \"perf-node-gather-daemonset-4q9nz\" (UID: \"9769494f-e3a3-4faa-b43c-06c694125a19\") " pod="openshift-must-gather-j8v88/perf-node-gather-daemonset-4q9nz"
Jul 28 14:28:12.291614 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:12.291599    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9769494f-e3a3-4faa-b43c-06c694125a19-podres\") pod \"perf-node-gather-daemonset-4q9nz\" (UID: \"9769494f-e3a3-4faa-b43c-06c694125a19\") " pod="openshift-must-gather-j8v88/perf-node-gather-daemonset-4q9nz"
Jul 28 14:28:12.302194 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:12.302167    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cvsk\" (UniqueName: \"kubernetes.io/projected/9769494f-e3a3-4faa-b43c-06c694125a19-kube-api-access-7cvsk\") pod \"perf-node-gather-daemonset-4q9nz\" (UID: \"9769494f-e3a3-4faa-b43c-06c694125a19\") " pod="openshift-must-gather-j8v88/perf-node-gather-daemonset-4q9nz"
Jul 28 14:28:12.374787 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:12.374757    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j8v88/perf-node-gather-daemonset-4q9nz"
Jul 28 14:28:12.510426 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:12.510401    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j8v88/perf-node-gather-daemonset-4q9nz"]
Jul 28 14:28:12.512422 ip-10-0-129-177 kubenswrapper[2554]: W0728 14:28:12.512395    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9769494f_e3a3_4faa_b43c_06c694125a19.slice/crio-6b70a60cc61a8a78c67bb8a18085e23b20d147fda6ce86766f3cc6c5b4bc0bf0 WatchSource:0}: Error finding container 6b70a60cc61a8a78c67bb8a18085e23b20d147fda6ce86766f3cc6c5b4bc0bf0: Status 404 returned error can't find the container with id 6b70a60cc61a8a78c67bb8a18085e23b20d147fda6ce86766f3cc6c5b4bc0bf0
Jul 28 14:28:12.607474 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:12.607453    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-lnd2f_d0e527a0-8a32-4bff-bd54-c3fb3293d699/volume-data-source-validator/0.log"
Jul 28 14:28:13.256705 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:13.256670    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j8v88/perf-node-gather-daemonset-4q9nz" event={"ID":"9769494f-e3a3-4faa-b43c-06c694125a19","Type":"ContainerStarted","Data":"6d6f03c7075c399a34185722672550290a553d42c4e3dde837162d8a9a397dd8"}
Jul 28 14:28:13.257189 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:13.256727    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j8v88/perf-node-gather-daemonset-4q9nz" event={"ID":"9769494f-e3a3-4faa-b43c-06c694125a19","Type":"ContainerStarted","Data":"6b70a60cc61a8a78c67bb8a18085e23b20d147fda6ce86766f3cc6c5b4bc0bf0"}
Jul 28 14:28:13.257189 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:13.256796    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-j8v88/perf-node-gather-daemonset-4q9nz"
Jul 28 14:28:13.273858 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:13.273815    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-j8v88/perf-node-gather-daemonset-4q9nz" podStartSLOduration=1.273802941 podStartE2EDuration="1.273802941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 14:28:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 14:28:13.272935148 +0000 UTC m=+5690.058810260" watchObservedRunningTime="2026-07-28 14:28:13.273802941 +0000 UTC m=+5690.059678091"
Jul 28 14:28:13.581666 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:13.581594    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-t4cwf_0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7/dns/0.log"
Jul 28 14:28:13.605184 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:13.605163    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-t4cwf_0cb79c0b-58f8-414b-bc96-8f3c83f2f7a7/kube-rbac-proxy/0.log"
Jul 28 14:28:13.816569 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:13.816539    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-sn9v6_d336b5ed-db42-40f6-9e7a-a1882cb28484/dns-node-resolver/0.log"
Jul 28 14:28:14.293358 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:14.293333    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-dd79996f-p9sgt_d85329d3-3a73-4ffa-9b7a-4aa122921a9a/registry/0.log"
Jul 28 14:28:14.316587 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:14.316561    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-68tbg_48e4bcde-dcb9-4e7e-8b15-e8682eb564d1/node-ca/0.log"
Jul 28 14:28:15.103011 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:15.102982    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-767866489f-t5k98_128b78d0-c26a-4fb6-b914-1cafd9565ff7/router/0.log"
Jul 28 14:28:15.450623 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:15.450585    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-29778_41f6b60b-caf3-4c4c-92b0-386a82bcfc4a/serve-healthcheck-canary/0.log"
Jul 28 14:28:15.884451 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:15.884375    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-mpgfw_324a14fe-0e26-4cea-b17f-5efda6b502de/insights-operator/0.log"
Jul 28 14:28:15.885357 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:15.885340    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-mpgfw_324a14fe-0e26-4cea-b17f-5efda6b502de/insights-operator/1.log"
Jul 28 14:28:16.040495 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:16.040464    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rgh7t_fbd7f2cb-b409-4e32-854e-043f12d72d21/kube-rbac-proxy/0.log"
Jul 28 14:28:16.062690 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:16.062664    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rgh7t_fbd7f2cb-b409-4e32-854e-043f12d72d21/exporter/0.log"
Jul 28 14:28:16.087765 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:16.087743    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rgh7t_fbd7f2cb-b409-4e32-854e-043f12d72d21/extractor/0.log"
Jul 28 14:28:18.393170 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:18.393137    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-ff96d5959-dssz7_f3565914-e2f8-4684-a46d-f3bc4659b7de/manager/0.log"
Jul 28 14:28:18.451253 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:18.451228    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-jrxz2_de86784c-96ee-431a-a222-054aef330287/server/0.log"
Jul 28 14:28:18.738310 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:18.738263    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-75d9cdbddf-7lz7w_19f8fbde-a596-4350-a520-e6dabe88a1c0/manager/0.log"
Jul 28 14:28:18.973204 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:18.973171    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-zdmzp_8b41140e-33c2-496d-8330-fd37f1a99165/seaweedfs/0.log"
Jul 28 14:28:19.004889 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:19.004825    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-custom-8b44d767d-rlhbx_93f9a4a0-b773-4d3d-bd4f-d72a7089f8b1/seaweedfs-tls-custom/0.log"
Jul 28 14:28:19.034228 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:19.034205    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-serving-67b7477c6d-s4k5q_a2044a0c-70f6-4d6d-93fc-2468ce691b0d/seaweedfs-tls-serving/0.log"
Jul 28 14:28:19.269780 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:19.269718    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-j8v88/perf-node-gather-daemonset-4q9nz"
Jul 28 14:28:24.802468 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:24.802441    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ckrz7_1118726b-19e7-4111-9934-60e937e54642/kube-multus-additional-cni-plugins/0.log"
Jul 28 14:28:24.826453 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:24.826433    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ckrz7_1118726b-19e7-4111-9934-60e937e54642/egress-router-binary-copy/0.log"
Jul 28 14:28:24.856988 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:24.856970    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ckrz7_1118726b-19e7-4111-9934-60e937e54642/cni-plugins/0.log"
Jul 28 14:28:24.884110 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:24.884093    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ckrz7_1118726b-19e7-4111-9934-60e937e54642/bond-cni-plugin/0.log"
Jul 28 14:28:24.907671 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:24.907641    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ckrz7_1118726b-19e7-4111-9934-60e937e54642/routeoverride-cni/0.log"
Jul 28 14:28:24.930582 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:24.930560    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ckrz7_1118726b-19e7-4111-9934-60e937e54642/whereabouts-cni-bincopy/0.log"
Jul 28 14:28:24.954278 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:24.954258    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ckrz7_1118726b-19e7-4111-9934-60e937e54642/whereabouts-cni/0.log"
Jul 28 14:28:25.363898 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:25.363847    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jd2p2_f02e9322-ad0d-4b41-b3e4-a4c6a71f566d/kube-multus/0.log"
Jul 28 14:28:25.540053 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:25.539941    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-vvbch_d04af0e5-3e6d-4027-8509-0af92acc4035/network-metrics-daemon/0.log"
Jul 28 14:28:25.560310 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:25.560268    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-vvbch_d04af0e5-3e6d-4027-8509-0af92acc4035/kube-rbac-proxy/0.log"
Jul 28 14:28:26.759372 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:26.759342    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-st9kv_c549168d-d9be-4f07-8fbe-bf4d7c49d8c8/ovn-controller/0.log"
Jul 28 14:28:26.776156 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:26.776132    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-st9kv_c549168d-d9be-4f07-8fbe-bf4d7c49d8c8/ovn-acl-logging/0.log"
Jul 28 14:28:26.803328 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:26.803302    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-st9kv_c549168d-d9be-4f07-8fbe-bf4d7c49d8c8/kube-rbac-proxy-node/0.log"
Jul 28 14:28:26.825215 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:26.825195    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-st9kv_c549168d-d9be-4f07-8fbe-bf4d7c49d8c8/kube-rbac-proxy-ovn-metrics/0.log"
Jul 28 14:28:26.846540 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:26.846518    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-st9kv_c549168d-d9be-4f07-8fbe-bf4d7c49d8c8/northd/0.log"
Jul 28 14:28:26.869626 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:26.869590    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-st9kv_c549168d-d9be-4f07-8fbe-bf4d7c49d8c8/nbdb/0.log"
Jul 28 14:28:26.893626 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:26.893597    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-st9kv_c549168d-d9be-4f07-8fbe-bf4d7c49d8c8/sbdb/0.log"
Jul 28 14:28:26.944368 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:26.944335    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-st9kv_c549168d-d9be-4f07-8fbe-bf4d7c49d8c8/ovnkube-controller/0.log"
Jul 28 14:28:27.944776 ip-10-0-129-177 kubenswrapper[2554]: I0728 14:28:27.944744    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-4l2q4_78f39c2f-8193-46af-b59d-8dc730d2e61a/network-check-target-container/0.log"