Jul 27 21:02:33.924979 ip-10-0-142-219 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 27 21:02:33.924991 ip-10-0-142-219 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 27 21:02:33.925001 ip-10-0-142-219 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 27 21:02:33.925307 ip-10-0-142-219 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 27 21:02:45.477164 ip-10-0-142-219 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 27 21:02:45.477182 ip-10-0-142-219 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot eebded14b8504d8b8966db92bef5516a --
Jul 27 21:04:35.345754 ip-10-0-142-219 systemd[1]: Starting Kubernetes Kubelet...
Jul 27 21:04:35.733093 ip-10-0-142-219 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 21:04:35.733093 ip-10-0-142-219 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 27 21:04:35.733093 ip-10-0-142-219 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 21:04:35.733093 ip-10-0-142-219 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 27 21:04:35.733093 ip-10-0-142-219 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 21:04:35.733792 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.733158    2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 27 21:04:35.740084 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740055    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 21:04:35.740084 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740078    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 21:04:35.740084 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740084    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 21:04:35.740084 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740089    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 21:04:35.740084 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740093    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 21:04:35.740394 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740098    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 21:04:35.740394 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740103    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 21:04:35.740394 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740107    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 21:04:35.740394 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740111    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 21:04:35.740394 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740114    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 21:04:35.740394 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740118    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 21:04:35.740394 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740122    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 21:04:35.740394 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740125    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 21:04:35.740394 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740131    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 21:04:35.740394 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740137    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 21:04:35.740394 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740142    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 21:04:35.740394 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740146    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 21:04:35.740394 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740149    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 21:04:35.740394 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740153    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 21:04:35.740394 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740157    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 21:04:35.740394 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740161    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 21:04:35.740394 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740164    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 21:04:35.740394 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740167    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 21:04:35.740394 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740171    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 21:04:35.741179 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740175    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 21:04:35.741179 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740179    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 21:04:35.741179 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740182    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 21:04:35.741179 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740186    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 21:04:35.741179 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740190    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 21:04:35.741179 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740194    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 21:04:35.741179 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740198    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 21:04:35.741179 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740202    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 21:04:35.741179 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740214    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 21:04:35.741179 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740218    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 21:04:35.741179 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740222    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 21:04:35.741179 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740226    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 21:04:35.741179 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740230    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 21:04:35.741179 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740234    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 21:04:35.741179 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740239    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 21:04:35.741179 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740244    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 21:04:35.741179 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740248    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 21:04:35.741179 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740253    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 21:04:35.741179 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740257    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 21:04:35.741179 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740261    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 21:04:35.742079 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740266    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 21:04:35.742079 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740270    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 21:04:35.742079 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740274    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 21:04:35.742079 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740278    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 21:04:35.742079 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740283    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 21:04:35.742079 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740287    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 21:04:35.742079 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740294    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 21:04:35.742079 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740299    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 21:04:35.742079 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740303    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 21:04:35.742079 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740307    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 21:04:35.742079 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740312    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 21:04:35.742079 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740316    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 21:04:35.742079 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740321    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 21:04:35.742079 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740326    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 21:04:35.742079 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740331    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 21:04:35.742079 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740337    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 21:04:35.742079 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740341    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 21:04:35.742079 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740345    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 21:04:35.742079 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740350    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 21:04:35.742079 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740354    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 21:04:35.742918 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740358    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 21:04:35.742918 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740362    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 21:04:35.742918 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740367    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 21:04:35.742918 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740372    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 21:04:35.742918 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740377    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 21:04:35.742918 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740381    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 21:04:35.742918 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740387    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 21:04:35.742918 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740391    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 21:04:35.742918 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740395    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 21:04:35.742918 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740400    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 21:04:35.742918 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740404    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 21:04:35.742918 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740408    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 21:04:35.742918 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740415    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 21:04:35.742918 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740421    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 21:04:35.742918 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740426    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 21:04:35.742918 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740431    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 21:04:35.742918 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740435    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 21:04:35.742918 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740439    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 21:04:35.742918 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740444    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 21:04:35.743747 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740451    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 21:04:35.743747 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740458    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 21:04:35.743747 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740462    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 21:04:35.743747 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740482    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 21:04:35.743747 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740488    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 21:04:35.743747 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740492    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 21:04:35.743747 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740498    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 21:04:35.743747 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740502    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 21:04:35.743747 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740507    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 21:04:35.743747 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740511    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 21:04:35.743747 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740515    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 21:04:35.743747 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740519    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 21:04:35.743747 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740523    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 21:04:35.743747 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740528    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 21:04:35.743747 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740532    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 21:04:35.743747 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.740537    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 21:04:35.743747 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.741749    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 21:04:35.743747 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.741761    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 21:04:35.743747 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.741766    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 21:04:35.744565 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.741771    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 21:04:35.744565 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.741776    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 21:04:35.744565 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.741780    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 21:04:35.744565 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.741784    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 21:04:35.744565 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.741789    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 21:04:35.744565 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.741793    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 21:04:35.744565 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.741798    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 21:04:35.744565 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.741802    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 21:04:35.744565 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.741806    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 21:04:35.744565 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.741810    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 21:04:35.744565 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.741815    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 21:04:35.744565 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.741819    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 21:04:35.744565 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.741825    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 21:04:35.744565 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.741831    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 21:04:35.744565 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.741838    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 21:04:35.744565 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.741843    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 21:04:35.744565 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.741848    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 21:04:35.744565 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.741853    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 21:04:35.744565 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.741857    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 21:04:35.745230 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.741862    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 21:04:35.745230 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.741866    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 21:04:35.745230 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.741870    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 21:04:35.745230 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.741875    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 21:04:35.745230 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.741879    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 21:04:35.745230 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.741884    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 21:04:35.745230 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.741888    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 21:04:35.745230 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.741892    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 21:04:35.745230 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.741896    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 21:04:35.745230 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.741900    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 21:04:35.745230 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.741904    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 21:04:35.745230 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.741908    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 21:04:35.745230 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.741913    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 21:04:35.745230 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.741917    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 21:04:35.745230 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.741922    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 21:04:35.745230 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.741926    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 21:04:35.745230 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.741930    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 21:04:35.745230 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.741935    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 21:04:35.745230 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.741939    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 21:04:35.745230 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.741945    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 21:04:35.745867 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.741949    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 21:04:35.745867 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.741953    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 21:04:35.745867 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.741960    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 21:04:35.745867 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.741965    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 21:04:35.745867 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.741970    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 21:04:35.745867 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.741975    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 21:04:35.745867 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.741979    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 21:04:35.745867 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.741983    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 21:04:35.745867 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.741990    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 21:04:35.745867 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.741994    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 21:04:35.745867 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.741999    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 21:04:35.745867 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742003    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 21:04:35.745867 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742008    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 21:04:35.745867 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742013    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 21:04:35.745867 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742017    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 21:04:35.745867 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742021    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 21:04:35.745867 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742026    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 21:04:35.745867 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742030    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 21:04:35.745867 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742034    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 21:04:35.746442 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742040    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 21:04:35.746442 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742045    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 21:04:35.746442 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742049    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 21:04:35.746442 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742053    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 21:04:35.746442 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742057    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 21:04:35.746442 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742062    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 21:04:35.746442 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742066    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 21:04:35.746442 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742071    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 21:04:35.746442 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742075    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 21:04:35.746442 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742079    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 21:04:35.746442 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742082    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 21:04:35.746442 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742086    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 21:04:35.746442 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742091    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 21:04:35.746442 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742095    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 21:04:35.746442 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742099    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 21:04:35.746442 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742103    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 21:04:35.746442 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742107    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 21:04:35.746442 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742112    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 21:04:35.746442 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742116    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 21:04:35.746442 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742121    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 21:04:35.747131 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742127    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 21:04:35.747131 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742133    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 21:04:35.747131 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742139    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 21:04:35.747131 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742143    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 21:04:35.747131 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742147    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 21:04:35.747131 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742151    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 21:04:35.747131 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742156    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 21:04:35.747131 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742160    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 21:04:35.747131 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742164    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 21:04:35.747131 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742169    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 21:04:35.747131 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742173    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 21:04:35.747131 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742181    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 21:04:35.747131 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742185    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 21:04:35.747131 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742188    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 21:04:35.747131 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742192    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 21:04:35.747131 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742197    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 21:04:35.747131 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742201    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 21:04:35.747131 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742205    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 21:04:35.747131 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742895    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 21:04:35.747628 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742904    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 21:04:35.747628 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742909    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 21:04:35.747628 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742913    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 21:04:35.747628 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742917    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 21:04:35.747628 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742921    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 21:04:35.747628 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742926    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 21:04:35.747628 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742929    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 21:04:35.747628 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742934    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 21:04:35.747628 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742938    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 21:04:35.747628 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742943    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 21:04:35.747628 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742947    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 21:04:35.747628 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742951    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 21:04:35.747628 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742956    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 21:04:35.747628 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742961    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 21:04:35.747628 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742965    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 21:04:35.747628 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742978    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 21:04:35.747628 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742983    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 21:04:35.747628 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742987    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 21:04:35.747628 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742991    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 21:04:35.747628 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.742996    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 21:04:35.748218 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743000    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 21:04:35.748218 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743004    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 21:04:35.748218 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743009    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 21:04:35.748218 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743013    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 21:04:35.748218 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743019    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 21:04:35.748218 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743023    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 21:04:35.748218 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743027    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 21:04:35.748218 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743031    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 21:04:35.748218 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743036    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 21:04:35.748218 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743040    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 21:04:35.748218 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743046    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 21:04:35.748218 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743050    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 21:04:35.748218 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743054    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 21:04:35.748218 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743058    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 21:04:35.748218 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743062    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 21:04:35.748218 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743066    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 21:04:35.748218 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743070    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 21:04:35.748218 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743074    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 21:04:35.748218 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743078    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 21:04:35.748713 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743082    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 21:04:35.748713 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743086    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 21:04:35.748713 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743093    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 21:04:35.748713 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743099    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 21:04:35.748713 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743106    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 21:04:35.748713 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743110    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 21:04:35.748713 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743115    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 21:04:35.748713 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743119    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 21:04:35.748713 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743123    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 21:04:35.748713 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743135    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 21:04:35.748713 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743139    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 21:04:35.748713 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743144    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 21:04:35.748713 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743148    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 21:04:35.748713 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743152    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 21:04:35.748713 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743156    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 21:04:35.748713 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743161    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 21:04:35.748713 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743165    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 21:04:35.748713 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743171    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 21:04:35.748713 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743176    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 21:04:35.748713 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743180    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 21:04:35.749211 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743184    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 21:04:35.749211 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743188    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 21:04:35.749211 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743192    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 21:04:35.749211 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743196    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 21:04:35.749211 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743200    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 21:04:35.749211 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743204    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 21:04:35.749211 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743208    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 21:04:35.749211 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743222    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 21:04:35.749211 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743227    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 21:04:35.749211 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743231    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 21:04:35.749211 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743235    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 21:04:35.749211 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743239    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 21:04:35.749211 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743243    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 21:04:35.749211 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743256    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 21:04:35.749211 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743262    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 21:04:35.749211 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743268    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 21:04:35.749211 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743276    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 21:04:35.749211 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743281    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 21:04:35.749211 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743285    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 21:04:35.749689 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743290    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 21:04:35.749689 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743295    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 21:04:35.749689 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743299    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 21:04:35.749689 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743311    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 21:04:35.749689 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743315    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 21:04:35.749689 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743321    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 21:04:35.749689 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743327    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 21:04:35.749689 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743331    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 21:04:35.749689 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743336    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 21:04:35.749689 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743340    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 21:04:35.749689 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743345    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 21:04:35.749689 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743350    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 21:04:35.749689 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743354    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 21:04:35.749689 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743358    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 21:04:35.749689 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743362    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 21:04:35.749689 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743366    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 21:04:35.749689 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743370    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 21:04:35.749689 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743375    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 21:04:35.749689 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743379    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 21:04:35.749689 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.743383    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 21:04:35.750161 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743505    2577 flags.go:64] FLAG: --address="0.0.0.0"
Jul 27 21:04:35.750161 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743517    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 27 21:04:35.750161 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743526    2577 flags.go:64] FLAG: --anonymous-auth="true"
Jul 27 21:04:35.750161 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743533    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 27 21:04:35.750161 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743539    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 27 21:04:35.750161 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743545    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 27 21:04:35.750161 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743551    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 27 21:04:35.750161 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743559    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 27 21:04:35.750161 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743564    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 27 21:04:35.750161 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743570    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 27 21:04:35.750161 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743575    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 27 21:04:35.750161 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743580    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 27 21:04:35.750161 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743586    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 27 21:04:35.750161 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743591    2577 flags.go:64] FLAG: --cgroup-root=""
Jul 27 21:04:35.750161 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743595    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 27 21:04:35.750161 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743600    2577 flags.go:64] FLAG: --client-ca-file=""
Jul 27 21:04:35.750161 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743613    2577 flags.go:64] FLAG: --cloud-provider="external"
Jul 27 21:04:35.750161 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743618    2577 flags.go:64] FLAG: --cluster-dns="[]"
Jul 27 21:04:35.750161 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743626    2577 flags.go:64] FLAG: --cluster-domain=""
Jul 27 21:04:35.750161 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743631    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 27 21:04:35.750161 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743636    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 27 21:04:35.750161 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743641    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 27 21:04:35.750161 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743648    2577 flags.go:64] FLAG: --container-log-max-files="5"
Jul 27 21:04:35.750161 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743655    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 27 21:04:35.750750 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743659    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 27 21:04:35.750750 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743664    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 27 21:04:35.750750 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743669    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 27 21:04:35.750750 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743674    2577 flags.go:64] FLAG: --contention-profiling="false"
Jul 27 21:04:35.750750 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743679    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 27 21:04:35.750750 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743683    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 27 21:04:35.750750 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743688    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 27 21:04:35.750750 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743693    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 27 21:04:35.750750 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743700    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 27 21:04:35.750750 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743705    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 27 21:04:35.750750 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743709    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 27 21:04:35.750750 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743714    2577 flags.go:64] FLAG: --enable-load-reader="false"
Jul 27 21:04:35.750750 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743719    2577 flags.go:64] FLAG: --enable-server="true"
Jul 27 21:04:35.750750 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743724    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 27 21:04:35.750750 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743731    2577 flags.go:64] FLAG: --event-burst="100"
Jul 27 21:04:35.750750 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743736    2577 flags.go:64] FLAG: --event-qps="50"
Jul 27 21:04:35.750750 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743741    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 27 21:04:35.750750 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743748    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 27 21:04:35.750750 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743752    2577 flags.go:64] FLAG: --eviction-hard=""
Jul 27 21:04:35.750750 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743758    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 27 21:04:35.750750 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743762    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 27 21:04:35.750750 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743767    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 27 21:04:35.750750 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743772    2577 flags.go:64] FLAG: --eviction-soft=""
Jul 27 21:04:35.750750 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743776    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 27 21:04:35.750750 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743781    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 27 21:04:35.751480 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743787    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 27 21:04:35.751480 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743792    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 27 21:04:35.751480 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743796    2577 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 27 21:04:35.751480 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743801    2577 flags.go:64] FLAG: --fail-swap-on="true"
Jul 27 21:04:35.751480 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743806    2577 flags.go:64] FLAG: --feature-gates=""
Jul 27 21:04:35.751480 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743812    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 27 21:04:35.751480 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743817    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 27 21:04:35.751480 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743822    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 27 21:04:35.751480 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743827    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 27 21:04:35.751480 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743832    2577 flags.go:64] FLAG: --healthz-port="10248"
Jul 27 21:04:35.751480 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743837    2577 flags.go:64] FLAG: --help="false"
Jul 27 21:04:35.751480 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743842    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-142-219.ec2.internal"
Jul 27 21:04:35.751480 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743847    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 27 21:04:35.751480 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743852    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 27 21:04:35.751480 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743856    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 27 21:04:35.751480 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743862    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 27 21:04:35.751480 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743868    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 27 21:04:35.751480 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743873    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 27 21:04:35.751480 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743878    2577 flags.go:64] FLAG: --image-service-endpoint=""
Jul 27 21:04:35.751480 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743882    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 27 21:04:35.751480 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743887    2577 flags.go:64] FLAG: --kube-api-burst="100"
Jul 27 21:04:35.751480 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743891    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 27 21:04:35.751480 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743896    2577 flags.go:64] FLAG: --kube-api-qps="50"
Jul 27 21:04:35.751480 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743901    2577 flags.go:64] FLAG: --kube-reserved=""
Jul 27 21:04:35.752073 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743906    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 27 21:04:35.752073 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743911    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 27 21:04:35.752073 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743916    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 27 21:04:35.752073 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743921    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 27 21:04:35.752073 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743926    2577 flags.go:64] FLAG: --lock-file=""
Jul 27 21:04:35.752073 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743930    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 27 21:04:35.752073 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743935    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 27 21:04:35.752073 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743940    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 27 21:04:35.752073 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743949    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 27 21:04:35.752073 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743954    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 27 21:04:35.752073 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743959    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 27 21:04:35.752073 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743964    2577 flags.go:64] FLAG: --logging-format="text"
Jul 27 21:04:35.752073 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743968    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 27 21:04:35.752073 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743974    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 27 21:04:35.752073 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743979    2577 flags.go:64] FLAG: --manifest-url=""
Jul 27 21:04:35.752073 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743984    2577 flags.go:64] FLAG: --manifest-url-header=""
Jul 27 21:04:35.752073 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743994    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 27 21:04:35.752073 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.743999    2577 flags.go:64] FLAG: --max-open-files="1000000"
Jul 27 21:04:35.752073 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744007    2577 flags.go:64] FLAG: --max-pods="110"
Jul 27 21:04:35.752073 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744011    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 27 21:04:35.752073 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744016    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 27 21:04:35.752073 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744020    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 27 21:04:35.752073 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744025    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 27 21:04:35.752073 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744030    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 27 21:04:35.752073 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744035    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 27 21:04:35.752663 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744040    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 27 21:04:35.752663 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744051    2577 flags.go:64] FLAG: --node-status-max-images="50"
Jul 27 21:04:35.752663 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744056    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 27 21:04:35.752663 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744060    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 27 21:04:35.752663 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744065    2577 flags.go:64] FLAG: --pod-cidr=""
Jul 27 21:04:35.752663 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744069    2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 27 21:04:35.752663 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744077    2577 flags.go:64] FLAG: --pod-manifest-path=""
Jul 27 21:04:35.752663 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744082    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 27 21:04:35.752663 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744087    2577 flags.go:64] FLAG: --pods-per-core="0"
Jul 27 21:04:35.752663 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744092    2577 flags.go:64] FLAG: --port="10250"
Jul 27 21:04:35.752663 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744097    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 27 21:04:35.752663 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744101    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0959cfb740d37e5ac"
Jul 27 21:04:35.752663 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744107    2577 flags.go:64] FLAG: --qos-reserved=""
Jul 27 21:04:35.752663 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744112    2577 flags.go:64] FLAG: --read-only-port="10255"
Jul 27 21:04:35.752663 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744116    2577 flags.go:64] FLAG: --register-node="true"
Jul 27 21:04:35.752663 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744121    2577 flags.go:64] FLAG: --register-with-taints=""
Jul 27 21:04:35.752663 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744127    2577 flags.go:64] FLAG: --registry-burst="10"
Jul 27 21:04:35.752663 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744132    2577 flags.go:64] FLAG: --registry-qps="5"
Jul 27 21:04:35.752663 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744137    2577 flags.go:64] FLAG: --reserved-cpus=""
Jul 27 21:04:35.752663 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744141    2577 flags.go:64] FLAG: --reserved-memory=""
Jul 27 21:04:35.752663 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744148    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 27 21:04:35.752663 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744153    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 27 21:04:35.752663 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744158    2577 flags.go:64] FLAG: --rotate-certificates="false"
Jul 27 21:04:35.752663 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744162    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 27 21:04:35.752663 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744167    2577 flags.go:64] FLAG: --runonce="false"
Jul 27 21:04:35.753244 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744172    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 27 21:04:35.753244 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744177    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 27 21:04:35.753244 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744182    2577 flags.go:64] FLAG: --seccomp-default="false"
Jul 27 21:04:35.753244 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744186    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 27 21:04:35.753244 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744191    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 27 21:04:35.753244 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744196    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 27 21:04:35.753244 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744201    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 27 21:04:35.753244 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744206    2577 flags.go:64] FLAG: --storage-driver-password="root"
Jul 27 21:04:35.753244 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744210    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 27 21:04:35.753244 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744215    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 27 21:04:35.753244 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744219    2577 flags.go:64] FLAG: --storage-driver-user="root"
Jul 27 21:04:35.753244 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744224    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 27 21:04:35.753244 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744229    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 27 21:04:35.753244 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744234    2577 flags.go:64] FLAG: --system-cgroups=""
Jul 27 21:04:35.753244 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744239    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 27 21:04:35.753244 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744248    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 27 21:04:35.753244 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744253    2577 flags.go:64] FLAG: --tls-cert-file=""
Jul 27 21:04:35.753244 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744257    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 27 21:04:35.753244 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744264    2577 flags.go:64] FLAG: --tls-min-version=""
Jul 27 21:04:35.753244 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744269    2577 flags.go:64] FLAG: --tls-private-key-file=""
Jul 27 21:04:35.753244 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744274    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 27 21:04:35.753244 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744278    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 27 21:04:35.753244 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744283    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 27 21:04:35.753244 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744288    2577 flags.go:64] FLAG: --v="2"
Jul 27 21:04:35.753244 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744294    2577 flags.go:64] FLAG: --version="false"
Jul 27 21:04:35.753847 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744301    2577 flags.go:64] FLAG: --vmodule=""
Jul 27 21:04:35.753847 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744308    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 27 21:04:35.753847 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.744313    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 27 21:04:35.753847 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744525    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 21:04:35.753847 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744533    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 21:04:35.753847 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744538    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 21:04:35.753847 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744542    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 21:04:35.753847 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744549    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 21:04:35.753847 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744554    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 21:04:35.753847 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744559    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 21:04:35.753847 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744564    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 21:04:35.753847 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744568    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 21:04:35.753847 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744572    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 21:04:35.753847 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744575    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 21:04:35.753847 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744581    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 21:04:35.753847 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744587    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 21:04:35.753847 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744591    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 21:04:35.753847 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744595    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 21:04:35.753847 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744600    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 21:04:35.754320 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744604    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 21:04:35.754320 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744609    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 21:04:35.754320 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744613    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 21:04:35.754320 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744618    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 21:04:35.754320 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744622    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 21:04:35.754320 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744626    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 21:04:35.754320 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744630    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 21:04:35.754320 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744634    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 21:04:35.754320 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744638    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 21:04:35.754320 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744642    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 21:04:35.754320 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744646    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 21:04:35.754320 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744650    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 21:04:35.754320 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744655    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 21:04:35.754320 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744659    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 21:04:35.754320 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744664    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 21:04:35.754320 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744670    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 21:04:35.754320 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744674    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 21:04:35.754320 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744678    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 21:04:35.754320 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744683    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 21:04:35.754320 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744687    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 21:04:35.754829 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744693    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 21:04:35.754829 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744697    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 21:04:35.754829 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744702    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 21:04:35.754829 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744706    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 21:04:35.754829 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744710    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 21:04:35.754829 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744714    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 21:04:35.754829 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744718    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 21:04:35.754829 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744722    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 21:04:35.754829 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744727    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 21:04:35.754829 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744731    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 21:04:35.754829 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744736    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 21:04:35.754829 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744740    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 21:04:35.754829 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744744    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 21:04:35.754829 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744751    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 21:04:35.754829 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744756    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 21:04:35.754829 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744761    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 21:04:35.754829 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744765    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 21:04:35.754829 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744770    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 21:04:35.754829 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744774    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 21:04:35.754829 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744778    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 21:04:35.755343 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744783    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 21:04:35.755343 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744786    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 21:04:35.755343 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744791    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 21:04:35.755343 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744794    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 21:04:35.755343 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744799    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 21:04:35.755343 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744803    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 21:04:35.755343 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744808    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 21:04:35.755343 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744815    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 21:04:35.755343 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744819    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 21:04:35.755343 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744824    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 21:04:35.755343 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744828    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 21:04:35.755343 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744832    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 21:04:35.755343 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744838    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 21:04:35.755343 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744842    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 21:04:35.755343 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744846    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 21:04:35.755343 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744850    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 21:04:35.755343 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744855    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 21:04:35.755343 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744859    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 21:04:35.755343 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744864    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 21:04:35.755343 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744868    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 21:04:35.755891 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744872    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 21:04:35.755891 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744876    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 21:04:35.755891 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744880    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 21:04:35.755891 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744884    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 21:04:35.755891 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744888    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 21:04:35.755891 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744893    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 21:04:35.755891 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744897    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 21:04:35.755891 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744901    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 21:04:35.755891 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744905    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 21:04:35.755891 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744909    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 21:04:35.755891 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744913    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 21:04:35.755891 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744917    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 21:04:35.755891 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744921    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 21:04:35.755891 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744925    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 21:04:35.755891 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744930    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 21:04:35.755891 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744934    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 21:04:35.755891 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744938    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 21:04:35.755891 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744942    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 21:04:35.755891 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744946    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 21:04:35.755891 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744952    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 21:04:35.756397 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744957    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 21:04:35.756397 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744962    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 21:04:35.756397 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.744966    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 21:04:35.756397 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.745513    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 21:04:35.756397 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.752402    2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 27 21:04:35.756397 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.752417    2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 27 21:04:35.756397 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.752437    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 27 21:04:35.756397 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.752443    2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 27 21:04:35.756397 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752512    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 21:04:35.756397 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752517    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 21:04:35.756397 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752520    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 21:04:35.756397 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752523    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 21:04:35.756397 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752528    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 21:04:35.756397 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752532    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 21:04:35.756397 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752534    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 21:04:35.756780 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752537    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 21:04:35.756780 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752540    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 21:04:35.756780 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752542    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 21:04:35.756780 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752545    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 21:04:35.756780 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752547    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 21:04:35.756780 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752550    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 21:04:35.756780 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752553    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 21:04:35.756780 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752556    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 21:04:35.756780 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752558    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 21:04:35.756780 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752561    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 21:04:35.756780 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752563    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 21:04:35.756780 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752565    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 21:04:35.756780 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752568    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 21:04:35.756780 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752571    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 21:04:35.756780 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752573    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 21:04:35.756780 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752576    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 21:04:35.756780 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752578    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 21:04:35.756780 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752581    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 21:04:35.756780 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752584    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 21:04:35.756780 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752586    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 21:04:35.757246 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752589    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 21:04:35.757246 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752592    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 21:04:35.757246 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752596    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 21:04:35.757246 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752598    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 21:04:35.757246 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752601    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 21:04:35.757246 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752603    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 21:04:35.757246 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752606    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 21:04:35.757246 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752609    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 21:04:35.757246 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752611    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 21:04:35.757246 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752614    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 21:04:35.757246 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752617    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 21:04:35.757246 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752619    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 21:04:35.757246 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752621    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 21:04:35.757246 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752624    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 21:04:35.757246 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752626    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 21:04:35.757246 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752629    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 21:04:35.757246 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752632    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 21:04:35.757246 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752634    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 21:04:35.757246 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752637    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 21:04:35.757246 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752640    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 21:04:35.758076 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752643    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 21:04:35.758076 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752645    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 21:04:35.758076 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752648    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 21:04:35.758076 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752650    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 21:04:35.758076 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752653    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 21:04:35.758076 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752655    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 21:04:35.758076 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752657    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 21:04:35.758076 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752660    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 21:04:35.758076 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752663    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 21:04:35.758076 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752666    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 21:04:35.758076 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752668    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 21:04:35.758076 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752671    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 21:04:35.758076 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752674    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 21:04:35.758076 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752677    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 21:04:35.758076 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752680    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 21:04:35.758076 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752683    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 21:04:35.758076 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752686    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 21:04:35.758076 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752688    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 21:04:35.758076 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752691    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 21:04:35.758076 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752694    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 21:04:35.758612 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752696    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 21:04:35.758612 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752699    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 21:04:35.758612 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752701    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 21:04:35.758612 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752703    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 21:04:35.758612 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752706    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 21:04:35.758612 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752709    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 21:04:35.758612 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752711    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 21:04:35.758612 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752714    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 21:04:35.758612 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752716    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 21:04:35.758612 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752719    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 21:04:35.758612 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752721    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 21:04:35.758612 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752724    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 21:04:35.758612 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752728    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 21:04:35.758612 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752731    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 21:04:35.758612 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752734    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 21:04:35.758612 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752737    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 21:04:35.758612 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752739    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 21:04:35.758612 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752742    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 21:04:35.758612 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752745    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 21:04:35.758612 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752747    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 21:04:35.759129 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752750    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 21:04:35.759129 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752753    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 21:04:35.759129 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752755    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 21:04:35.759129 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752758    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 21:04:35.759129 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752761    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 21:04:35.759129 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752764    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 21:04:35.759129 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752767    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 21:04:35.759129 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752769    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 21:04:35.759129 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752772    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 21:04:35.759129 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752775    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 21:04:35.759129 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752779    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 21:04:35.759129 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752782    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 21:04:35.759129 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.752787    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 21:04:35.759129 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752901    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 21:04:35.759129 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752907    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 21:04:35.759514 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752910    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 21:04:35.759514 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752913    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 21:04:35.759514 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752915    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 21:04:35.759514 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752918    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 21:04:35.759514 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752921    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 21:04:35.759514 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752924    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 21:04:35.759514 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752926    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 21:04:35.759514 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752929    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 21:04:35.759514 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752931    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 21:04:35.759514 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752934    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 21:04:35.759514 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752936    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 21:04:35.759514 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752939    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 21:04:35.759514 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752941    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 21:04:35.759514 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752943    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 21:04:35.759514 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752946    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 21:04:35.759514 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752948    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 21:04:35.759514 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752950    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 21:04:35.759514 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752953    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 21:04:35.759514 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752955    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 21:04:35.759514 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752958    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 21:04:35.760000 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752961    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 21:04:35.760000 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752963    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 21:04:35.760000 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752966    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 21:04:35.760000 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752968    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 21:04:35.760000 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752972    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 21:04:35.760000 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752975    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 21:04:35.760000 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752979    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 21:04:35.760000 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752982    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 21:04:35.760000 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752985    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 21:04:35.760000 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752987    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 21:04:35.760000 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752990    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 21:04:35.760000 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752993    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 21:04:35.760000 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752996    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 21:04:35.760000 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.752998    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 21:04:35.760000 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753001    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 21:04:35.760000 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753003    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 21:04:35.760000 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753006    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 21:04:35.760000 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753008    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 21:04:35.760000 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753011    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 21:04:35.760000 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753013    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 21:04:35.760506 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753015    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 21:04:35.760506 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753017    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 21:04:35.760506 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753021    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 21:04:35.760506 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753024    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 21:04:35.760506 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753027    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 21:04:35.760506 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753030    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 21:04:35.760506 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753033    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 21:04:35.760506 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753035    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 21:04:35.760506 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753038    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 21:04:35.760506 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753041    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 21:04:35.760506 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753043    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 21:04:35.760506 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753046    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 21:04:35.760506 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753048    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 21:04:35.760506 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753051    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 21:04:35.760506 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753053    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 21:04:35.760506 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753056    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 21:04:35.760506 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753058    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 21:04:35.760506 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753061    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 21:04:35.760506 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753063    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 21:04:35.760971 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753066    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 21:04:35.760971 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753068    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 21:04:35.760971 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753071    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 21:04:35.760971 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753073    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 21:04:35.760971 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753076    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 21:04:35.760971 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753078    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 21:04:35.760971 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753081    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 21:04:35.760971 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753083    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 21:04:35.760971 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753086    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 21:04:35.760971 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753088    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 21:04:35.760971 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753091    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 21:04:35.760971 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753093    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 21:04:35.760971 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753096    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 21:04:35.760971 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753098    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 21:04:35.760971 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753101    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 21:04:35.760971 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753103    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 21:04:35.760971 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753106    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 21:04:35.760971 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753108    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 21:04:35.760971 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753110    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 21:04:35.760971 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753113    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 21:04:35.761442 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753115    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 21:04:35.761442 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753118    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 21:04:35.761442 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753120    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 21:04:35.761442 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753122    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 21:04:35.761442 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753125    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 21:04:35.761442 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753128    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 21:04:35.761442 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753132    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 21:04:35.761442 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753134    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 21:04:35.761442 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753137    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 21:04:35.761442 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753140    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 21:04:35.761442 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753142    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 21:04:35.761442 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753145    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 21:04:35.761442 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753147    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 21:04:35.761442 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753150    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 21:04:35.761442 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753153    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 21:04:35.761442 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753156    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 21:04:35.761442 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753173    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 21:04:35.761442 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:35.753176    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 21:04:35.761997 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.753181    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 21:04:35.761997 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.754085    2577 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 27 21:04:35.761997 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.755994    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 27 21:04:35.761997 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.757188    2577 server.go:1019] "Starting client certificate rotation"
Jul 27 21:04:35.761997 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.757284    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 27 21:04:35.761997 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.757321    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 27 21:04:35.761997 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.759449    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 27 21:04:35.762358 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.762341    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 27 21:04:35.773090 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.773062    2577 log.go:25] "Validated CRI v1 runtime API"
Jul 27 21:04:35.775144 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.775128    2577 log.go:25] "Validated CRI v1 image API"
Jul 27 21:04:35.775938 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.775915    2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 27 21:04:35.778400 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.778381    2577 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8576fe9a-06f1-49d2-8829-a6fae5c911af:/dev/nvme0n1p4 fe1ee59c-981a-4ddc-8bb9-f48e47b153a5:/dev/nvme0n1p3]
Jul 27 21:04:35.778455 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.778400    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 27 21:04:35.783632 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.783520    2577 manager.go:216] Machine: {Timestamp:2026-07-27 21:04:35.782213785 +0000 UTC m=+0.316698222 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3168225 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2de741010be334e70690ad505b64ea SystemUUID:ec2de741-010b-e334-e706-90ad505b64ea BootID:eebded14-b850-4d8b-8966-db92bef5516a Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:24:b6:bf:fb:7f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:24:b6:bf:fb:7f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d6:f7:6d:36:ec:af Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 27 21:04:35.783632 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.783627    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 27 21:04:35.783741 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.783708    2577 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 27 21:04:35.786836 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.786808    2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 27 21:04:35.787000 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.786838    2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-219.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 27 21:04:35.787562 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.787542    2577 topology_manager.go:138] "Creating topology manager with none policy"
Jul 27 21:04:35.787562 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.787566    2577 container_manager_linux.go:309] "Creating device plugin manager"
Jul 27 21:04:35.787632 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.787573    2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 27 21:04:35.788212 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.788192    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 27 21:04:35.788429 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.788417    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 27 21:04:35.789268 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.789257    2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 27 21:04:35.791950 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.791938    2577 state_mem.go:36] "Initialized new in-memory state store"
Jul 27 21:04:35.792072 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.792063    2577 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 27 21:04:35.792106 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.792092    2577 kubelet.go:486] "Attempting to sync node with API server"
Jul 27 21:04:35.792136 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.792110    2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 27 21:04:35.792136 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.792131    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 27 21:04:35.792190 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.792140    2577 kubelet.go:398] "Adding apiserver pod source"
Jul 27 21:04:35.792190 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.792149    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 27 21:04:35.792502 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.792491    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 27 21:04:35.792546 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.792508    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 27 21:04:35.799591 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.799576    2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 27 21:04:35.801410 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.801397    2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 27 21:04:35.801453 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.801426    2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 27 21:04:35.802812 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.802797    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 27 21:04:35.802859 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.802832    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 27 21:04:35.802859 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.802839    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 27 21:04:35.802859 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.802844    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 27 21:04:35.802859 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.802852    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 27 21:04:35.802859 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.802857    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 27 21:04:35.802988 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.802865    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 27 21:04:35.802988 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.802870    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 27 21:04:35.802988 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.802877    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 27 21:04:35.802988 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.802882    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 27 21:04:35.802988 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.802895    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 27 21:04:35.802988 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.802900    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 27 21:04:35.803279 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:35.803259    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-219.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 27 21:04:35.803313 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:35.803260    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 27 21:04:35.803572 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.803562    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 27 21:04:35.803607 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.803574    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 27 21:04:35.805526 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.805507    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fwrqp"
Jul 27 21:04:35.806820 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.806805    2577 server.go:1268] "Started kubelet"
Jul 27 21:04:35.806893 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.806862    2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 27 21:04:35.807326 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.807186    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 27 21:04:35.807508 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.807488    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 27 21:04:35.807876 ip-10-0-142-219 systemd[1]: Started Kubernetes Kubelet.
Jul 27 21:04:35.811628 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.811601    2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 27 21:04:35.812696 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.812665    2577 server.go:310] "Adding debug handlers to kubelet server"
Jul 27 21:04:35.814091 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.814066    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fwrqp"
Jul 27 21:04:35.814566 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:35.814548    2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 27 21:04:35.817644 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.817626    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-219.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 27 21:04:35.818461 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:35.817552    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-219.ec2.internal.18c64175a35c8a77  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-219.ec2.internal,UID:ip-10-0-142-219.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-219.ec2.internal,},FirstTimestamp:2026-07-27 21:04:35.806710391 +0000 UTC m=+0.341194833,LastTimestamp:2026-07-27 21:04:35.806710391 +0000 UTC m=+0.341194833,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-219.ec2.internal,}"
Jul 27 21:04:35.819445 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.819425    2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 27 21:04:35.819445 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.819435    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 27 21:04:35.819664 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:35.819548    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-219.ec2.internal\" not found"
Jul 27 21:04:35.819998 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.819982    2577 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 27 21:04:35.819998 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.820000    2577 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 27 21:04:35.820152 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.819984    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 27 21:04:35.820152 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.820089    2577 reconstruct.go:97] "Volume reconstruction finished"
Jul 27 21:04:35.820152 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.820096    2577 reconciler.go:29] "Reconciler: start to sync state"
Jul 27 21:04:35.820401 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.820383    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 27 21:04:35.820401 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.820400    2577 factory.go:55] Registering systemd factory
Jul 27 21:04:35.820557 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.820408    2577 factory.go:223] Registration of the systemd container factory successfully
Jul 27 21:04:35.820670 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.820656    2577 factory.go:153] Registering CRI-O factory
Jul 27 21:04:35.820706 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.820675    2577 factory.go:223] Registration of the crio container factory successfully
Jul 27 21:04:35.820706 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.820700    2577 factory.go:103] Registering Raw factory
Jul 27 21:04:35.820760 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.820714    2577 manager.go:1196] Started watching for new ooms in manager
Jul 27 21:04:35.821228 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.821216    2577 manager.go:349] Starting recovery of all containers
Jul 27 21:04:35.821332 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:35.821311    2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-219.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 27 21:04:35.821417 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:35.821398    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 27 21:04:35.830160 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.830142    2577 manager.go:354] Recovery completed
Jul 27 21:04:35.835428 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.835415    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 21:04:35.837207 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.837190    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-219.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 21:04:35.837258 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.837225    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-219.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 21:04:35.837258 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.837239    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-219.ec2.internal" event="NodeHasSufficientPID"
Jul 27 21:04:35.838308 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.838293    2577 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 27 21:04:35.838308 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.838306    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 21:04:35.838413 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.838306    2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 27 21:04:35.838413 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.838384    2577 state_mem.go:36] "Initialized new in-memory state store"
Jul 27 21:04:35.838962 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.838949    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-219.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 21:04:35.839013 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.838973    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-219.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 21:04:35.839013 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.838985    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-219.ec2.internal" event="NodeHasSufficientPID"
Jul 27 21:04:35.840976 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.840962    2577 policy_none.go:49] "None policy: Start"
Jul 27 21:04:35.840976 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.840978    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 27 21:04:35.841072 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.840988    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 27 21:04:35.842927 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.842916    2577 policy_none.go:47] "Start"
Jul 27 21:04:35.876033 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.875865    2577 manager.go:337] "Starting Device Plugin manager"
Jul 27 21:04:35.876118 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:35.876062    2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 27 21:04:35.876118 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.876076    2577 server.go:87] "Starting device plugin registration server"
Jul 27 21:04:35.876598 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.876585    2577 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 27 21:04:35.876630 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.876603    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 27 21:04:35.876729 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.876712    2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 27 21:04:35.877079 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.877053    2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 27 21:04:35.877079 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:35.877069    2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 27 21:04:35.877320 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:35.877302    2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 27 21:04:35.877378 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:35.877358    2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-219.ec2.internal\" not found"
Jul 27 21:04:37.128438 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:37.128406    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 27 21:04:37.129636 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:37.129615    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 27 21:04:37.129695 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:37.129665    2577 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 27 21:04:37.129695 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:37.129694    2577 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 27 21:04:37.129794 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:37.129738    2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 27 21:04:37.132004 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:37.131980    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 21:04:37.144749 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:37.144723    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 21:04:37.145690 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:37.145675    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-219.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 21:04:37.145754 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:37.145714    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-219.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 21:04:37.145754 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:37.145725    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-219.ec2.internal" event="NodeHasSufficientPID"
Jul 27 21:04:37.145754 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:37.145748    2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-219.ec2.internal"
Jul 27 21:04:37.156090 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:37.156047    2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-219.ec2.internal"
Jul 27 21:04:37.156090 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:37.156071    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-219.ec2.internal\": node \"ip-10-0-142-219.ec2.internal\" not found"
Jul 27 21:04:37.176392 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:37.176361    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-219.ec2.internal\" not found"
Jul 27 21:04:37.230658 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:37.230616    2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-219.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-219.ec2.internal"]
Jul 27 21:04:37.230745 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:37.230702    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 21:04:37.232583 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:37.232568    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-219.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 21:04:37.232656 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:37.232599    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-219.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 21:04:37.232656 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:37.232616    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-219.ec2.internal" event="NodeHasSufficientPID"
Jul 27 21:04:37.236136 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:37.236123    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 21:04:37.236255 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:37.236241    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-219.ec2.internal"
Jul 27 21:04:37.236290 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:37.236283    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 21:04:37.236894 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:37.236880    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-219.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 21:04:37.236944 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:37.236907    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-219.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 21:04:37.236944 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:37.236918    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-219.ec2.internal" event="NodeHasSufficientPID"
Jul 27 21:04:37.237013 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:37.236880    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-219.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 21:04:37.237013 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:37.236996    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-219.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 21:04:37.237082 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:37.237017    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-219.ec2.internal" event="NodeHasSufficientPID"
Jul 27 21:04:37.238997 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:37.238979    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-219.ec2.internal"
Jul 27 21:04:37.239061 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:37.239015    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 21:04:37.239982 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:37.239967    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-219.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 21:04:37.240048 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:37.239993    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-219.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 21:04:37.240048 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:37.240002    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-219.ec2.internal" event="NodeHasSufficientPID"
Jul 27 21:04:37.263488 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:37.263463    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-219.ec2.internal\" not found" node="ip-10-0-142-219.ec2.internal"
Jul 27 21:04:37.268093 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:37.268076    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-219.ec2.internal\" not found" node="ip-10-0-142-219.ec2.internal"
Jul 27 21:04:37.277103 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:37.277088    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-219.ec2.internal\" not found"
Jul 27 21:04:37.288868 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:37.288839    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7fe2fc32304c33ef13ea6c3a49bb44af-config\") pod \"kube-apiserver-proxy-ip-10-0-142-219.ec2.internal\" (UID: \"7fe2fc32304c33ef13ea6c3a49bb44af\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-219.ec2.internal"
Jul 27 21:04:37.288976 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:37.288872    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/abb843a58c4a692b2202031f20068533-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-219.ec2.internal\" (UID: \"abb843a58c4a692b2202031f20068533\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-219.ec2.internal"
Jul 27 21:04:37.288976 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:37.288900    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/abb843a58c4a692b2202031f20068533-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-219.ec2.internal\" (UID: \"abb843a58c4a692b2202031f20068533\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-219.ec2.internal"
Jul 27 21:04:37.378146 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:37.378115    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-219.ec2.internal\" not found"
Jul 27 21:04:37.389816 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:37.389786    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7fe2fc32304c33ef13ea6c3a49bb44af-config\") pod \"kube-apiserver-proxy-ip-10-0-142-219.ec2.internal\" (UID: \"7fe2fc32304c33ef13ea6c3a49bb44af\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-219.ec2.internal"
Jul 27 21:04:37.389816 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:37.389816    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/abb843a58c4a692b2202031f20068533-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-219.ec2.internal\" (UID: \"abb843a58c4a692b2202031f20068533\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-219.ec2.internal"
Jul 27 21:04:37.389990 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:37.389839    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/abb843a58c4a692b2202031f20068533-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-219.ec2.internal\" (UID: \"abb843a58c4a692b2202031f20068533\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-219.ec2.internal"
Jul 27 21:04:37.389990 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:37.389883    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/abb843a58c4a692b2202031f20068533-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-219.ec2.internal\" (UID: \"abb843a58c4a692b2202031f20068533\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-219.ec2.internal"
Jul 27 21:04:37.389990 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:37.389902    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/abb843a58c4a692b2202031f20068533-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-219.ec2.internal\" (UID: \"abb843a58c4a692b2202031f20068533\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-219.ec2.internal"
Jul 27 21:04:37.389990 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:37.389903    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7fe2fc32304c33ef13ea6c3a49bb44af-config\") pod \"kube-apiserver-proxy-ip-10-0-142-219.ec2.internal\" (UID: \"7fe2fc32304c33ef13ea6c3a49bb44af\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-219.ec2.internal"
Jul 27 21:04:37.478255 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:37.478175    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-219.ec2.internal\" not found"
Jul 27 21:04:37.566788 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:37.566744    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-219.ec2.internal"
Jul 27 21:04:37.573440 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:37.573422    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-219.ec2.internal"
Jul 27 21:04:37.579258 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:37.579242    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-219.ec2.internal\" not found"
Jul 27 21:04:37.679919 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:37.679883    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-219.ec2.internal\" not found"
Jul 27 21:04:37.780513 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:37.780438    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-219.ec2.internal\" not found"
Jul 27 21:04:37.881000 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:37.880966    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-219.ec2.internal\" not found"
Jul 27 21:04:37.924511 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:37.924484    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 27 21:04:37.925063 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:37.924685    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 27 21:04:37.946538 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:37.946507    2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 21:04:37.982019 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:37.981987    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-219.ec2.internal\" not found"
Jul 27 21:04:37.987340 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:37.987294    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-26 20:59:36 +0000 UTC" deadline="2028-01-26 10:31:26.668323634 +0000 UTC"
Jul 27 21:04:37.987398 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:37.987340    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13141h26m48.680986308s"
Jul 27 21:04:38.082716 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:38.082641    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-219.ec2.internal\" not found"
Jul 27 21:04:38.183420 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:38.183389    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-219.ec2.internal\" not found"
Jul 27 21:04:38.252256 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:38.252221    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabb843a58c4a692b2202031f20068533.slice/crio-1c45b08b10809a49649f44e41397b8b1d6725c9d89042b655f3640d274f41b88 WatchSource:0}: Error finding container 1c45b08b10809a49649f44e41397b8b1d6725c9d89042b655f3640d274f41b88: Status 404 returned error can't find the container with id 1c45b08b10809a49649f44e41397b8b1d6725c9d89042b655f3640d274f41b88
Jul 27 21:04:38.252523 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:38.252499    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fe2fc32304c33ef13ea6c3a49bb44af.slice/crio-686ade05c79b2844ae7d5f87f13306e9e3b1faacffa2980fa5f787c2701e7eb4 WatchSource:0}: Error finding container 686ade05c79b2844ae7d5f87f13306e9e3b1faacffa2980fa5f787c2701e7eb4: Status 404 returned error can't find the container with id 686ade05c79b2844ae7d5f87f13306e9e3b1faacffa2980fa5f787c2701e7eb4
Jul 27 21:04:38.257864 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.257848    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 21:04:38.284036 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:38.284008    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-219.ec2.internal\" not found"
Jul 27 21:04:38.373861 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.373798    2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 21:04:38.384767 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:38.384738    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-219.ec2.internal\" not found"
Jul 27 21:04:38.484913 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:38.484883    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-219.ec2.internal\" not found"
Jul 27 21:04:38.484913 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.484903    2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 21:04:38.487808 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.487793    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-219.ec2.internal"
Jul 27 21:04:38.497952 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.497933    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 27 21:04:38.498930 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.498918    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-219.ec2.internal"
Jul 27 21:04:38.507745 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.507707    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 27 21:04:38.960528 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.960497    2577 apiserver.go:52] "Watching apiserver"
Jul 27 21:04:38.968826 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.968799    2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 27 21:04:38.969277 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.969248    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-b9b6s","kube-system/kube-apiserver-proxy-ip-10-0-142-219.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d68g","openshift-cluster-node-tuning-operator/tuned-62zwj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-219.ec2.internal","openshift-multus/network-metrics-daemon-wb7n2","openshift-network-diagnostics/network-check-target-nj9np","openshift-dns/node-resolver-frmv7","openshift-image-registry/node-ca-dtklb","openshift-multus/multus-additional-cni-plugins-667ww","openshift-multus/multus-xgbjk","openshift-network-operator/iptables-alerter-fdrp9","openshift-ovn-kubernetes/ovnkube-node-pftzz","kube-system/global-pull-secret-syncer-5d7vf"]
Jul 27 21:04:38.971828 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.971776    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nj9np"
Jul 27 21:04:38.971938 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:38.971858    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nj9np" podUID="c5915867-c603-4d51-8417-42aada6603ef"
Jul 27 21:04:38.973966 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.973943    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xgbjk"
Jul 27 21:04:38.977789 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.976807    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 27 21:04:38.977789 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.977064    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 27 21:04:38.977789 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.977328    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 27 21:04:38.977789 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.977614    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-5z67m\""
Jul 27 21:04:38.978031 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.977889    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 27 21:04:38.978283 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.976097    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d68g"
Jul 27 21:04:38.979389 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.979373    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-62zwj"
Jul 27 21:04:38.980787 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.980752    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5d7vf"
Jul 27 21:04:38.980887 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:38.980845    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5d7vf" podUID="a4af1379-e40a-471d-a011-d50ab76b7f51"
Jul 27 21:04:38.981450 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.981426    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 27 21:04:38.981544 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.981453    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 27 21:04:38.981632 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.981606    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 27 21:04:38.981690 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.981640    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4nkch\""
Jul 27 21:04:38.982567 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.982431    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8qv7t\""
Jul 27 21:04:38.982675 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.982628    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 27 21:04:38.982902 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.982876    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 27 21:04:38.983500 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.983473    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb7n2"
Jul 27 21:04:38.983582 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:38.983558    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wb7n2" podUID="3af7400b-f39f-4f41-a492-586171130c30"
Jul 27 21:04:38.985596 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.985576    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dtklb"
Jul 27 21:04:38.987930 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.987909    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 27 21:04:38.988253 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.988232    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:38.989168 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.989152    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2tz4d\""
Jul 27 21:04:38.989386 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.989367    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 27 21:04:38.989612 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.989594    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 27 21:04:38.989702 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.989632    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 27 21:04:38.990547 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.990293    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-frmv7"
Jul 27 21:04:38.991224 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.991090    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 27 21:04:38.991224 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.991163    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 27 21:04:38.991358 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.991238    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 27 21:04:38.991358 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.991272    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 27 21:04:38.991459 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.991351    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 27 21:04:38.991510 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.991484    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-w7zbc\""
Jul 27 21:04:38.991925 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.991909    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 27 21:04:38.992640 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.992622    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hj6cq\""
Jul 27 21:04:38.992926 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.992892    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 27 21:04:38.993088 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.993072    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 27 21:04:38.993290 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.993274    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-667ww"
Jul 27 21:04:38.995471 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.995453    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-b9b6s"
Jul 27 21:04:38.995590 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.995573    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 27 21:04:38.995654 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.995615    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 27 21:04:38.995923 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.995905    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dtzg5\""
Jul 27 21:04:38.997684 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.997660    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 27 21:04:38.997955 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.997933    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 27 21:04:38.998134 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.998119    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7hkr2\""
Jul 27 21:04:38.998288 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.998268    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fdrp9"
Jul 27 21:04:38.999584 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.999261    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72112606-3839-4478-bac9-dfc58a7809f8-host-run-ovn-kubernetes\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:38.999584 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.999292    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/72112606-3839-4478-bac9-dfc58a7809f8-ovnkube-config\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:38.999584 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.999314    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c-os-release\") pod \"multus-xgbjk\" (UID: \"7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c\") " pod="openshift-multus/multus-xgbjk"
Jul 27 21:04:38.999584 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.999331    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab-etc-tuned\") pod \"tuned-62zwj\" (UID: \"02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab\") " pod="openshift-cluster-node-tuning-operator/tuned-62zwj"
Jul 27 21:04:38.999584 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.999353    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a4af1379-e40a-471d-a011-d50ab76b7f51-kubelet-config\") pod \"global-pull-secret-syncer-5d7vf\" (UID: \"a4af1379-e40a-471d-a011-d50ab76b7f51\") " pod="kube-system/global-pull-secret-syncer-5d7vf"
Jul 27 21:04:38.999584 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.999366    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/72112606-3839-4478-bac9-dfc58a7809f8-host-kubelet\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:38.999584 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.999379    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/72112606-3839-4478-bac9-dfc58a7809f8-log-socket\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:38.999584 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.999392    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c-system-cni-dir\") pod \"multus-xgbjk\" (UID: \"7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c\") " pod="openshift-multus/multus-xgbjk"
Jul 27 21:04:38.999584 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.999406    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c-multus-socket-dir-parent\") pod \"multus-xgbjk\" (UID: \"7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c\") " pod="openshift-multus/multus-xgbjk"
Jul 27 21:04:38.999584 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.999422    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c-multus-conf-dir\") pod \"multus-xgbjk\" (UID: \"7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c\") " pod="openshift-multus/multus-xgbjk"
Jul 27 21:04:38.999584 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.999445    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72112606-3839-4478-bac9-dfc58a7809f8-etc-openvswitch\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:38.999584 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.999472    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b2bf895f-3920-43a8-977c-fffbd75796e4-etc-selinux\") pod \"aws-ebs-csi-driver-node-9d68g\" (UID: \"b2bf895f-3920-43a8-977c-fffbd75796e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d68g"
Jul 27 21:04:38.999584 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.999486    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab-etc-sysctl-d\") pod \"tuned-62zwj\" (UID: \"02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab\") " pod="openshift-cluster-node-tuning-operator/tuned-62zwj"
Jul 27 21:04:38.999584 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.999500    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72112606-3839-4478-bac9-dfc58a7809f8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:38.999584 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.999516    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab-etc-systemd\") pod \"tuned-62zwj\" (UID: \"02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab\") " pod="openshift-cluster-node-tuning-operator/tuned-62zwj"
Jul 27 21:04:39.000276 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.999603    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab-host\") pod \"tuned-62zwj\" (UID: \"02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab\") " pod="openshift-cluster-node-tuning-operator/tuned-62zwj"
Jul 27 21:04:39.000276 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.999651    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/72112606-3839-4478-bac9-dfc58a7809f8-host-run-netns\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:39.000276 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.999677    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/72112606-3839-4478-bac9-dfc58a7809f8-ovnkube-script-lib\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:39.000276 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.999701    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c-multus-cni-dir\") pod \"multus-xgbjk\" (UID: \"7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c\") " pod="openshift-multus/multus-xgbjk"
Jul 27 21:04:39.000276 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.999724    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c-etc-kubernetes\") pod \"multus-xgbjk\" (UID: \"7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c\") " pod="openshift-multus/multus-xgbjk"
Jul 27 21:04:39.000276 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.999744    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c-cnibin\") pod \"multus-xgbjk\" (UID: \"7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c\") " pod="openshift-multus/multus-xgbjk"
Jul 27 21:04:39.000276 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.999841    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c-hostroot\") pod \"multus-xgbjk\" (UID: \"7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c\") " pod="openshift-multus/multus-xgbjk"
Jul 27 21:04:39.000276 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.999887    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4sw5\" (UniqueName: \"kubernetes.io/projected/7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c-kube-api-access-j4sw5\") pod \"multus-xgbjk\" (UID: \"7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c\") " pod="openshift-multus/multus-xgbjk"
Jul 27 21:04:39.000276 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.999925    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b2bf895f-3920-43a8-977c-fffbd75796e4-socket-dir\") pod \"aws-ebs-csi-driver-node-9d68g\" (UID: \"b2bf895f-3920-43a8-977c-fffbd75796e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d68g"
Jul 27 21:04:39.000276 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.999951    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmvcb\" (UniqueName: \"kubernetes.io/projected/c5915867-c603-4d51-8417-42aada6603ef-kube-api-access-mmvcb\") pod \"network-check-target-nj9np\" (UID: \"c5915867-c603-4d51-8417-42aada6603ef\") " pod="openshift-network-diagnostics/network-check-target-nj9np"
Jul 27 21:04:39.000276 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:38.999990    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c-host-run-netns\") pod \"multus-xgbjk\" (UID: \"7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c\") " pod="openshift-multus/multus-xgbjk"
Jul 27 21:04:39.000276 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.000106    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c-host-var-lib-cni-multus\") pod \"multus-xgbjk\" (UID: \"7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c\") " pod="openshift-multus/multus-xgbjk"
Jul 27 21:04:39.000276 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.000155    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/72112606-3839-4478-bac9-dfc58a7809f8-systemd-units\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:39.000276 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.000193    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a4af1379-e40a-471d-a011-d50ab76b7f51-original-pull-secret\") pod \"global-pull-secret-syncer-5d7vf\" (UID: \"a4af1379-e40a-471d-a011-d50ab76b7f51\") " pod="kube-system/global-pull-secret-syncer-5d7vf"
Jul 27 21:04:39.000276 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.000242    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0b84c165-5dfa-416c-bec6-28c9d55197ab-serviceca\") pod \"node-ca-dtklb\" (UID: \"0b84c165-5dfa-416c-bec6-28c9d55197ab\") " pod="openshift-image-registry/node-ca-dtklb"
Jul 27 21:04:39.000276 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.000264    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/72112606-3839-4478-bac9-dfc58a7809f8-run-systemd\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:39.000929 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.000299    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab-etc-sysctl-conf\") pod \"tuned-62zwj\" (UID: \"02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab\") " pod="openshift-cluster-node-tuning-operator/tuned-62zwj"
Jul 27 21:04:39.000929 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.000321    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/72112606-3839-4478-bac9-dfc58a7809f8-node-log\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:39.000929 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.000344    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbrbf\" (UniqueName: \"kubernetes.io/projected/b2bf895f-3920-43a8-977c-fffbd75796e4-kube-api-access-nbrbf\") pod \"aws-ebs-csi-driver-node-9d68g\" (UID: \"b2bf895f-3920-43a8-977c-fffbd75796e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d68g"
Jul 27 21:04:39.000929 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.000379    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t9zx\" (UniqueName: \"kubernetes.io/projected/3af7400b-f39f-4f41-a492-586171130c30-kube-api-access-4t9zx\") pod \"network-metrics-daemon-wb7n2\" (UID: \"3af7400b-f39f-4f41-a492-586171130c30\") " pod="openshift-multus/network-metrics-daemon-wb7n2"
Jul 27 21:04:39.000929 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.000414    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab-etc-modprobe-d\") pod \"tuned-62zwj\" (UID: \"02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab\") " pod="openshift-cluster-node-tuning-operator/tuned-62zwj"
Jul 27 21:04:39.000929 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.000444    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab-etc-sysconfig\") pod \"tuned-62zwj\" (UID: \"02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab\") " pod="openshift-cluster-node-tuning-operator/tuned-62zwj"
Jul 27 21:04:39.000929 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.000477    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab-var-lib-kubelet\") pod \"tuned-62zwj\" (UID: \"02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab\") " pod="openshift-cluster-node-tuning-operator/tuned-62zwj"
Jul 27 21:04:39.000929 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.000500    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t6n6\" (UniqueName: \"kubernetes.io/projected/0b84c165-5dfa-416c-bec6-28c9d55197ab-kube-api-access-7t6n6\") pod \"node-ca-dtklb\" (UID: \"0b84c165-5dfa-416c-bec6-28c9d55197ab\") " pod="openshift-image-registry/node-ca-dtklb"
Jul 27 21:04:39.000929 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.000522    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/72112606-3839-4478-bac9-dfc58a7809f8-ovn-node-metrics-cert\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:39.000929 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.000539    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nrb7\" (UniqueName: \"kubernetes.io/projected/72112606-3839-4478-bac9-dfc58a7809f8-kube-api-access-9nrb7\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:39.000929 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.000554    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/72112606-3839-4478-bac9-dfc58a7809f8-host-slash\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:39.000929 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.000556    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-lx9fb\""
Jul 27 21:04:39.000929 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.000574    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3af7400b-f39f-4f41-a492-586171130c30-metrics-certs\") pod \"network-metrics-daemon-wb7n2\" (UID: \"3af7400b-f39f-4f41-a492-586171130c30\") " pod="openshift-multus/network-metrics-daemon-wb7n2"
Jul 27 21:04:39.000929 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.000606    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab-etc-kubernetes\") pod \"tuned-62zwj\" (UID: \"02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab\") " pod="openshift-cluster-node-tuning-operator/tuned-62zwj"
Jul 27 21:04:39.000929 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.000619    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 27 21:04:39.000929 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.000627    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab-sys\") pod \"tuned-62zwj\" (UID: \"02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab\") " pod="openshift-cluster-node-tuning-operator/tuned-62zwj"
Jul 27 21:04:39.000929 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.000647    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab-run\") pod \"tuned-62zwj\" (UID: \"02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab\") " pod="openshift-cluster-node-tuning-operator/tuned-62zwj"
Jul 27 21:04:39.001726 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.000696    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/72112606-3839-4478-bac9-dfc58a7809f8-host-cni-netd\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:39.001726 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.000719    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/72112606-3839-4478-bac9-dfc58a7809f8-env-overrides\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:39.001726 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.000744    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b2bf895f-3920-43a8-977c-fffbd75796e4-sys-fs\") pod \"aws-ebs-csi-driver-node-9d68g\" (UID: \"b2bf895f-3920-43a8-977c-fffbd75796e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d68g"
Jul 27 21:04:39.001726 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.000809    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c-cni-binary-copy\") pod \"multus-xgbjk\" (UID: \"7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c\") " pod="openshift-multus/multus-xgbjk"
Jul 27 21:04:39.001726 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.000831    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c-host-run-k8s-cni-cncf-io\") pod \"multus-xgbjk\" (UID: \"7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c\") " pod="openshift-multus/multus-xgbjk"
Jul 27 21:04:39.001726 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.000858    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b2bf895f-3920-43a8-977c-fffbd75796e4-registration-dir\") pod \"aws-ebs-csi-driver-node-9d68g\" (UID: \"b2bf895f-3920-43a8-977c-fffbd75796e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d68g"
Jul 27 21:04:39.001726 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.000838    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 27 21:04:39.001726 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.000856    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 27 21:04:39.001726 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.000872    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 27 21:04:39.001726 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.000889    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a4af1379-e40a-471d-a011-d50ab76b7f51-dbus\") pod \"global-pull-secret-syncer-5d7vf\" (UID: \"a4af1379-e40a-471d-a011-d50ab76b7f51\") " pod="kube-system/global-pull-secret-syncer-5d7vf"
Jul 27 21:04:39.001726 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.000917    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72112606-3839-4478-bac9-dfc58a7809f8-var-lib-openvswitch\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:39.001726 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.000940    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c-host-var-lib-cni-bin\") pod \"multus-xgbjk\" (UID: \"7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c\") " pod="openshift-multus/multus-xgbjk"
Jul 27 21:04:39.001726 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.000961    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c-host-var-lib-kubelet\") pod \"multus-xgbjk\" (UID: \"7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c\") " pod="openshift-multus/multus-xgbjk"
Jul 27 21:04:39.001726 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.000982    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0b84c165-5dfa-416c-bec6-28c9d55197ab-host\") pod \"node-ca-dtklb\" (UID: \"0b84c165-5dfa-416c-bec6-28c9d55197ab\") " pod="openshift-image-registry/node-ca-dtklb"
Jul 27 21:04:39.001726 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.001004    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b2bf895f-3920-43a8-977c-fffbd75796e4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9d68g\" (UID: \"b2bf895f-3920-43a8-977c-fffbd75796e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d68g"
Jul 27 21:04:39.001726 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.001025    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b2bf895f-3920-43a8-977c-fffbd75796e4-device-dir\") pod \"aws-ebs-csi-driver-node-9d68g\" (UID: \"b2bf895f-3920-43a8-977c-fffbd75796e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d68g"
Jul 27 21:04:39.001726 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.001096    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/72112606-3839-4478-bac9-dfc58a7809f8-host-cni-bin\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:39.001726 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.001135    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab-lib-modules\") pod \"tuned-62zwj\" (UID: \"02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab\") " pod="openshift-cluster-node-tuning-operator/tuned-62zwj"
Jul 27 21:04:39.002597 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.001158    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab-tmp\") pod \"tuned-62zwj\" (UID: \"02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab\") " pod="openshift-cluster-node-tuning-operator/tuned-62zwj"
Jul 27 21:04:39.002597 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.001191    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx4gq\" (UniqueName: \"kubernetes.io/projected/02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab-kube-api-access-nx4gq\") pod \"tuned-62zwj\" (UID: \"02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab\") " pod="openshift-cluster-node-tuning-operator/tuned-62zwj"
Jul 27 21:04:39.002597 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.001227    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72112606-3839-4478-bac9-dfc58a7809f8-run-openvswitch\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:39.002597 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.001260    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c-multus-daemon-config\") pod \"multus-xgbjk\" (UID: \"7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c\") " pod="openshift-multus/multus-xgbjk"
Jul 27 21:04:39.002597 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.001285    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c-host-run-multus-certs\") pod \"multus-xgbjk\" (UID: \"7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c\") " pod="openshift-multus/multus-xgbjk"
Jul 27 21:04:39.002597 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.001325    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/72112606-3839-4478-bac9-dfc58a7809f8-run-ovn\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:39.023973 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.023946    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vtjp8"
Jul 27 21:04:39.035124 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.035101    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vtjp8"
Jul 27 21:04:39.089027 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.088997    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 27 21:04:39.102011 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.101977    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c35bbe72-c030-4b9d-aece-459d907893dc-host-slash\") pod \"iptables-alerter-fdrp9\" (UID: \"c35bbe72-c030-4b9d-aece-459d907893dc\") " pod="openshift-network-operator/iptables-alerter-fdrp9"
Jul 27 21:04:39.102166 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.102027    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t9zx\" (UniqueName: \"kubernetes.io/projected/3af7400b-f39f-4f41-a492-586171130c30-kube-api-access-4t9zx\") pod \"network-metrics-daemon-wb7n2\" (UID: \"3af7400b-f39f-4f41-a492-586171130c30\") " pod="openshift-multus/network-metrics-daemon-wb7n2"
Jul 27 21:04:39.102166 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.102083    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab-etc-modprobe-d\") pod \"tuned-62zwj\" (UID: \"02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab\") " pod="openshift-cluster-node-tuning-operator/tuned-62zwj"
Jul 27 21:04:39.102166 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.102120    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab-etc-sysconfig\") pod \"tuned-62zwj\" (UID: \"02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab\") " pod="openshift-cluster-node-tuning-operator/tuned-62zwj"
Jul 27 21:04:39.102166 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.102147    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab-var-lib-kubelet\") pod \"tuned-62zwj\" (UID: \"02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab\") " pod="openshift-cluster-node-tuning-operator/tuned-62zwj"
Jul 27 21:04:39.102357 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.102169    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t6n6\" (UniqueName: \"kubernetes.io/projected/0b84c165-5dfa-416c-bec6-28c9d55197ab-kube-api-access-7t6n6\") pod \"node-ca-dtklb\" (UID: \"0b84c165-5dfa-416c-bec6-28c9d55197ab\") " pod="openshift-image-registry/node-ca-dtklb"
Jul 27 21:04:39.102357 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.102191    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/72112606-3839-4478-bac9-dfc58a7809f8-ovn-node-metrics-cert\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:39.102357 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.102210    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nrb7\" (UniqueName: \"kubernetes.io/projected/72112606-3839-4478-bac9-dfc58a7809f8-kube-api-access-9nrb7\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:39.102357 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.102237    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fd05d22e-5585-4f39-8597-4fc6978df793-cnibin\") pod \"multus-additional-cni-plugins-667ww\" (UID: \"fd05d22e-5585-4f39-8597-4fc6978df793\") " pod="openshift-multus/multus-additional-cni-plugins-667ww"
Jul 27 21:04:39.102357 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.102244    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab-etc-modprobe-d\") pod \"tuned-62zwj\" (UID: \"02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab\") " pod="openshift-cluster-node-tuning-operator/tuned-62zwj"
Jul 27 21:04:39.102357 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.102263    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/72112606-3839-4478-bac9-dfc58a7809f8-host-slash\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:39.102357 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.102263    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab-var-lib-kubelet\") pod \"tuned-62zwj\" (UID: \"02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab\") " pod="openshift-cluster-node-tuning-operator/tuned-62zwj"
Jul 27 21:04:39.102357 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.102287    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3af7400b-f39f-4f41-a492-586171130c30-metrics-certs\") pod \"network-metrics-daemon-wb7n2\" (UID: \"3af7400b-f39f-4f41-a492-586171130c30\") " pod="openshift-multus/network-metrics-daemon-wb7n2"
Jul 27 21:04:39.102357 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.102308    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab-etc-kubernetes\") pod \"tuned-62zwj\" (UID: \"02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab\") " pod="openshift-cluster-node-tuning-operator/tuned-62zwj"
Jul 27 21:04:39.102357 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.102325    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab-etc-sysconfig\") pod \"tuned-62zwj\" (UID: \"02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab\") " pod="openshift-cluster-node-tuning-operator/tuned-62zwj"
Jul 27 21:04:39.102357 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.102335    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mlv5\" (UniqueName: \"kubernetes.io/projected/fd05d22e-5585-4f39-8597-4fc6978df793-kube-api-access-4mlv5\") pod \"multus-additional-cni-plugins-667ww\" (UID: \"fd05d22e-5585-4f39-8597-4fc6978df793\") " pod="openshift-multus/multus-additional-cni-plugins-667ww"
Jul 27 21:04:39.102357 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.102344    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/72112606-3839-4478-bac9-dfc58a7809f8-host-slash\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:39.102357 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.102359    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr8hf\" (UniqueName: \"kubernetes.io/projected/c35bbe72-c030-4b9d-aece-459d907893dc-kube-api-access-qr8hf\") pod \"iptables-alerter-fdrp9\" (UID: \"c35bbe72-c030-4b9d-aece-459d907893dc\") " pod="openshift-network-operator/iptables-alerter-fdrp9"
Jul 27 21:04:39.102928 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.102385    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab-sys\") pod \"tuned-62zwj\" (UID: \"02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab\") " pod="openshift-cluster-node-tuning-operator/tuned-62zwj"
Jul 27 21:04:39.102928 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.102406    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab-run\") pod \"tuned-62zwj\" (UID: \"02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab\") " pod="openshift-cluster-node-tuning-operator/tuned-62zwj"
Jul 27 21:04:39.102928 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.102426    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/72112606-3839-4478-bac9-dfc58a7809f8-host-cni-netd\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:39.102928 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:39.102426    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:04:39.102928 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.102470    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab-sys\") pod \"tuned-62zwj\" (UID: \"02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab\") " pod="openshift-cluster-node-tuning-operator/tuned-62zwj"
Jul 27 21:04:39.102928 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.102405    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab-etc-kubernetes\") pod \"tuned-62zwj\" (UID: \"02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab\") " pod="openshift-cluster-node-tuning-operator/tuned-62zwj"
Jul 27 21:04:39.102928 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:39.102513    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3af7400b-f39f-4f41-a492-586171130c30-metrics-certs podName:3af7400b-f39f-4f41-a492-586171130c30 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:39.602490353 +0000 UTC m=+2.969680220 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3af7400b-f39f-4f41-a492-586171130c30-metrics-certs") pod "network-metrics-daemon-wb7n2" (UID: "3af7400b-f39f-4f41-a492-586171130c30") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:04:39.102928 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.102531    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab-run\") pod \"tuned-62zwj\" (UID: \"02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab\") " pod="openshift-cluster-node-tuning-operator/tuned-62zwj"
Jul 27 21:04:39.102928 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.102553    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/72112606-3839-4478-bac9-dfc58a7809f8-host-cni-netd\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:39.102928 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.102535    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/72112606-3839-4478-bac9-dfc58a7809f8-env-overrides\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:39.102928 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.102587    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b2bf895f-3920-43a8-977c-fffbd75796e4-sys-fs\") pod \"aws-ebs-csi-driver-node-9d68g\" (UID: \"b2bf895f-3920-43a8-977c-fffbd75796e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d68g"
Jul 27 21:04:39.102928 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.102611    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c-cni-binary-copy\") pod \"multus-xgbjk\" (UID: \"7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c\") " pod="openshift-multus/multus-xgbjk"
Jul 27 21:04:39.102928 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.102637    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c-host-run-k8s-cni-cncf-io\") pod \"multus-xgbjk\" (UID: \"7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c\") " pod="openshift-multus/multus-xgbjk"
Jul 27 21:04:39.102928 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.102644    2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 27 21:04:39.102928 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.102678    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c-host-run-k8s-cni-cncf-io\") pod \"multus-xgbjk\" (UID: \"7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c\") " pod="openshift-multus/multus-xgbjk"
Jul 27 21:04:39.102928 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.102711    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b2bf895f-3920-43a8-977c-fffbd75796e4-sys-fs\") pod \"aws-ebs-csi-driver-node-9d68g\" (UID: \"b2bf895f-3920-43a8-977c-fffbd75796e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d68g"
Jul 27 21:04:39.102928 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.102752    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b2bf895f-3920-43a8-977c-fffbd75796e4-registration-dir\") pod \"aws-ebs-csi-driver-node-9d68g\" (UID: \"b2bf895f-3920-43a8-977c-fffbd75796e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d68g"
Jul 27 21:04:39.102928 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.102808    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6-agent-certs\") pod \"konnectivity-agent-b9b6s\" (UID: \"0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6\") " pod="kube-system/konnectivity-agent-b9b6s"
Jul 27 21:04:39.103963 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.102836    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a4af1379-e40a-471d-a011-d50ab76b7f51-dbus\") pod \"global-pull-secret-syncer-5d7vf\" (UID: \"a4af1379-e40a-471d-a011-d50ab76b7f51\") " pod="kube-system/global-pull-secret-syncer-5d7vf"
Jul 27 21:04:39.103963 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.102861    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72112606-3839-4478-bac9-dfc58a7809f8-var-lib-openvswitch\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:39.103963 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.102906    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c-host-var-lib-cni-bin\") pod \"multus-xgbjk\" (UID: \"7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c\") " pod="openshift-multus/multus-xgbjk"
Jul 27 21:04:39.103963 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.102912    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b2bf895f-3920-43a8-977c-fffbd75796e4-registration-dir\") pod \"aws-ebs-csi-driver-node-9d68g\" (UID: \"b2bf895f-3920-43a8-977c-fffbd75796e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d68g"
Jul 27 21:04:39.103963 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.102968    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72112606-3839-4478-bac9-dfc58a7809f8-var-lib-openvswitch\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:39.103963 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.102974    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c-host-var-lib-kubelet\") pod \"multus-xgbjk\" (UID: \"7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c\") " pod="openshift-multus/multus-xgbjk"
Jul 27 21:04:39.103963 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.103001    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0b84c165-5dfa-416c-bec6-28c9d55197ab-host\") pod \"node-ca-dtklb\" (UID: \"0b84c165-5dfa-416c-bec6-28c9d55197ab\") " pod="openshift-image-registry/node-ca-dtklb"
Jul 27 21:04:39.103963 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.103028    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/72112606-3839-4478-bac9-dfc58a7809f8-env-overrides\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:39.103963 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.103044    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a4af1379-e40a-471d-a011-d50ab76b7f51-dbus\") pod \"global-pull-secret-syncer-5d7vf\" (UID: \"a4af1379-e40a-471d-a011-d50ab76b7f51\") " pod="kube-system/global-pull-secret-syncer-5d7vf"
Jul 27 21:04:39.103963 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.103054    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c-host-var-lib-cni-bin\") pod \"multus-xgbjk\" (UID: \"7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c\") " pod="openshift-multus/multus-xgbjk"
Jul 27 21:04:39.103963 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.103086    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b2bf895f-3920-43a8-977c-fffbd75796e4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9d68g\" (UID: \"b2bf895f-3920-43a8-977c-fffbd75796e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d68g"
Jul 27 21:04:39.103963 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.103089    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c-host-var-lib-kubelet\") pod \"multus-xgbjk\" (UID: \"7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c\") " pod="openshift-multus/multus-xgbjk"
Jul 27 21:04:39.103963 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.103119    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b2bf895f-3920-43a8-977c-fffbd75796e4-device-dir\") pod \"aws-ebs-csi-driver-node-9d68g\" (UID: \"b2bf895f-3920-43a8-977c-fffbd75796e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d68g"
Jul 27 21:04:39.103963 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.103126    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0b84c165-5dfa-416c-bec6-28c9d55197ab-host\") pod \"node-ca-dtklb\" (UID: \"0b84c165-5dfa-416c-bec6-28c9d55197ab\") " pod="openshift-image-registry/node-ca-dtklb"
Jul 27 21:04:39.103963 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.103142    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4zb7\" (UniqueName: \"kubernetes.io/projected/e610bffe-2c4d-4577-8a48-4849b0d3266d-kube-api-access-r4zb7\") pod \"node-resolver-frmv7\" (UID: \"e610bffe-2c4d-4577-8a48-4849b0d3266d\") " pod="openshift-dns/node-resolver-frmv7"
Jul 27 21:04:39.103963 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.103171    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/72112606-3839-4478-bac9-dfc58a7809f8-host-cni-bin\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:39.103963 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.103191    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab-lib-modules\") pod \"tuned-62zwj\" (UID: \"02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab\") " pod="openshift-cluster-node-tuning-operator/tuned-62zwj"
Jul 27 21:04:39.103963 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.103214    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab-tmp\") pod \"tuned-62zwj\" (UID: \"02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab\") " pod="openshift-cluster-node-tuning-operator/tuned-62zwj"
Jul 27 21:04:39.104860 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.103218    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b2bf895f-3920-43a8-977c-fffbd75796e4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9d68g\" (UID: \"b2bf895f-3920-43a8-977c-fffbd75796e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d68g"
Jul 27 21:04:39.104860 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.103224    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c-cni-binary-copy\") pod \"multus-xgbjk\" (UID: \"7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c\") " pod="openshift-multus/multus-xgbjk"
Jul 27 21:04:39.104860 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.103277    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/72112606-3839-4478-bac9-dfc58a7809f8-host-cni-bin\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:39.104860 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.103277    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b2bf895f-3920-43a8-977c-fffbd75796e4-device-dir\") pod \"aws-ebs-csi-driver-node-9d68g\" (UID: \"b2bf895f-3920-43a8-977c-fffbd75796e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d68g"
Jul 27 21:04:39.104860 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.103297    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx4gq\" (UniqueName: \"kubernetes.io/projected/02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab-kube-api-access-nx4gq\") pod \"tuned-62zwj\" (UID: \"02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab\") " pod="openshift-cluster-node-tuning-operator/tuned-62zwj"
Jul 27 21:04:39.104860 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.103323    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72112606-3839-4478-bac9-dfc58a7809f8-run-openvswitch\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:39.104860 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.103348    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c-multus-daemon-config\") pod \"multus-xgbjk\" (UID: \"7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c\") " pod="openshift-multus/multus-xgbjk"
Jul 27 21:04:39.104860 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.103369    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c-host-run-multus-certs\") pod \"multus-xgbjk\" (UID: \"7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c\") " pod="openshift-multus/multus-xgbjk"
Jul 27 21:04:39.104860 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.103393    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/72112606-3839-4478-bac9-dfc58a7809f8-run-ovn\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:39.104860 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.103417    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72112606-3839-4478-bac9-dfc58a7809f8-host-run-ovn-kubernetes\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:39.104860 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.103439    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/72112606-3839-4478-bac9-dfc58a7809f8-ovnkube-config\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:39.104860 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.103461    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c-os-release\") pod \"multus-xgbjk\" (UID: \"7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c\") " pod="openshift-multus/multus-xgbjk"
Jul 27 21:04:39.104860 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.103484    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab-etc-tuned\") pod \"tuned-62zwj\" (UID: \"02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab\") " pod="openshift-cluster-node-tuning-operator/tuned-62zwj"
Jul 27 21:04:39.104860 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.103506    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a4af1379-e40a-471d-a011-d50ab76b7f51-kubelet-config\") pod \"global-pull-secret-syncer-5d7vf\" (UID: \"a4af1379-e40a-471d-a011-d50ab76b7f51\") " pod="kube-system/global-pull-secret-syncer-5d7vf"
Jul 27 21:04:39.104860 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.103517    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/72112606-3839-4478-bac9-dfc58a7809f8-run-ovn\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:39.104860 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.103529    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/72112606-3839-4478-bac9-dfc58a7809f8-host-kubelet\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:39.104860 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.103554    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/72112606-3839-4478-bac9-dfc58a7809f8-log-socket\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:39.104860 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.103581    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c-system-cni-dir\") pod \"multus-xgbjk\" (UID: \"7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c\") " pod="openshift-multus/multus-xgbjk"
Jul 27 21:04:39.105446 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.103589    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c-host-run-multus-certs\") pod \"multus-xgbjk\" (UID: \"7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c\") " pod="openshift-multus/multus-xgbjk"
Jul 27 21:04:39.105446 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.103605    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c-multus-socket-dir-parent\") pod \"multus-xgbjk\" (UID: \"7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c\") " pod="openshift-multus/multus-xgbjk"
Jul 27 21:04:39.105446 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.103635    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a4af1379-e40a-471d-a011-d50ab76b7f51-kubelet-config\") pod \"global-pull-secret-syncer-5d7vf\" (UID: \"a4af1379-e40a-471d-a011-d50ab76b7f51\") " pod="kube-system/global-pull-secret-syncer-5d7vf"
Jul 27 21:04:39.105446 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.103638    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c-multus-conf-dir\") pod \"multus-xgbjk\" (UID: \"7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c\") " pod="openshift-multus/multus-xgbjk"
Jul 27 21:04:39.105446 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.103671    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c-multus-conf-dir\") pod \"multus-xgbjk\" (UID: \"7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c\") " pod="openshift-multus/multus-xgbjk"
Jul 27 21:04:39.105446 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.103672    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fd05d22e-5585-4f39-8597-4fc6978df793-cni-binary-copy\") pod \"multus-additional-cni-plugins-667ww\" (UID: \"fd05d22e-5585-4f39-8597-4fc6978df793\") " pod="openshift-multus/multus-additional-cni-plugins-667ww"
Jul 27 21:04:39.105446 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.103706    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72112606-3839-4478-bac9-dfc58a7809f8-host-run-ovn-kubernetes\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:39.105446 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.103716    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/72112606-3839-4478-bac9-dfc58a7809f8-host-kubelet\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:39.105446 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.103554    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab-lib-modules\") pod \"tuned-62zwj\" (UID: \"02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab\") " pod="openshift-cluster-node-tuning-operator/tuned-62zwj"
Jul 27 21:04:39.105446 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.103752    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72112606-3839-4478-bac9-dfc58a7809f8-run-openvswitch\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:39.105446 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.103830    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c-os-release\") pod \"multus-xgbjk\" (UID: \"7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c\") " pod="openshift-multus/multus-xgbjk"
Jul 27 21:04:39.105446 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.104002    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c-multus-daemon-config\") pod \"multus-xgbjk\" (UID: \"7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c\") " pod="openshift-multus/multus-xgbjk"
Jul 27 21:04:39.105446 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.104051    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/72112606-3839-4478-bac9-dfc58a7809f8-log-socket\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:39.105446 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.104115    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c-multus-socket-dir-parent\") pod \"multus-xgbjk\" (UID: \"7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c\") " pod="openshift-multus/multus-xgbjk"
Jul 27 21:04:39.105446 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.104165    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c-system-cni-dir\") pod \"multus-xgbjk\" (UID: \"7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c\") " pod="openshift-multus/multus-xgbjk"
Jul 27 21:04:39.105446 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.104195    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/72112606-3839-4478-bac9-dfc58a7809f8-ovnkube-config\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:39.105446 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.104195    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72112606-3839-4478-bac9-dfc58a7809f8-etc-openvswitch\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:39.105446 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.104232    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72112606-3839-4478-bac9-dfc58a7809f8-etc-openvswitch\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:39.106366 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.104237    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b2bf895f-3920-43a8-977c-fffbd75796e4-etc-selinux\") pod \"aws-ebs-csi-driver-node-9d68g\" (UID: \"b2bf895f-3920-43a8-977c-fffbd75796e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d68g"
Jul 27 21:04:39.106366 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.104269    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fd05d22e-5585-4f39-8597-4fc6978df793-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-667ww\" (UID: \"fd05d22e-5585-4f39-8597-4fc6978df793\") " pod="openshift-multus/multus-additional-cni-plugins-667ww"
Jul 27 21:04:39.106366 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.104296    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6-konnectivity-ca\") pod \"konnectivity-agent-b9b6s\" (UID: \"0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6\") " pod="kube-system/konnectivity-agent-b9b6s"
Jul 27 21:04:39.106366 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.104313    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b2bf895f-3920-43a8-977c-fffbd75796e4-etc-selinux\") pod \"aws-ebs-csi-driver-node-9d68g\" (UID: \"b2bf895f-3920-43a8-977c-fffbd75796e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d68g"
Jul 27 21:04:39.106366 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.104322    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab-etc-sysctl-d\") pod \"tuned-62zwj\" (UID: \"02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab\") " pod="openshift-cluster-node-tuning-operator/tuned-62zwj"
Jul 27 21:04:39.106366 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.104347    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72112606-3839-4478-bac9-dfc58a7809f8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:39.106366 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.104374    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab-etc-systemd\") pod \"tuned-62zwj\" (UID: \"02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab\") " pod="openshift-cluster-node-tuning-operator/tuned-62zwj"
Jul 27 21:04:39.106366 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.104397    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab-host\") pod \"tuned-62zwj\" (UID: \"02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab\") " pod="openshift-cluster-node-tuning-operator/tuned-62zwj"
Jul 27 21:04:39.106366 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.104420    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab-etc-sysctl-d\") pod \"tuned-62zwj\" (UID: \"02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab\") " pod="openshift-cluster-node-tuning-operator/tuned-62zwj"
Jul 27 21:04:39.106366 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.104421    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/72112606-3839-4478-bac9-dfc58a7809f8-host-run-netns\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:39.106366 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.104436    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72112606-3839-4478-bac9-dfc58a7809f8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:39.106366 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.104452    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/72112606-3839-4478-bac9-dfc58a7809f8-host-run-netns\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:39.106366 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.104453    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/72112606-3839-4478-bac9-dfc58a7809f8-ovnkube-script-lib\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:39.106366 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.104486    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c-multus-cni-dir\") pod \"multus-xgbjk\" (UID: \"7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c\") " pod="openshift-multus/multus-xgbjk"
Jul 27 21:04:39.106366 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.104497    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab-etc-systemd\") pod \"tuned-62zwj\" (UID: \"02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab\") " pod="openshift-cluster-node-tuning-operator/tuned-62zwj"
Jul 27 21:04:39.106366 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.104511    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c-etc-kubernetes\") pod \"multus-xgbjk\" (UID: \"7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c\") " pod="openshift-multus/multus-xgbjk"
Jul 27 21:04:39.106366 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.104542    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fd05d22e-5585-4f39-8597-4fc6978df793-os-release\") pod \"multus-additional-cni-plugins-667ww\" (UID: \"fd05d22e-5585-4f39-8597-4fc6978df793\") " pod="openshift-multus/multus-additional-cni-plugins-667ww"
Jul 27 21:04:39.107036 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.104567    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab-host\") pod \"tuned-62zwj\" (UID: \"02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab\") " pod="openshift-cluster-node-tuning-operator/tuned-62zwj"
Jul 27 21:04:39.107036 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.104570    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c-cnibin\") pod \"multus-xgbjk\" (UID: \"7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c\") " pod="openshift-multus/multus-xgbjk"
Jul 27 21:04:39.107036 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.104608    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c-hostroot\") pod \"multus-xgbjk\" (UID: \"7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c\") " pod="openshift-multus/multus-xgbjk"
Jul 27 21:04:39.107036 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.104633    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4sw5\" (UniqueName: \"kubernetes.io/projected/7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c-kube-api-access-j4sw5\") pod \"multus-xgbjk\" (UID: \"7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c\") " pod="openshift-multus/multus-xgbjk"
Jul 27 21:04:39.107036 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.104608    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c-etc-kubernetes\") pod \"multus-xgbjk\" (UID: \"7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c\") " pod="openshift-multus/multus-xgbjk"
Jul 27 21:04:39.107036 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.104651    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c-multus-cni-dir\") pod \"multus-xgbjk\" (UID: \"7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c\") " pod="openshift-multus/multus-xgbjk"
Jul 27 21:04:39.107036 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.104671    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b2bf895f-3920-43a8-977c-fffbd75796e4-socket-dir\") pod \"aws-ebs-csi-driver-node-9d68g\" (UID: \"b2bf895f-3920-43a8-977c-fffbd75796e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d68g"
Jul 27 21:04:39.107036 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.104676    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c-hostroot\") pod \"multus-xgbjk\" (UID: \"7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c\") " pod="openshift-multus/multus-xgbjk"
Jul 27 21:04:39.107036 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.104698    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e610bffe-2c4d-4577-8a48-4849b0d3266d-tmp-dir\") pod \"node-resolver-frmv7\" (UID: \"e610bffe-2c4d-4577-8a48-4849b0d3266d\") " pod="openshift-dns/node-resolver-frmv7"
Jul 27 21:04:39.107036 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.104725    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fd05d22e-5585-4f39-8597-4fc6978df793-system-cni-dir\") pod \"multus-additional-cni-plugins-667ww\" (UID: \"fd05d22e-5585-4f39-8597-4fc6978df793\") " pod="openshift-multus/multus-additional-cni-plugins-667ww"
Jul 27 21:04:39.107036 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.104734    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c-cnibin\") pod \"multus-xgbjk\" (UID: \"7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c\") " pod="openshift-multus/multus-xgbjk"
Jul 27 21:04:39.107036 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.104751    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fd05d22e-5585-4f39-8597-4fc6978df793-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-667ww\" (UID: \"fd05d22e-5585-4f39-8597-4fc6978df793\") " pod="openshift-multus/multus-additional-cni-plugins-667ww"
Jul 27 21:04:39.107036 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.104887    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b2bf895f-3920-43a8-977c-fffbd75796e4-socket-dir\") pod \"aws-ebs-csi-driver-node-9d68g\" (UID: \"b2bf895f-3920-43a8-977c-fffbd75796e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d68g"
Jul 27 21:04:39.107036 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.104933    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/72112606-3839-4478-bac9-dfc58a7809f8-ovnkube-script-lib\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:39.107036 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.104938    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c35bbe72-c030-4b9d-aece-459d907893dc-iptables-alerter-script\") pod \"iptables-alerter-fdrp9\" (UID: \"c35bbe72-c030-4b9d-aece-459d907893dc\") " pod="openshift-network-operator/iptables-alerter-fdrp9"
Jul 27 21:04:39.107036 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.104969    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmvcb\" (UniqueName: \"kubernetes.io/projected/c5915867-c603-4d51-8417-42aada6603ef-kube-api-access-mmvcb\") pod \"network-check-target-nj9np\" (UID: \"c5915867-c603-4d51-8417-42aada6603ef\") " pod="openshift-network-diagnostics/network-check-target-nj9np"
Jul 27 21:04:39.107036 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.104993    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c-host-run-netns\") pod \"multus-xgbjk\" (UID: \"7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c\") " pod="openshift-multus/multus-xgbjk"
Jul 27 21:04:39.107730 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.105017    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c-host-var-lib-cni-multus\") pod \"multus-xgbjk\" (UID: \"7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c\") " pod="openshift-multus/multus-xgbjk"
Jul 27 21:04:39.107730 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.105038    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/72112606-3839-4478-bac9-dfc58a7809f8-systemd-units\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:39.107730 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.105067    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c-host-run-netns\") pod \"multus-xgbjk\" (UID: \"7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c\") " pod="openshift-multus/multus-xgbjk"
Jul 27 21:04:39.107730 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.105065    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a4af1379-e40a-471d-a011-d50ab76b7f51-original-pull-secret\") pod \"global-pull-secret-syncer-5d7vf\" (UID: \"a4af1379-e40a-471d-a011-d50ab76b7f51\") " pod="kube-system/global-pull-secret-syncer-5d7vf"
Jul 27 21:04:39.107730 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.105105    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0b84c165-5dfa-416c-bec6-28c9d55197ab-serviceca\") pod \"node-ca-dtklb\" (UID: \"0b84c165-5dfa-416c-bec6-28c9d55197ab\") " pod="openshift-image-registry/node-ca-dtklb"
Jul 27 21:04:39.107730 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.105113    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c-host-var-lib-cni-multus\") pod \"multus-xgbjk\" (UID: \"7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c\") " pod="openshift-multus/multus-xgbjk"
Jul 27 21:04:39.107730 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.105127    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/72112606-3839-4478-bac9-dfc58a7809f8-run-systemd\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:39.107730 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.105157    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab-etc-sysctl-conf\") pod \"tuned-62zwj\" (UID: \"02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab\") " pod="openshift-cluster-node-tuning-operator/tuned-62zwj"
Jul 27 21:04:39.107730 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.105181    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/72112606-3839-4478-bac9-dfc58a7809f8-node-log\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:39.107730 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.105209    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbrbf\" (UniqueName: \"kubernetes.io/projected/b2bf895f-3920-43a8-977c-fffbd75796e4-kube-api-access-nbrbf\") pod \"aws-ebs-csi-driver-node-9d68g\" (UID: \"b2bf895f-3920-43a8-977c-fffbd75796e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d68g"
Jul 27 21:04:39.107730 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.105215    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/72112606-3839-4478-bac9-dfc58a7809f8-systemd-units\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:39.107730 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.105231    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/72112606-3839-4478-bac9-dfc58a7809f8-run-systemd\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:39.107730 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.105238    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e610bffe-2c4d-4577-8a48-4849b0d3266d-hosts-file\") pod \"node-resolver-frmv7\" (UID: \"e610bffe-2c4d-4577-8a48-4849b0d3266d\") " pod="openshift-dns/node-resolver-frmv7"
Jul 27 21:04:39.107730 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:39.105185    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 21:04:39.107730 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.105276    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/72112606-3839-4478-bac9-dfc58a7809f8-node-log\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:39.107730 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:39.105310    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4af1379-e40a-471d-a011-d50ab76b7f51-original-pull-secret podName:a4af1379-e40a-471d-a011-d50ab76b7f51 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:39.605290947 +0000 UTC m=+2.972480793 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a4af1379-e40a-471d-a011-d50ab76b7f51-original-pull-secret") pod "global-pull-secret-syncer-5d7vf" (UID: "a4af1379-e40a-471d-a011-d50ab76b7f51") : object "kube-system"/"original-pull-secret" not registered
Jul 27 21:04:39.107730 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.105329    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fd05d22e-5585-4f39-8597-4fc6978df793-tuning-conf-dir\") pod \"multus-additional-cni-plugins-667ww\" (UID: \"fd05d22e-5585-4f39-8597-4fc6978df793\") " pod="openshift-multus/multus-additional-cni-plugins-667ww"
Jul 27 21:04:39.108350 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.105391    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab-etc-sysctl-conf\") pod \"tuned-62zwj\" (UID: \"02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab\") " pod="openshift-cluster-node-tuning-operator/tuned-62zwj"
Jul 27 21:04:39.108350 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.105626    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0b84c165-5dfa-416c-bec6-28c9d55197ab-serviceca\") pod \"node-ca-dtklb\" (UID: \"0b84c165-5dfa-416c-bec6-28c9d55197ab\") " pod="openshift-image-registry/node-ca-dtklb"
Jul 27 21:04:39.108350 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.106487    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/72112606-3839-4478-bac9-dfc58a7809f8-ovn-node-metrics-cert\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:39.108350 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.106743    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab-etc-tuned\") pod \"tuned-62zwj\" (UID: \"02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab\") " pod="openshift-cluster-node-tuning-operator/tuned-62zwj"
Jul 27 21:04:39.108350 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.107445    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab-tmp\") pod \"tuned-62zwj\" (UID: \"02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab\") " pod="openshift-cluster-node-tuning-operator/tuned-62zwj"
Jul 27 21:04:39.110647 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:39.109852    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 21:04:39.110647 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:39.109875    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 21:04:39.110647 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:39.109888    2577 projected.go:196] Error preparing data for projected volume kube-api-access-mmvcb for pod openshift-network-diagnostics/network-check-target-nj9np: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:04:39.110647 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:39.109952    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c5915867-c603-4d51-8417-42aada6603ef-kube-api-access-mmvcb podName:c5915867-c603-4d51-8417-42aada6603ef nodeName:}" failed. No retries permitted until 2026-07-27 21:04:39.609935644 +0000 UTC m=+2.977125510 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mmvcb" (UniqueName: "kubernetes.io/projected/c5915867-c603-4d51-8417-42aada6603ef-kube-api-access-mmvcb") pod "network-check-target-nj9np" (UID: "c5915867-c603-4d51-8417-42aada6603ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:04:39.111465 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.111382    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t9zx\" (UniqueName: \"kubernetes.io/projected/3af7400b-f39f-4f41-a492-586171130c30-kube-api-access-4t9zx\") pod \"network-metrics-daemon-wb7n2\" (UID: \"3af7400b-f39f-4f41-a492-586171130c30\") " pod="openshift-multus/network-metrics-daemon-wb7n2"
Jul 27 21:04:39.111684 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.111662    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t6n6\" (UniqueName: \"kubernetes.io/projected/0b84c165-5dfa-416c-bec6-28c9d55197ab-kube-api-access-7t6n6\") pod \"node-ca-dtklb\" (UID: \"0b84c165-5dfa-416c-bec6-28c9d55197ab\") " pod="openshift-image-registry/node-ca-dtklb"
Jul 27 21:04:39.112138 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.112103    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx4gq\" (UniqueName: \"kubernetes.io/projected/02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab-kube-api-access-nx4gq\") pod \"tuned-62zwj\" (UID: \"02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab\") " pod="openshift-cluster-node-tuning-operator/tuned-62zwj"
Jul 27 21:04:39.112602 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.112584    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nrb7\" (UniqueName: \"kubernetes.io/projected/72112606-3839-4478-bac9-dfc58a7809f8-kube-api-access-9nrb7\") pod \"ovnkube-node-pftzz\" (UID: \"72112606-3839-4478-bac9-dfc58a7809f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:39.113483 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.113467    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4sw5\" (UniqueName: \"kubernetes.io/projected/7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c-kube-api-access-j4sw5\") pod \"multus-xgbjk\" (UID: \"7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c\") " pod="openshift-multus/multus-xgbjk"
Jul 27 21:04:39.113561 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.113467    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbrbf\" (UniqueName: \"kubernetes.io/projected/b2bf895f-3920-43a8-977c-fffbd75796e4-kube-api-access-nbrbf\") pod \"aws-ebs-csi-driver-node-9d68g\" (UID: \"b2bf895f-3920-43a8-977c-fffbd75796e4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d68g"
Jul 27 21:04:39.134671 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.134619    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-219.ec2.internal" event={"ID":"abb843a58c4a692b2202031f20068533","Type":"ContainerStarted","Data":"1c45b08b10809a49649f44e41397b8b1d6725c9d89042b655f3640d274f41b88"}
Jul 27 21:04:39.135741 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.135691    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-219.ec2.internal" event={"ID":"7fe2fc32304c33ef13ea6c3a49bb44af","Type":"ContainerStarted","Data":"686ade05c79b2844ae7d5f87f13306e9e3b1faacffa2980fa5f787c2701e7eb4"}
Jul 27 21:04:39.205946 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.205911    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fd05d22e-5585-4f39-8597-4fc6978df793-cnibin\") pod \"multus-additional-cni-plugins-667ww\" (UID: \"fd05d22e-5585-4f39-8597-4fc6978df793\") " pod="openshift-multus/multus-additional-cni-plugins-667ww"
Jul 27 21:04:39.206137 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.205963    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mlv5\" (UniqueName: \"kubernetes.io/projected/fd05d22e-5585-4f39-8597-4fc6978df793-kube-api-access-4mlv5\") pod \"multus-additional-cni-plugins-667ww\" (UID: \"fd05d22e-5585-4f39-8597-4fc6978df793\") " pod="openshift-multus/multus-additional-cni-plugins-667ww"
Jul 27 21:04:39.206137 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.205985    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr8hf\" (UniqueName: \"kubernetes.io/projected/c35bbe72-c030-4b9d-aece-459d907893dc-kube-api-access-qr8hf\") pod \"iptables-alerter-fdrp9\" (UID: \"c35bbe72-c030-4b9d-aece-459d907893dc\") " pod="openshift-network-operator/iptables-alerter-fdrp9"
Jul 27 21:04:39.206137 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.206013    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6-agent-certs\") pod \"konnectivity-agent-b9b6s\" (UID: \"0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6\") " pod="kube-system/konnectivity-agent-b9b6s"
Jul 27 21:04:39.206137 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.206037    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fd05d22e-5585-4f39-8597-4fc6978df793-cnibin\") pod \"multus-additional-cni-plugins-667ww\" (UID: \"fd05d22e-5585-4f39-8597-4fc6978df793\") " pod="openshift-multus/multus-additional-cni-plugins-667ww"
Jul 27 21:04:39.206137 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.206043    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4zb7\" (UniqueName: \"kubernetes.io/projected/e610bffe-2c4d-4577-8a48-4849b0d3266d-kube-api-access-r4zb7\") pod \"node-resolver-frmv7\" (UID: \"e610bffe-2c4d-4577-8a48-4849b0d3266d\") " pod="openshift-dns/node-resolver-frmv7"
Jul 27 21:04:39.206386 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.206161    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fd05d22e-5585-4f39-8597-4fc6978df793-cni-binary-copy\") pod \"multus-additional-cni-plugins-667ww\" (UID: \"fd05d22e-5585-4f39-8597-4fc6978df793\") " pod="openshift-multus/multus-additional-cni-plugins-667ww"
Jul 27 21:04:39.206386 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.206193    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fd05d22e-5585-4f39-8597-4fc6978df793-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-667ww\" (UID: \"fd05d22e-5585-4f39-8597-4fc6978df793\") " pod="openshift-multus/multus-additional-cni-plugins-667ww"
Jul 27 21:04:39.206386 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.206220    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6-konnectivity-ca\") pod \"konnectivity-agent-b9b6s\" (UID: \"0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6\") " pod="kube-system/konnectivity-agent-b9b6s"
Jul 27 21:04:39.206386 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.206253    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fd05d22e-5585-4f39-8597-4fc6978df793-os-release\") pod \"multus-additional-cni-plugins-667ww\" (UID: \"fd05d22e-5585-4f39-8597-4fc6978df793\") " pod="openshift-multus/multus-additional-cni-plugins-667ww"
Jul 27 21:04:39.206386 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.206285    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e610bffe-2c4d-4577-8a48-4849b0d3266d-tmp-dir\") pod \"node-resolver-frmv7\" (UID: \"e610bffe-2c4d-4577-8a48-4849b0d3266d\") " pod="openshift-dns/node-resolver-frmv7"
Jul 27 21:04:39.206386 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.206307    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fd05d22e-5585-4f39-8597-4fc6978df793-system-cni-dir\") pod \"multus-additional-cni-plugins-667ww\" (UID: \"fd05d22e-5585-4f39-8597-4fc6978df793\") " pod="openshift-multus/multus-additional-cni-plugins-667ww"
Jul 27 21:04:39.206386 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.206332    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fd05d22e-5585-4f39-8597-4fc6978df793-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-667ww\" (UID: \"fd05d22e-5585-4f39-8597-4fc6978df793\") " pod="openshift-multus/multus-additional-cni-plugins-667ww"
Jul 27 21:04:39.206386 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.206351    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c35bbe72-c030-4b9d-aece-459d907893dc-iptables-alerter-script\") pod \"iptables-alerter-fdrp9\" (UID: \"c35bbe72-c030-4b9d-aece-459d907893dc\") " pod="openshift-network-operator/iptables-alerter-fdrp9"
Jul 27 21:04:39.206757 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.206395    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e610bffe-2c4d-4577-8a48-4849b0d3266d-hosts-file\") pod \"node-resolver-frmv7\" (UID: \"e610bffe-2c4d-4577-8a48-4849b0d3266d\") " pod="openshift-dns/node-resolver-frmv7"
Jul 27 21:04:39.206757 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.206413    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fd05d22e-5585-4f39-8597-4fc6978df793-tuning-conf-dir\") pod \"multus-additional-cni-plugins-667ww\" (UID: \"fd05d22e-5585-4f39-8597-4fc6978df793\") " pod="openshift-multus/multus-additional-cni-plugins-667ww"
Jul 27 21:04:39.206757 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.206435    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c35bbe72-c030-4b9d-aece-459d907893dc-host-slash\") pod \"iptables-alerter-fdrp9\" (UID: \"c35bbe72-c030-4b9d-aece-459d907893dc\") " pod="openshift-network-operator/iptables-alerter-fdrp9"
Jul 27 21:04:39.206757 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.206488    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fd05d22e-5585-4f39-8597-4fc6978df793-os-release\") pod \"multus-additional-cni-plugins-667ww\" (UID: \"fd05d22e-5585-4f39-8597-4fc6978df793\") " pod="openshift-multus/multus-additional-cni-plugins-667ww"
Jul 27 21:04:39.206757 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.206499    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c35bbe72-c030-4b9d-aece-459d907893dc-host-slash\") pod \"iptables-alerter-fdrp9\" (UID: \"c35bbe72-c030-4b9d-aece-459d907893dc\") " pod="openshift-network-operator/iptables-alerter-fdrp9"
Jul 27 21:04:39.206757 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.206573    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e610bffe-2c4d-4577-8a48-4849b0d3266d-tmp-dir\") pod \"node-resolver-frmv7\" (UID: \"e610bffe-2c4d-4577-8a48-4849b0d3266d\") " pod="openshift-dns/node-resolver-frmv7"
Jul 27 21:04:39.206757 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.206634    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fd05d22e-5585-4f39-8597-4fc6978df793-system-cni-dir\") pod \"multus-additional-cni-plugins-667ww\" (UID: \"fd05d22e-5585-4f39-8597-4fc6978df793\") " pod="openshift-multus/multus-additional-cni-plugins-667ww"
Jul 27 21:04:39.206757 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.206642    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fd05d22e-5585-4f39-8597-4fc6978df793-tuning-conf-dir\") pod \"multus-additional-cni-plugins-667ww\" (UID: \"fd05d22e-5585-4f39-8597-4fc6978df793\") " pod="openshift-multus/multus-additional-cni-plugins-667ww"
Jul 27 21:04:39.207085 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.206790    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e610bffe-2c4d-4577-8a48-4849b0d3266d-hosts-file\") pod \"node-resolver-frmv7\" (UID: \"e610bffe-2c4d-4577-8a48-4849b0d3266d\") " pod="openshift-dns/node-resolver-frmv7"
Jul 27 21:04:39.207085 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.206855    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6-konnectivity-ca\") pod \"konnectivity-agent-b9b6s\" (UID: \"0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6\") " pod="kube-system/konnectivity-agent-b9b6s"
Jul 27 21:04:39.207085 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.207019    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c35bbe72-c030-4b9d-aece-459d907893dc-iptables-alerter-script\") pod \"iptables-alerter-fdrp9\" (UID: \"c35bbe72-c030-4b9d-aece-459d907893dc\") " pod="openshift-network-operator/iptables-alerter-fdrp9"
Jul 27 21:04:39.207420 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.207398    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fd05d22e-5585-4f39-8597-4fc6978df793-cni-binary-copy\") pod \"multus-additional-cni-plugins-667ww\" (UID: \"fd05d22e-5585-4f39-8597-4fc6978df793\") " pod="openshift-multus/multus-additional-cni-plugins-667ww"
Jul 27 21:04:39.207521 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.207449    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fd05d22e-5585-4f39-8597-4fc6978df793-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-667ww\" (UID: \"fd05d22e-5585-4f39-8597-4fc6978df793\") " pod="openshift-multus/multus-additional-cni-plugins-667ww"
Jul 27 21:04:39.207599 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.207527    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fd05d22e-5585-4f39-8597-4fc6978df793-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-667ww\" (UID: \"fd05d22e-5585-4f39-8597-4fc6978df793\") " pod="openshift-multus/multus-additional-cni-plugins-667ww"
Jul 27 21:04:39.208639 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.208618    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6-agent-certs\") pod \"konnectivity-agent-b9b6s\" (UID: \"0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6\") " pod="kube-system/konnectivity-agent-b9b6s"
Jul 27 21:04:39.214281 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.214232    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mlv5\" (UniqueName: \"kubernetes.io/projected/fd05d22e-5585-4f39-8597-4fc6978df793-kube-api-access-4mlv5\") pod \"multus-additional-cni-plugins-667ww\" (UID: \"fd05d22e-5585-4f39-8597-4fc6978df793\") " pod="openshift-multus/multus-additional-cni-plugins-667ww"
Jul 27 21:04:39.214281 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.214259    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr8hf\" (UniqueName: \"kubernetes.io/projected/c35bbe72-c030-4b9d-aece-459d907893dc-kube-api-access-qr8hf\") pod \"iptables-alerter-fdrp9\" (UID: \"c35bbe72-c030-4b9d-aece-459d907893dc\") " pod="openshift-network-operator/iptables-alerter-fdrp9"
Jul 27 21:04:39.214281 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.214233    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4zb7\" (UniqueName: \"kubernetes.io/projected/e610bffe-2c4d-4577-8a48-4849b0d3266d-kube-api-access-r4zb7\") pod \"node-resolver-frmv7\" (UID: \"e610bffe-2c4d-4577-8a48-4849b0d3266d\") " pod="openshift-dns/node-resolver-frmv7"
Jul 27 21:04:39.291327 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.291291    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xgbjk"
Jul 27 21:04:39.302756 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.302726    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d68g"
Jul 27 21:04:39.310571 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.310547    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-62zwj"
Jul 27 21:04:39.315029 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.315011    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dtklb"
Jul 27 21:04:39.323700 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.323679    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:04:39.332230 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.332207    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-frmv7"
Jul 27 21:04:39.338857 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.338838    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-667ww"
Jul 27 21:04:39.347387 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.347365    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-b9b6s"
Jul 27 21:04:39.353884 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.353860    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fdrp9"
Jul 27 21:04:39.385411 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.385381    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 21:04:39.609871 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.609785    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a4af1379-e40a-471d-a011-d50ab76b7f51-original-pull-secret\") pod \"global-pull-secret-syncer-5d7vf\" (UID: \"a4af1379-e40a-471d-a011-d50ab76b7f51\") " pod="kube-system/global-pull-secret-syncer-5d7vf"
Jul 27 21:04:39.609871 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.609839    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3af7400b-f39f-4f41-a492-586171130c30-metrics-certs\") pod \"network-metrics-daemon-wb7n2\" (UID: \"3af7400b-f39f-4f41-a492-586171130c30\") " pod="openshift-multus/network-metrics-daemon-wb7n2"
Jul 27 21:04:39.610068 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:39.609931    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:04:39.610068 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:39.609987    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3af7400b-f39f-4f41-a492-586171130c30-metrics-certs podName:3af7400b-f39f-4f41-a492-586171130c30 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:40.609967581 +0000 UTC m=+3.977157426 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3af7400b-f39f-4f41-a492-586171130c30-metrics-certs") pod "network-metrics-daemon-wb7n2" (UID: "3af7400b-f39f-4f41-a492-586171130c30") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:04:39.610068 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:39.609931    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 21:04:39.610196 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:39.610074    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4af1379-e40a-471d-a011-d50ab76b7f51-original-pull-secret podName:a4af1379-e40a-471d-a011-d50ab76b7f51 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:40.610034679 +0000 UTC m=+3.977224530 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a4af1379-e40a-471d-a011-d50ab76b7f51-original-pull-secret") pod "global-pull-secret-syncer-5d7vf" (UID: "a4af1379-e40a-471d-a011-d50ab76b7f51") : object "kube-system"/"original-pull-secret" not registered
Jul 27 21:04:39.710530 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:39.710494    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmvcb\" (UniqueName: \"kubernetes.io/projected/c5915867-c603-4d51-8417-42aada6603ef-kube-api-access-mmvcb\") pod \"network-check-target-nj9np\" (UID: \"c5915867-c603-4d51-8417-42aada6603ef\") " pod="openshift-network-diagnostics/network-check-target-nj9np"
Jul 27 21:04:39.710715 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:39.710647    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 21:04:39.710715 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:39.710672    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 21:04:39.710715 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:39.710686    2577 projected.go:196] Error preparing data for projected volume kube-api-access-mmvcb for pod openshift-network-diagnostics/network-check-target-nj9np: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:04:39.710896 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:39.710747    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c5915867-c603-4d51-8417-42aada6603ef-kube-api-access-mmvcb podName:c5915867-c603-4d51-8417-42aada6603ef nodeName:}" failed. No retries permitted until 2026-07-27 21:04:40.710731311 +0000 UTC m=+4.077921180 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mmvcb" (UniqueName: "kubernetes.io/projected/c5915867-c603-4d51-8417-42aada6603ef-kube-api-access-mmvcb") pod "network-check-target-nj9np" (UID: "c5915867-c603-4d51-8417-42aada6603ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:04:40.039187 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:40.039065    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 20:59:39 +0000 UTC" deadline="2028-01-24 22:27:34.851508576 +0000 UTC"
Jul 27 21:04:40.039187 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:40.039137    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13105h22m54.81237494s"
Jul 27 21:04:40.102868 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:40.102841    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc35bbe72_c030_4b9d_aece_459d907893dc.slice/crio-9fbeada81f4c896a309668a3b414cadfafee7b8261959ba57b0ccd1a09816b96 WatchSource:0}: Error finding container 9fbeada81f4c896a309668a3b414cadfafee7b8261959ba57b0ccd1a09816b96: Status 404 returned error can't find the container with id 9fbeada81f4c896a309668a3b414cadfafee7b8261959ba57b0ccd1a09816b96
Jul 27 21:04:40.107320 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:40.107234    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b84c165_5dfa_416c_bec6_28c9d55197ab.slice/crio-be5f079777cd4a75413037b5a23c1428fdd5c5c421eef679122fd89245ce3d04 WatchSource:0}: Error finding container be5f079777cd4a75413037b5a23c1428fdd5c5c421eef679122fd89245ce3d04: Status 404 returned error can't find the container with id be5f079777cd4a75413037b5a23c1428fdd5c5c421eef679122fd89245ce3d04
Jul 27 21:04:40.108069 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:40.108043    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd05d22e_5585_4f39_8597_4fc6978df793.slice/crio-218f0c14493ed9b493e0fa1adb94f5d1917f2e5c811bcb4951f1d651bc154a56 WatchSource:0}: Error finding container 218f0c14493ed9b493e0fa1adb94f5d1917f2e5c811bcb4951f1d651bc154a56: Status 404 returned error can't find the container with id 218f0c14493ed9b493e0fa1adb94f5d1917f2e5c811bcb4951f1d651bc154a56
Jul 27 21:04:40.108580 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:40.108556    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c0431ce_d23b_441e_86fc_aa2a5f1c4b0c.slice/crio-ab91592a7965e5617cdc7bc65921ca038e43dd271fb8f9c993e6d9072618dee8 WatchSource:0}: Error finding container ab91592a7965e5617cdc7bc65921ca038e43dd271fb8f9c993e6d9072618dee8: Status 404 returned error can't find the container with id ab91592a7965e5617cdc7bc65921ca038e43dd271fb8f9c993e6d9072618dee8
Jul 27 21:04:40.110433 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:40.110410    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02c8e99e_0fb6_488d_a75a_fd9efe3cc4ab.slice/crio-948cb86489768e4df256dd9e1f69b1ee31b50fb933e3e2c82cf41323f6ba9c21 WatchSource:0}: Error finding container 948cb86489768e4df256dd9e1f69b1ee31b50fb933e3e2c82cf41323f6ba9c21: Status 404 returned error can't find the container with id 948cb86489768e4df256dd9e1f69b1ee31b50fb933e3e2c82cf41323f6ba9c21
Jul 27 21:04:40.111501 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:40.111168    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72112606_3839_4478_bac9_dfc58a7809f8.slice/crio-b5dcffecbd4c16ad109c1062a7f8d0f68c7b92a3fe86ded923d5db610abd58bd WatchSource:0}: Error finding container b5dcffecbd4c16ad109c1062a7f8d0f68c7b92a3fe86ded923d5db610abd58bd: Status 404 returned error can't find the container with id b5dcffecbd4c16ad109c1062a7f8d0f68c7b92a3fe86ded923d5db610abd58bd
Jul 27 21:04:40.112825 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:40.112713    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b7c9f3b_8cf6_44dd_8378_3b5af6b644b6.slice/crio-fc9ae5bdc288cb21a66918f37c806e9af75a44276c9f2ccd579902963b84b6d9 WatchSource:0}: Error finding container fc9ae5bdc288cb21a66918f37c806e9af75a44276c9f2ccd579902963b84b6d9: Status 404 returned error can't find the container with id fc9ae5bdc288cb21a66918f37c806e9af75a44276c9f2ccd579902963b84b6d9
Jul 27 21:04:40.114506 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:04:40.113910    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode610bffe_2c4d_4577_8a48_4849b0d3266d.slice/crio-bd3718ef33956c6e1cedbe65e4ad7f1f9199fff653ef579330c934cb092ad4e1 WatchSource:0}: Error finding container bd3718ef33956c6e1cedbe65e4ad7f1f9199fff653ef579330c934cb092ad4e1: Status 404 returned error can't find the container with id bd3718ef33956c6e1cedbe65e4ad7f1f9199fff653ef579330c934cb092ad4e1
Jul 27 21:04:40.130714 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:40.130678    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nj9np"
Jul 27 21:04:40.130714 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:40.130705    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5d7vf"
Jul 27 21:04:40.130948 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:40.130819    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nj9np" podUID="c5915867-c603-4d51-8417-42aada6603ef"
Jul 27 21:04:40.131000 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:40.130948    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5d7vf" podUID="a4af1379-e40a-471d-a011-d50ab76b7f51"
Jul 27 21:04:40.138292 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:40.138263    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dtklb" event={"ID":"0b84c165-5dfa-416c-bec6-28c9d55197ab","Type":"ContainerStarted","Data":"be5f079777cd4a75413037b5a23c1428fdd5c5c421eef679122fd89245ce3d04"}
Jul 27 21:04:40.139624 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:40.139601    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fdrp9" event={"ID":"c35bbe72-c030-4b9d-aece-459d907893dc","Type":"ContainerStarted","Data":"9fbeada81f4c896a309668a3b414cadfafee7b8261959ba57b0ccd1a09816b96"}
Jul 27 21:04:40.141129 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:40.141104    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-62zwj" event={"ID":"02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab","Type":"ContainerStarted","Data":"948cb86489768e4df256dd9e1f69b1ee31b50fb933e3e2c82cf41323f6ba9c21"}
Jul 27 21:04:40.142707 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:40.142686    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xgbjk" event={"ID":"7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c","Type":"ContainerStarted","Data":"ab91592a7965e5617cdc7bc65921ca038e43dd271fb8f9c993e6d9072618dee8"}
Jul 27 21:04:40.145818 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:40.145794    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d68g" event={"ID":"b2bf895f-3920-43a8-977c-fffbd75796e4","Type":"ContainerStarted","Data":"21b5f5775053812924ecbcd1b06e477269fe7081b62b14a0401677f57d76fd6d"}
Jul 27 21:04:40.151217 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:40.151093    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-frmv7" event={"ID":"e610bffe-2c4d-4577-8a48-4849b0d3266d","Type":"ContainerStarted","Data":"bd3718ef33956c6e1cedbe65e4ad7f1f9199fff653ef579330c934cb092ad4e1"}
Jul 27 21:04:40.152094 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:40.152077    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-b9b6s" event={"ID":"0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6","Type":"ContainerStarted","Data":"fc9ae5bdc288cb21a66918f37c806e9af75a44276c9f2ccd579902963b84b6d9"}
Jul 27 21:04:40.153462 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:40.153422    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pftzz" event={"ID":"72112606-3839-4478-bac9-dfc58a7809f8","Type":"ContainerStarted","Data":"b5dcffecbd4c16ad109c1062a7f8d0f68c7b92a3fe86ded923d5db610abd58bd"}
Jul 27 21:04:40.154403 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:40.154386    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-667ww" event={"ID":"fd05d22e-5585-4f39-8597-4fc6978df793","Type":"ContainerStarted","Data":"218f0c14493ed9b493e0fa1adb94f5d1917f2e5c811bcb4951f1d651bc154a56"}
Jul 27 21:04:40.616732 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:40.615910    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a4af1379-e40a-471d-a011-d50ab76b7f51-original-pull-secret\") pod \"global-pull-secret-syncer-5d7vf\" (UID: \"a4af1379-e40a-471d-a011-d50ab76b7f51\") " pod="kube-system/global-pull-secret-syncer-5d7vf"
Jul 27 21:04:40.616732 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:40.615966    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3af7400b-f39f-4f41-a492-586171130c30-metrics-certs\") pod \"network-metrics-daemon-wb7n2\" (UID: \"3af7400b-f39f-4f41-a492-586171130c30\") " pod="openshift-multus/network-metrics-daemon-wb7n2"
Jul 27 21:04:40.616732 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:40.616103    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:04:40.616732 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:40.616178    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3af7400b-f39f-4f41-a492-586171130c30-metrics-certs podName:3af7400b-f39f-4f41-a492-586171130c30 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:42.616151304 +0000 UTC m=+5.983341165 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3af7400b-f39f-4f41-a492-586171130c30-metrics-certs") pod "network-metrics-daemon-wb7n2" (UID: "3af7400b-f39f-4f41-a492-586171130c30") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:04:40.616732 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:40.616570    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 21:04:40.616732 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:40.616648    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4af1379-e40a-471d-a011-d50ab76b7f51-original-pull-secret podName:a4af1379-e40a-471d-a011-d50ab76b7f51 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:42.616632176 +0000 UTC m=+5.983822024 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a4af1379-e40a-471d-a011-d50ab76b7f51-original-pull-secret") pod "global-pull-secret-syncer-5d7vf" (UID: "a4af1379-e40a-471d-a011-d50ab76b7f51") : object "kube-system"/"original-pull-secret" not registered
Jul 27 21:04:40.719029 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:40.718993    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmvcb\" (UniqueName: \"kubernetes.io/projected/c5915867-c603-4d51-8417-42aada6603ef-kube-api-access-mmvcb\") pod \"network-check-target-nj9np\" (UID: \"c5915867-c603-4d51-8417-42aada6603ef\") " pod="openshift-network-diagnostics/network-check-target-nj9np"
Jul 27 21:04:40.719218 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:40.719202    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 21:04:40.719287 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:40.719230    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 21:04:40.719287 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:40.719243    2577 projected.go:196] Error preparing data for projected volume kube-api-access-mmvcb for pod openshift-network-diagnostics/network-check-target-nj9np: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:04:40.719440 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:40.719302    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c5915867-c603-4d51-8417-42aada6603ef-kube-api-access-mmvcb podName:c5915867-c603-4d51-8417-42aada6603ef nodeName:}" failed. No retries permitted until 2026-07-27 21:04:42.719283468 +0000 UTC m=+6.086473334 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mmvcb" (UniqueName: "kubernetes.io/projected/c5915867-c603-4d51-8417-42aada6603ef-kube-api-access-mmvcb") pod "network-check-target-nj9np" (UID: "c5915867-c603-4d51-8417-42aada6603ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:04:41.039962 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:41.039871    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 20:59:39 +0000 UTC" deadline="2028-04-16 05:29:05.01383361 +0000 UTC"
Jul 27 21:04:41.039962 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:41.039912    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15080h24m23.973926419s"
Jul 27 21:04:41.134197 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:41.132242    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb7n2"
Jul 27 21:04:41.134197 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:41.132356    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wb7n2" podUID="3af7400b-f39f-4f41-a492-586171130c30"
Jul 27 21:04:41.170818 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:41.170078    2577 generic.go:356] "Generic (PLEG): container finished" podID="abb843a58c4a692b2202031f20068533" containerID="ef12bf494176f4737e26cb21378f954df3d78983a11442a3ce80840aaadf3c63" exitCode=0
Jul 27 21:04:41.170818 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:41.170164    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-219.ec2.internal" event={"ID":"abb843a58c4a692b2202031f20068533","Type":"ContainerDied","Data":"ef12bf494176f4737e26cb21378f954df3d78983a11442a3ce80840aaadf3c63"}
Jul 27 21:04:41.189150 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:41.189118    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-219.ec2.internal" event={"ID":"7fe2fc32304c33ef13ea6c3a49bb44af","Type":"ContainerStarted","Data":"b8421f918efce6912c7d204e8d7a3346350f2fce64ba7c287ddcb7bb3bdfae63"}
Jul 27 21:04:41.208142 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:41.208086    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-219.ec2.internal" podStartSLOduration=3.208064912 podStartE2EDuration="3.208064912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:04:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:41.207210285 +0000 UTC m=+4.574400143" watchObservedRunningTime="2026-07-27 21:04:41.208064912 +0000 UTC m=+4.575254780"
Jul 27 21:04:42.131108 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:42.130203    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5d7vf"
Jul 27 21:04:42.131108 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:42.130333    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5d7vf" podUID="a4af1379-e40a-471d-a011-d50ab76b7f51"
Jul 27 21:04:42.131108 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:42.130698    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nj9np"
Jul 27 21:04:42.131108 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:42.130818    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nj9np" podUID="c5915867-c603-4d51-8417-42aada6603ef"
Jul 27 21:04:42.208967 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:42.208895    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-219.ec2.internal" event={"ID":"abb843a58c4a692b2202031f20068533","Type":"ContainerStarted","Data":"75b6232e9f619eeeff4d6b71fe8c9fef9d5e2ce112bb7dfb3ce4da3973aa5e53"}
Jul 27 21:04:42.635434 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:42.635403    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a4af1379-e40a-471d-a011-d50ab76b7f51-original-pull-secret\") pod \"global-pull-secret-syncer-5d7vf\" (UID: \"a4af1379-e40a-471d-a011-d50ab76b7f51\") " pod="kube-system/global-pull-secret-syncer-5d7vf"
Jul 27 21:04:42.635604 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:42.635455    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3af7400b-f39f-4f41-a492-586171130c30-metrics-certs\") pod \"network-metrics-daemon-wb7n2\" (UID: \"3af7400b-f39f-4f41-a492-586171130c30\") " pod="openshift-multus/network-metrics-daemon-wb7n2"
Jul 27 21:04:42.635604 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:42.635587    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:04:42.635715 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:42.635642    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3af7400b-f39f-4f41-a492-586171130c30-metrics-certs podName:3af7400b-f39f-4f41-a492-586171130c30 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:46.635624009 +0000 UTC m=+10.002813863 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3af7400b-f39f-4f41-a492-586171130c30-metrics-certs") pod "network-metrics-daemon-wb7n2" (UID: "3af7400b-f39f-4f41-a492-586171130c30") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:04:42.636043 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:42.636023    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 21:04:42.636131 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:42.636080    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4af1379-e40a-471d-a011-d50ab76b7f51-original-pull-secret podName:a4af1379-e40a-471d-a011-d50ab76b7f51 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:46.63606339 +0000 UTC m=+10.003253255 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a4af1379-e40a-471d-a011-d50ab76b7f51-original-pull-secret") pod "global-pull-secret-syncer-5d7vf" (UID: "a4af1379-e40a-471d-a011-d50ab76b7f51") : object "kube-system"/"original-pull-secret" not registered
Jul 27 21:04:42.735967 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:42.735935    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmvcb\" (UniqueName: \"kubernetes.io/projected/c5915867-c603-4d51-8417-42aada6603ef-kube-api-access-mmvcb\") pod \"network-check-target-nj9np\" (UID: \"c5915867-c603-4d51-8417-42aada6603ef\") " pod="openshift-network-diagnostics/network-check-target-nj9np"
Jul 27 21:04:42.736140 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:42.736120    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 21:04:42.736195 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:42.736141    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 21:04:42.736195 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:42.736155    2577 projected.go:196] Error preparing data for projected volume kube-api-access-mmvcb for pod openshift-network-diagnostics/network-check-target-nj9np: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:04:42.736297 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:42.736212    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c5915867-c603-4d51-8417-42aada6603ef-kube-api-access-mmvcb podName:c5915867-c603-4d51-8417-42aada6603ef nodeName:}" failed. No retries permitted until 2026-07-27 21:04:46.73619513 +0000 UTC m=+10.103384989 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mmvcb" (UniqueName: "kubernetes.io/projected/c5915867-c603-4d51-8417-42aada6603ef-kube-api-access-mmvcb") pod "network-check-target-nj9np" (UID: "c5915867-c603-4d51-8417-42aada6603ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:04:43.130839 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:43.130741    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb7n2"
Jul 27 21:04:43.131001 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:43.130898    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wb7n2" podUID="3af7400b-f39f-4f41-a492-586171130c30"
Jul 27 21:04:44.130588 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:44.129986    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nj9np"
Jul 27 21:04:44.130588 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:44.130116    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nj9np" podUID="c5915867-c603-4d51-8417-42aada6603ef"
Jul 27 21:04:44.130588 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:44.130176    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5d7vf"
Jul 27 21:04:44.130588 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:44.130253    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5d7vf" podUID="a4af1379-e40a-471d-a011-d50ab76b7f51"
Jul 27 21:04:45.130484 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:45.130426    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb7n2"
Jul 27 21:04:45.130650 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:45.130587    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wb7n2" podUID="3af7400b-f39f-4f41-a492-586171130c30"
Jul 27 21:04:46.130588 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:46.130557    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5d7vf"
Jul 27 21:04:46.130790 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:46.130655    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5d7vf" podUID="a4af1379-e40a-471d-a011-d50ab76b7f51"
Jul 27 21:04:46.130790 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:46.130736    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nj9np"
Jul 27 21:04:46.131189 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:46.130795    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nj9np" podUID="c5915867-c603-4d51-8417-42aada6603ef"
Jul 27 21:04:46.672940 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:46.672207    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a4af1379-e40a-471d-a011-d50ab76b7f51-original-pull-secret\") pod \"global-pull-secret-syncer-5d7vf\" (UID: \"a4af1379-e40a-471d-a011-d50ab76b7f51\") " pod="kube-system/global-pull-secret-syncer-5d7vf"
Jul 27 21:04:46.672940 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:46.672267    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3af7400b-f39f-4f41-a492-586171130c30-metrics-certs\") pod \"network-metrics-daemon-wb7n2\" (UID: \"3af7400b-f39f-4f41-a492-586171130c30\") " pod="openshift-multus/network-metrics-daemon-wb7n2"
Jul 27 21:04:46.672940 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:46.672426    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:04:46.672940 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:46.672487    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3af7400b-f39f-4f41-a492-586171130c30-metrics-certs podName:3af7400b-f39f-4f41-a492-586171130c30 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:54.67246835 +0000 UTC m=+18.039658207 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3af7400b-f39f-4f41-a492-586171130c30-metrics-certs") pod "network-metrics-daemon-wb7n2" (UID: "3af7400b-f39f-4f41-a492-586171130c30") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:04:46.672940 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:46.672867    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 21:04:46.672940 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:46.672918    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4af1379-e40a-471d-a011-d50ab76b7f51-original-pull-secret podName:a4af1379-e40a-471d-a011-d50ab76b7f51 nodeName:}" failed. No retries permitted until 2026-07-27 21:04:54.672904051 +0000 UTC m=+18.040093899 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a4af1379-e40a-471d-a011-d50ab76b7f51-original-pull-secret") pod "global-pull-secret-syncer-5d7vf" (UID: "a4af1379-e40a-471d-a011-d50ab76b7f51") : object "kube-system"/"original-pull-secret" not registered
Jul 27 21:04:46.773352 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:46.773220    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmvcb\" (UniqueName: \"kubernetes.io/projected/c5915867-c603-4d51-8417-42aada6603ef-kube-api-access-mmvcb\") pod \"network-check-target-nj9np\" (UID: \"c5915867-c603-4d51-8417-42aada6603ef\") " pod="openshift-network-diagnostics/network-check-target-nj9np"
Jul 27 21:04:46.773528 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:46.773394    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 21:04:46.773528 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:46.773417    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 21:04:46.773528 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:46.773429    2577 projected.go:196] Error preparing data for projected volume kube-api-access-mmvcb for pod openshift-network-diagnostics/network-check-target-nj9np: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:04:46.773528 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:46.773488    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c5915867-c603-4d51-8417-42aada6603ef-kube-api-access-mmvcb podName:c5915867-c603-4d51-8417-42aada6603ef nodeName:}" failed. No retries permitted until 2026-07-27 21:04:54.77347042 +0000 UTC m=+18.140660275 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mmvcb" (UniqueName: "kubernetes.io/projected/c5915867-c603-4d51-8417-42aada6603ef-kube-api-access-mmvcb") pod "network-check-target-nj9np" (UID: "c5915867-c603-4d51-8417-42aada6603ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:04:47.132112 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:47.131575    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb7n2"
Jul 27 21:04:47.132112 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:47.131698    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wb7n2" podUID="3af7400b-f39f-4f41-a492-586171130c30"
Jul 27 21:04:48.130075 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:48.130040    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5d7vf"
Jul 27 21:04:48.130243 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:48.130180    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5d7vf" podUID="a4af1379-e40a-471d-a011-d50ab76b7f51"
Jul 27 21:04:48.130310 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:48.130259    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nj9np"
Jul 27 21:04:48.130420 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:48.130396    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nj9np" podUID="c5915867-c603-4d51-8417-42aada6603ef"
Jul 27 21:04:49.130150 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:49.130112    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb7n2"
Jul 27 21:04:49.130648 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:49.130271    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wb7n2" podUID="3af7400b-f39f-4f41-a492-586171130c30"
Jul 27 21:04:50.130038 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:50.130005    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nj9np"
Jul 27 21:04:50.130038 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:50.130029    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5d7vf"
Jul 27 21:04:50.130527 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:50.130126    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nj9np" podUID="c5915867-c603-4d51-8417-42aada6603ef"
Jul 27 21:04:50.130527 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:50.130271    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5d7vf" podUID="a4af1379-e40a-471d-a011-d50ab76b7f51"
Jul 27 21:04:51.129999 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:51.129967    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb7n2"
Jul 27 21:04:51.130202 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:51.130116    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wb7n2" podUID="3af7400b-f39f-4f41-a492-586171130c30"
Jul 27 21:04:52.130327 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:52.130297    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nj9np"
Jul 27 21:04:52.130814 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:52.130408    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nj9np" podUID="c5915867-c603-4d51-8417-42aada6603ef"
Jul 27 21:04:52.130814 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:52.130467    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5d7vf"
Jul 27 21:04:52.130814 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:52.130582    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5d7vf" podUID="a4af1379-e40a-471d-a011-d50ab76b7f51"
Jul 27 21:04:53.130471 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:53.130432    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb7n2"
Jul 27 21:04:53.130890 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:53.130573    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wb7n2" podUID="3af7400b-f39f-4f41-a492-586171130c30"
Jul 27 21:04:54.130704 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:54.130680    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nj9np"
Jul 27 21:04:54.131042 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:54.130779    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nj9np" podUID="c5915867-c603-4d51-8417-42aada6603ef"
Jul 27 21:04:54.131042 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:54.130824    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5d7vf"
Jul 27 21:04:54.131042 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:54.130909    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5d7vf" podUID="a4af1379-e40a-471d-a011-d50ab76b7f51"
Jul 27 21:04:54.731283 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:54.731248    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a4af1379-e40a-471d-a011-d50ab76b7f51-original-pull-secret\") pod \"global-pull-secret-syncer-5d7vf\" (UID: \"a4af1379-e40a-471d-a011-d50ab76b7f51\") " pod="kube-system/global-pull-secret-syncer-5d7vf"
Jul 27 21:04:54.731437 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:54.731293    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3af7400b-f39f-4f41-a492-586171130c30-metrics-certs\") pod \"network-metrics-daemon-wb7n2\" (UID: \"3af7400b-f39f-4f41-a492-586171130c30\") " pod="openshift-multus/network-metrics-daemon-wb7n2"
Jul 27 21:04:54.731437 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:54.731395    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:04:54.731546 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:54.731453    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3af7400b-f39f-4f41-a492-586171130c30-metrics-certs podName:3af7400b-f39f-4f41-a492-586171130c30 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:10.731439581 +0000 UTC m=+34.098629425 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3af7400b-f39f-4f41-a492-586171130c30-metrics-certs") pod "network-metrics-daemon-wb7n2" (UID: "3af7400b-f39f-4f41-a492-586171130c30") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:04:54.731546 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:54.731395    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 21:04:54.731650 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:54.731549    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4af1379-e40a-471d-a011-d50ab76b7f51-original-pull-secret podName:a4af1379-e40a-471d-a011-d50ab76b7f51 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:10.731527928 +0000 UTC m=+34.098717776 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a4af1379-e40a-471d-a011-d50ab76b7f51-original-pull-secret") pod "global-pull-secret-syncer-5d7vf" (UID: "a4af1379-e40a-471d-a011-d50ab76b7f51") : object "kube-system"/"original-pull-secret" not registered
Jul 27 21:04:54.832435 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:54.832396    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmvcb\" (UniqueName: \"kubernetes.io/projected/c5915867-c603-4d51-8417-42aada6603ef-kube-api-access-mmvcb\") pod \"network-check-target-nj9np\" (UID: \"c5915867-c603-4d51-8417-42aada6603ef\") " pod="openshift-network-diagnostics/network-check-target-nj9np"
Jul 27 21:04:54.832662 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:54.832541    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 21:04:54.832662 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:54.832560    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 21:04:54.832662 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:54.832572    2577 projected.go:196] Error preparing data for projected volume kube-api-access-mmvcb for pod openshift-network-diagnostics/network-check-target-nj9np: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:04:54.832662 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:54.832628    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c5915867-c603-4d51-8417-42aada6603ef-kube-api-access-mmvcb podName:c5915867-c603-4d51-8417-42aada6603ef nodeName:}" failed. No retries permitted until 2026-07-27 21:05:10.832612714 +0000 UTC m=+34.199802558 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mmvcb" (UniqueName: "kubernetes.io/projected/c5915867-c603-4d51-8417-42aada6603ef-kube-api-access-mmvcb") pod "network-check-target-nj9np" (UID: "c5915867-c603-4d51-8417-42aada6603ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:04:55.130704 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:55.130629    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb7n2"
Jul 27 21:04:55.130871 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:55.130743    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wb7n2" podUID="3af7400b-f39f-4f41-a492-586171130c30"
Jul 27 21:04:56.129942 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:56.129904    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nj9np"
Jul 27 21:04:56.130111 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:56.130046    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nj9np" podUID="c5915867-c603-4d51-8417-42aada6603ef"
Jul 27 21:04:56.130509 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:56.129914    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5d7vf"
Jul 27 21:04:56.130509 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:56.130466    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5d7vf" podUID="a4af1379-e40a-471d-a011-d50ab76b7f51"
Jul 27 21:04:57.131259 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:57.131229    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb7n2"
Jul 27 21:04:57.131675 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:57.131323    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wb7n2" podUID="3af7400b-f39f-4f41-a492-586171130c30"
Jul 27 21:04:58.130888 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:58.130855    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nj9np"
Jul 27 21:04:58.131037 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:58.130855    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5d7vf"
Jul 27 21:04:58.131037 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:58.130955    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nj9np" podUID="c5915867-c603-4d51-8417-42aada6603ef"
Jul 27 21:04:58.131037 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:58.131027    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5d7vf" podUID="a4af1379-e40a-471d-a011-d50ab76b7f51"
Jul 27 21:04:59.130464 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:59.130299    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb7n2"
Jul 27 21:04:59.131134 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:04:59.130556    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wb7n2" podUID="3af7400b-f39f-4f41-a492-586171130c30"
Jul 27 21:04:59.239374 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:59.239344    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-62zwj" event={"ID":"02c8e99e-0fb6-488d-a75a-fd9efe3cc4ab","Type":"ContainerStarted","Data":"dfb67e2debd2a2906f00e5782dbc17d87671df22626787a52ca9cca2aa540c87"}
Jul 27 21:04:59.241105 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:59.241083    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xgbjk" event={"ID":"7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c","Type":"ContainerStarted","Data":"e16ee22187df0deb44cd2eea43ed9fab5a7237cf48d5378cf5f96cf65c56eb8d"}
Jul 27 21:04:59.242840 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:59.242786    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d68g" event={"ID":"b2bf895f-3920-43a8-977c-fffbd75796e4","Type":"ContainerStarted","Data":"97dfe550ce142bc917f6dd5ac55578fabe76ba2c3a161a487a9381198b7b0391"}
Jul 27 21:04:59.244297 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:59.244252    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-frmv7" event={"ID":"e610bffe-2c4d-4577-8a48-4849b0d3266d","Type":"ContainerStarted","Data":"b46b35f6c7b9517bf10dc0b84149c3bf59b430448a3b5191f75769c4d17206b9"}
Jul 27 21:04:59.245978 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:59.245940    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-b9b6s" event={"ID":"0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6","Type":"ContainerStarted","Data":"5541664a4739a07420a28fb07870329f05417c29c94c211606aa4bb5b63f97a4"}
Jul 27 21:04:59.249601 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:59.249577    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pftzz" event={"ID":"72112606-3839-4478-bac9-dfc58a7809f8","Type":"ContainerStarted","Data":"ea92f818a846e99daccb041e1f17dfc6eec35bab25157d24437fc53183582237"}
Jul 27 21:04:59.249683 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:59.249653    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pftzz" event={"ID":"72112606-3839-4478-bac9-dfc58a7809f8","Type":"ContainerStarted","Data":"8cdf4c3871da16d38931f6e8098dd1fbd27b76f04e0a077b53f5529e884e5936"}
Jul 27 21:04:59.249730 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:59.249706    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pftzz" event={"ID":"72112606-3839-4478-bac9-dfc58a7809f8","Type":"ContainerStarted","Data":"25d4228c76544a62bd744b0e4cd2729c8d0c11577739a0bf33b578e3baf47870"}
Jul 27 21:04:59.249788 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:59.249752    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pftzz" event={"ID":"72112606-3839-4478-bac9-dfc58a7809f8","Type":"ContainerStarted","Data":"0da41388dad6be02920fa00897e8a2b6eeacc83b4d21706c9f47684cadcea68f"}
Jul 27 21:04:59.249837 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:59.249824    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pftzz" event={"ID":"72112606-3839-4478-bac9-dfc58a7809f8","Type":"ContainerStarted","Data":"78b71301ffb2a6be17610cc961b301aaa965303b2db8c1f11a52c1d83fd2d6a0"}
Jul 27 21:04:59.249887 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:59.249873    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pftzz" event={"ID":"72112606-3839-4478-bac9-dfc58a7809f8","Type":"ContainerStarted","Data":"e37d5b00cb4799777c3ea729ed0e2cf0fadd7fb47bd4327119472a5710074693"}
Jul 27 21:04:59.251667 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:59.251640    2577 generic.go:356] "Generic (PLEG): container finished" podID="fd05d22e-5585-4f39-8597-4fc6978df793" containerID="29afba338d91ae523e2f7b2a2d8ffdda6f6cef1ca92b437b97243e77a473949f" exitCode=0
Jul 27 21:04:59.251740 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:59.251725    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-667ww" event={"ID":"fd05d22e-5585-4f39-8597-4fc6978df793","Type":"ContainerDied","Data":"29afba338d91ae523e2f7b2a2d8ffdda6f6cef1ca92b437b97243e77a473949f"}
Jul 27 21:04:59.253325 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:59.253254    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dtklb" event={"ID":"0b84c165-5dfa-416c-bec6-28c9d55197ab","Type":"ContainerStarted","Data":"eef2e80262f89852c9c42a2f01b3279e16b2259acfeb15400461a67451c650cd"}
Jul 27 21:04:59.270396 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:59.270350    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-219.ec2.internal" podStartSLOduration=21.270334555 podStartE2EDuration="21.270334555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:04:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:42.226159793 +0000 UTC m=+5.593349659" watchObservedRunningTime="2026-07-27 21:04:59.270334555 +0000 UTC m=+22.637524423"
Jul 27 21:04:59.300015 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:59.299967    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-62zwj" podStartSLOduration=3.86555304 podStartE2EDuration="22.299953929s" totalImagesPullingTime="18.434400889s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:04:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:59.271666007 +0000 UTC m=+22.638855874" watchObservedRunningTime="2026-07-27 21:04:59.299953929 +0000 UTC m=+22.667143795"
Jul 27 21:04:59.300294 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:59.300267    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-frmv7" podStartSLOduration=3.869332517 podStartE2EDuration="22.30022764s" totalImagesPullingTime="18.430895123s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:04:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:59.299831998 +0000 UTC m=+22.667021866" watchObservedRunningTime="2026-07-27 21:04:59.30022764 +0000 UTC m=+22.667417696"
Jul 27 21:04:59.378987 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:59.378945    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-b9b6s" podStartSLOduration=8.398103849 podStartE2EDuration="22.378931255s" totalImagesPullingTime="13.980827406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:04:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:59.359467755 +0000 UTC m=+22.726657620" watchObservedRunningTime="2026-07-27 21:04:59.378931255 +0000 UTC m=+22.746121120"
Jul 27 21:04:59.379393 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:59.379368    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-xgbjk" podStartSLOduration=3.922816707 podStartE2EDuration="22.37936181s" totalImagesPullingTime="18.456545103s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:04:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:59.379197639 +0000 UTC m=+22.746387505" watchObservedRunningTime="2026-07-27 21:04:59.37936181 +0000 UTC m=+22.746551676"
Jul 27 21:04:59.395671 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:59.395602    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dtklb" podStartSLOduration=3.957576353 podStartE2EDuration="22.395584618s" totalImagesPullingTime="18.438008265s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:04:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:04:59.395044092 +0000 UTC m=+22.762233956" watchObservedRunningTime="2026-07-27 21:04:59.395584618 +0000 UTC m=+22.762774483"
Jul 27 21:04:59.995407 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:04:59.995386    2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 27 21:05:00.059884 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:00.059799    2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-27T21:04:59.995402659Z","UUID":"f70395e9-3a27-4e49-943a-ee5263cda14e","Handler":null,"Name":"","Endpoint":""}
Jul 27 21:05:00.061385 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:00.061362    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 27 21:05:00.061493 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:00.061407    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 27 21:05:00.130491 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:00.130420    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5d7vf"
Jul 27 21:05:00.130958 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:00.130420    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nj9np"
Jul 27 21:05:00.130958 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:00.130543    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5d7vf" podUID="a4af1379-e40a-471d-a011-d50ab76b7f51"
Jul 27 21:05:00.130958 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:00.130621    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nj9np" podUID="c5915867-c603-4d51-8417-42aada6603ef"
Jul 27 21:05:00.258177 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:00.258142    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d68g" event={"ID":"b2bf895f-3920-43a8-977c-fffbd75796e4","Type":"ContainerStarted","Data":"38bd6b9f1a540b708631742ea99aabf6a3da71492d48d4a5972e088651825d1a"}
Jul 27 21:05:00.260268 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:00.259821    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fdrp9" event={"ID":"c35bbe72-c030-4b9d-aece-459d907893dc","Type":"ContainerStarted","Data":"efa4549cd20fe6fabe1447c576f53ba048ce5de5087d6894a8aa3d65938cb4e5"}
Jul 27 21:05:00.277392 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:00.277334    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-fdrp9" podStartSLOduration=4.835391865 podStartE2EDuration="23.27731379s" totalImagesPullingTime="18.441921925s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:04:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:05:00.277236119 +0000 UTC m=+23.644425985" watchObservedRunningTime="2026-07-27 21:05:00.27731379 +0000 UTC m=+23.644503656"
Jul 27 21:05:01.130220 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:01.130187    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb7n2"
Jul 27 21:05:01.130390 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:01.130329    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wb7n2" podUID="3af7400b-f39f-4f41-a492-586171130c30"
Jul 27 21:05:01.264968 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:01.264888    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d68g" event={"ID":"b2bf895f-3920-43a8-977c-fffbd75796e4","Type":"ContainerStarted","Data":"6c61eb51581cb7a6fd860506301f3f2b9023850f7c564da15fed2b531aeaf7d9"}
Jul 27 21:05:01.284815 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:01.284698    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d68g" podStartSLOduration=3.762655058 podStartE2EDuration="24.28462194s" totalImagesPullingTime="20.521966882s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:04:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:05:01.284181901 +0000 UTC m=+24.651371768" watchObservedRunningTime="2026-07-27 21:05:01.28462194 +0000 UTC m=+24.651811805"
Jul 27 21:05:02.130603 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:02.130437    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5d7vf"
Jul 27 21:05:02.130839 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:02.130443    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nj9np"
Jul 27 21:05:02.130839 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:02.130679    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5d7vf" podUID="a4af1379-e40a-471d-a011-d50ab76b7f51"
Jul 27 21:05:02.130839 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:02.130811    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nj9np" podUID="c5915867-c603-4d51-8417-42aada6603ef"
Jul 27 21:05:02.270826 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:02.270791    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pftzz" event={"ID":"72112606-3839-4478-bac9-dfc58a7809f8","Type":"ContainerStarted","Data":"9ad9721364b0b6316b1a47aa02d80e9ae326a1283220a2e2ef4652c08ae6d72b"}
Jul 27 21:05:03.130780 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:03.130737    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb7n2"
Jul 27 21:05:03.130940 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:03.130881    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wb7n2" podUID="3af7400b-f39f-4f41-a492-586171130c30"
Jul 27 21:05:04.060513 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:04.060312    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-b9b6s"
Jul 27 21:05:04.061076 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:04.060840    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-b9b6s"
Jul 27 21:05:04.130610 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:04.130586    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nj9np"
Jul 27 21:05:04.130735 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:04.130586    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5d7vf"
Jul 27 21:05:04.130735 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:04.130691    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nj9np" podUID="c5915867-c603-4d51-8417-42aada6603ef"
Jul 27 21:05:04.130854 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:04.130747    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5d7vf" podUID="a4af1379-e40a-471d-a011-d50ab76b7f51"
Jul 27 21:05:04.278214 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:04.278180    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pftzz" event={"ID":"72112606-3839-4478-bac9-dfc58a7809f8","Type":"ContainerStarted","Data":"a6efc7b5068fe83aeb45c7a951d73758cea004f2e28d6b1597c4c5925488ac61"}
Jul 27 21:05:04.278379 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:04.278269    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:05:04.278379 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:04.278324    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:05:04.278379 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:04.278367    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:05:04.279999 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:04.279972    2577 generic.go:356] "Generic (PLEG): container finished" podID="fd05d22e-5585-4f39-8597-4fc6978df793" containerID="aaf4006652ac8cb5f2eec5225f85a35261bb41fd7c7ef2a27f5d5bbc27d5c70a" exitCode=0
Jul 27 21:05:04.280112 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:04.280052    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-667ww" event={"ID":"fd05d22e-5585-4f39-8597-4fc6978df793","Type":"ContainerDied","Data":"aaf4006652ac8cb5f2eec5225f85a35261bb41fd7c7ef2a27f5d5bbc27d5c70a"}
Jul 27 21:05:04.280214 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:04.280200    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-b9b6s"
Jul 27 21:05:04.280723 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:04.280653    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-b9b6s"
Jul 27 21:05:04.294413 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:04.294395    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:05:04.294500 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:04.294476    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:05:04.306120 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:04.306083    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-pftzz" podStartSLOduration=8.755936394 podStartE2EDuration="27.306071716s" totalImagesPullingTime="18.550135322s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:04:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:05:04.305603611 +0000 UTC m=+27.672793478" watchObservedRunningTime="2026-07-27 21:05:04.306071716 +0000 UTC m=+27.673261582"
Jul 27 21:05:05.130683 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:05.130650    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb7n2"
Jul 27 21:05:05.131076 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:05.130798    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wb7n2" podUID="3af7400b-f39f-4f41-a492-586171130c30"
Jul 27 21:05:05.674416 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:05.674202    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5d7vf"]
Jul 27 21:05:05.674590 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:05.674570    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5d7vf"
Jul 27 21:05:05.674688 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:05.674663    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5d7vf" podUID="a4af1379-e40a-471d-a011-d50ab76b7f51"
Jul 27 21:05:05.676320 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:05.676296    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nj9np"]
Jul 27 21:05:05.676962 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:05.676942    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nj9np"
Jul 27 21:05:05.677068 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:05.677047    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nj9np" podUID="c5915867-c603-4d51-8417-42aada6603ef"
Jul 27 21:05:05.677139 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:05.677099    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wb7n2"]
Jul 27 21:05:05.677193 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:05.677183    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb7n2"
Jul 27 21:05:05.677279 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:05.677264    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wb7n2" podUID="3af7400b-f39f-4f41-a492-586171130c30"
Jul 27 21:05:06.285169 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:06.285138    2577 generic.go:356] "Generic (PLEG): container finished" podID="fd05d22e-5585-4f39-8597-4fc6978df793" containerID="5053d1adf2162a74854f0b0bea06c31cf4ae9e079af9e998a47be53996a84605" exitCode=0
Jul 27 21:05:06.285541 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:06.285224    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-667ww" event={"ID":"fd05d22e-5585-4f39-8597-4fc6978df793","Type":"ContainerDied","Data":"5053d1adf2162a74854f0b0bea06c31cf4ae9e079af9e998a47be53996a84605"}
Jul 27 21:05:07.131012 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:07.130985    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nj9np"
Jul 27 21:05:07.131217 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:07.131056    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nj9np" podUID="c5915867-c603-4d51-8417-42aada6603ef"
Jul 27 21:05:07.131217 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:07.131068    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb7n2"
Jul 27 21:05:07.131217 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:07.131180    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wb7n2" podUID="3af7400b-f39f-4f41-a492-586171130c30"
Jul 27 21:05:08.130309 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:08.130278    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5d7vf"
Jul 27 21:05:08.130911 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:08.130373    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5d7vf" podUID="a4af1379-e40a-471d-a011-d50ab76b7f51"
Jul 27 21:05:08.292012 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:08.291979    2577 generic.go:356] "Generic (PLEG): container finished" podID="fd05d22e-5585-4f39-8597-4fc6978df793" containerID="c3f852bee3a77ad8677e74c0acc4c8c3cc96e768fbac67bcef70ffbd3a3847d2" exitCode=0
Jul 27 21:05:08.292169 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:08.292034    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-667ww" event={"ID":"fd05d22e-5585-4f39-8597-4fc6978df793","Type":"ContainerDied","Data":"c3f852bee3a77ad8677e74c0acc4c8c3cc96e768fbac67bcef70ffbd3a3847d2"}
Jul 27 21:05:09.130426 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:09.130402    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nj9np"
Jul 27 21:05:09.130426 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:09.130414    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb7n2"
Jul 27 21:05:09.130906 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:09.130515    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nj9np" podUID="c5915867-c603-4d51-8417-42aada6603ef"
Jul 27 21:05:09.130906 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:09.130641    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wb7n2" podUID="3af7400b-f39f-4f41-a492-586171130c30"
Jul 27 21:05:10.129924 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:10.129891    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5d7vf"
Jul 27 21:05:10.130091 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:10.130026    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5d7vf" podUID="a4af1379-e40a-471d-a011-d50ab76b7f51"
Jul 27 21:05:10.469023 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:10.468993    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:10.570004 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:10.569967    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:10.759571 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:10.759477    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a4af1379-e40a-471d-a011-d50ab76b7f51-original-pull-secret\") pod \"global-pull-secret-syncer-5d7vf\" (UID: \"a4af1379-e40a-471d-a011-d50ab76b7f51\") " pod="kube-system/global-pull-secret-syncer-5d7vf"
Jul 27 21:05:10.759571 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:10.759523    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3af7400b-f39f-4f41-a492-586171130c30-metrics-certs\") pod \"network-metrics-daemon-wb7n2\" (UID: \"3af7400b-f39f-4f41-a492-586171130c30\") " pod="openshift-multus/network-metrics-daemon-wb7n2"
Jul 27 21:05:10.759824 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:10.759653    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 21:05:10.759824 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:10.759744    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4af1379-e40a-471d-a011-d50ab76b7f51-original-pull-secret podName:a4af1379-e40a-471d-a011-d50ab76b7f51 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:42.759724748 +0000 UTC m=+66.126914609 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a4af1379-e40a-471d-a011-d50ab76b7f51-original-pull-secret") pod "global-pull-secret-syncer-5d7vf" (UID: "a4af1379-e40a-471d-a011-d50ab76b7f51") : object "kube-system"/"original-pull-secret" not registered
Jul 27 21:05:10.759824 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:10.759653    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:05:10.759960 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:10.759855    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3af7400b-f39f-4f41-a492-586171130c30-metrics-certs podName:3af7400b-f39f-4f41-a492-586171130c30 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:42.759834145 +0000 UTC m=+66.127024009 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3af7400b-f39f-4f41-a492-586171130c30-metrics-certs") pod "network-metrics-daemon-wb7n2" (UID: "3af7400b-f39f-4f41-a492-586171130c30") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:05:10.860544 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:10.860508    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmvcb\" (UniqueName: \"kubernetes.io/projected/c5915867-c603-4d51-8417-42aada6603ef-kube-api-access-mmvcb\") pod \"network-check-target-nj9np\" (UID: \"c5915867-c603-4d51-8417-42aada6603ef\") " pod="openshift-network-diagnostics/network-check-target-nj9np"
Jul 27 21:05:10.860683 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:10.860666    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 21:05:10.860721 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:10.860686    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 21:05:10.860721 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:10.860697    2577 projected.go:196] Error preparing data for projected volume kube-api-access-mmvcb for pod openshift-network-diagnostics/network-check-target-nj9np: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:05:10.860806 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:10.860745    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c5915867-c603-4d51-8417-42aada6603ef-kube-api-access-mmvcb podName:c5915867-c603-4d51-8417-42aada6603ef nodeName:}" failed. No retries permitted until 2026-07-27 21:05:42.860732673 +0000 UTC m=+66.227922521 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-mmvcb" (UniqueName: "kubernetes.io/projected/c5915867-c603-4d51-8417-42aada6603ef-kube-api-access-mmvcb") pod "network-check-target-nj9np" (UID: "c5915867-c603-4d51-8417-42aada6603ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:05:10.924092 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:10.924059    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:11.031653 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:11.031569    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:11.130588 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:11.130555    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nj9np"
Jul 27 21:05:11.130722 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:11.130555    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb7n2"
Jul 27 21:05:11.130722 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:11.130683    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nj9np" podUID="c5915867-c603-4d51-8417-42aada6603ef"
Jul 27 21:05:11.130905 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:11.130782    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wb7n2" podUID="3af7400b-f39f-4f41-a492-586171130c30"
Jul 27 21:05:11.495853 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:11.495794    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-219.ec2.internal" event="NodeReady"
Jul 27 21:05:11.496339 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:11.495965    2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 27 21:05:11.585834 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:11.585802    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4dq86"]
Jul 27 21:05:11.588581 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:11.588486    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4dq86"
Jul 27 21:05:11.589365 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:11.589344    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jk8jc"]
Jul 27 21:05:11.590950 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:11.590933    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jk8jc"
Jul 27 21:05:11.600983 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:11.600961    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 27 21:05:11.601578 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:11.601558    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 27 21:05:11.601739 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:11.601721    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-sfq8h\""
Jul 27 21:05:11.602001 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:11.601981    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6vzp2\""
Jul 27 21:05:11.602616 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:11.602598    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 27 21:05:11.602719 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:11.602637    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 27 21:05:11.603089 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:11.603043    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 27 21:05:11.615707 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:11.615686    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4dq86"]
Jul 27 21:05:11.616384 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:11.616362    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jk8jc"]
Jul 27 21:05:11.766664 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:11.766583    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8931191d-6ec9-492d-a5e1-3293972c77bf-cert\") pod \"ingress-canary-jk8jc\" (UID: \"8931191d-6ec9-492d-a5e1-3293972c77bf\") " pod="openshift-ingress-canary/ingress-canary-jk8jc"
Jul 27 21:05:11.766664 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:11.766620    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c9aaedde-8f0d-4cf9-8210-f6d41d5a9138-config-volume\") pod \"dns-default-4dq86\" (UID: \"c9aaedde-8f0d-4cf9-8210-f6d41d5a9138\") " pod="openshift-dns/dns-default-4dq86"
Jul 27 21:05:11.766911 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:11.766691    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrmzf\" (UniqueName: \"kubernetes.io/projected/8931191d-6ec9-492d-a5e1-3293972c77bf-kube-api-access-zrmzf\") pod \"ingress-canary-jk8jc\" (UID: \"8931191d-6ec9-492d-a5e1-3293972c77bf\") " pod="openshift-ingress-canary/ingress-canary-jk8jc"
Jul 27 21:05:11.766911 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:11.766712    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c9aaedde-8f0d-4cf9-8210-f6d41d5a9138-metrics-tls\") pod \"dns-default-4dq86\" (UID: \"c9aaedde-8f0d-4cf9-8210-f6d41d5a9138\") " pod="openshift-dns/dns-default-4dq86"
Jul 27 21:05:11.766911 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:11.766727    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9aaedde-8f0d-4cf9-8210-f6d41d5a9138-tmp-dir\") pod \"dns-default-4dq86\" (UID: \"c9aaedde-8f0d-4cf9-8210-f6d41d5a9138\") " pod="openshift-dns/dns-default-4dq86"
Jul 27 21:05:11.766911 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:11.766840    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qfhm\" (UniqueName: \"kubernetes.io/projected/c9aaedde-8f0d-4cf9-8210-f6d41d5a9138-kube-api-access-6qfhm\") pod \"dns-default-4dq86\" (UID: \"c9aaedde-8f0d-4cf9-8210-f6d41d5a9138\") " pod="openshift-dns/dns-default-4dq86"
Jul 27 21:05:11.867871 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:11.867842    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrmzf\" (UniqueName: \"kubernetes.io/projected/8931191d-6ec9-492d-a5e1-3293972c77bf-kube-api-access-zrmzf\") pod \"ingress-canary-jk8jc\" (UID: \"8931191d-6ec9-492d-a5e1-3293972c77bf\") " pod="openshift-ingress-canary/ingress-canary-jk8jc"
Jul 27 21:05:11.867871 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:11.867873    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c9aaedde-8f0d-4cf9-8210-f6d41d5a9138-metrics-tls\") pod \"dns-default-4dq86\" (UID: \"c9aaedde-8f0d-4cf9-8210-f6d41d5a9138\") " pod="openshift-dns/dns-default-4dq86"
Jul 27 21:05:11.868079 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:11.867890    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9aaedde-8f0d-4cf9-8210-f6d41d5a9138-tmp-dir\") pod \"dns-default-4dq86\" (UID: \"c9aaedde-8f0d-4cf9-8210-f6d41d5a9138\") " pod="openshift-dns/dns-default-4dq86"
Jul 27 21:05:11.868079 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:11.867939    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qfhm\" (UniqueName: \"kubernetes.io/projected/c9aaedde-8f0d-4cf9-8210-f6d41d5a9138-kube-api-access-6qfhm\") pod \"dns-default-4dq86\" (UID: \"c9aaedde-8f0d-4cf9-8210-f6d41d5a9138\") " pod="openshift-dns/dns-default-4dq86"
Jul 27 21:05:11.868079 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:11.867966    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8931191d-6ec9-492d-a5e1-3293972c77bf-cert\") pod \"ingress-canary-jk8jc\" (UID: \"8931191d-6ec9-492d-a5e1-3293972c77bf\") " pod="openshift-ingress-canary/ingress-canary-jk8jc"
Jul 27 21:05:11.868079 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:11.867986    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c9aaedde-8f0d-4cf9-8210-f6d41d5a9138-config-volume\") pod \"dns-default-4dq86\" (UID: \"c9aaedde-8f0d-4cf9-8210-f6d41d5a9138\") " pod="openshift-dns/dns-default-4dq86"
Jul 27 21:05:11.868380 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:11.868351    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 21:05:11.868510 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:11.868433    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9aaedde-8f0d-4cf9-8210-f6d41d5a9138-metrics-tls podName:c9aaedde-8f0d-4cf9-8210-f6d41d5a9138 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:12.368413642 +0000 UTC m=+35.735603491 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c9aaedde-8f0d-4cf9-8210-f6d41d5a9138-metrics-tls") pod "dns-default-4dq86" (UID: "c9aaedde-8f0d-4cf9-8210-f6d41d5a9138") : secret "dns-default-metrics-tls" not found
Jul 27 21:05:11.868510 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:11.868502    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9aaedde-8f0d-4cf9-8210-f6d41d5a9138-tmp-dir\") pod \"dns-default-4dq86\" (UID: \"c9aaedde-8f0d-4cf9-8210-f6d41d5a9138\") " pod="openshift-dns/dns-default-4dq86"
Jul 27 21:05:11.868628 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:11.868612    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c9aaedde-8f0d-4cf9-8210-f6d41d5a9138-config-volume\") pod \"dns-default-4dq86\" (UID: \"c9aaedde-8f0d-4cf9-8210-f6d41d5a9138\") " pod="openshift-dns/dns-default-4dq86"
Jul 27 21:05:11.872730 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:11.872710    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8931191d-6ec9-492d-a5e1-3293972c77bf-cert\") pod \"ingress-canary-jk8jc\" (UID: \"8931191d-6ec9-492d-a5e1-3293972c77bf\") " pod="openshift-ingress-canary/ingress-canary-jk8jc"
Jul 27 21:05:11.877958 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:11.877914    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qfhm\" (UniqueName: \"kubernetes.io/projected/c9aaedde-8f0d-4cf9-8210-f6d41d5a9138-kube-api-access-6qfhm\") pod \"dns-default-4dq86\" (UID: \"c9aaedde-8f0d-4cf9-8210-f6d41d5a9138\") " pod="openshift-dns/dns-default-4dq86"
Jul 27 21:05:11.878446 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:11.878426    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrmzf\" (UniqueName: \"kubernetes.io/projected/8931191d-6ec9-492d-a5e1-3293972c77bf-kube-api-access-zrmzf\") pod \"ingress-canary-jk8jc\" (UID: \"8931191d-6ec9-492d-a5e1-3293972c77bf\") " pod="openshift-ingress-canary/ingress-canary-jk8jc"
Jul 27 21:05:11.905355 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:11.905333    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jk8jc"
Jul 27 21:05:12.130660 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:12.130401    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5d7vf"
Jul 27 21:05:12.133338 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:12.133318    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 27 21:05:12.372075 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:12.372042    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c9aaedde-8f0d-4cf9-8210-f6d41d5a9138-metrics-tls\") pod \"dns-default-4dq86\" (UID: \"c9aaedde-8f0d-4cf9-8210-f6d41d5a9138\") " pod="openshift-dns/dns-default-4dq86"
Jul 27 21:05:12.372184 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:12.372151    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 21:05:12.372222 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:12.372212    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9aaedde-8f0d-4cf9-8210-f6d41d5a9138-metrics-tls podName:c9aaedde-8f0d-4cf9-8210-f6d41d5a9138 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:13.372195185 +0000 UTC m=+36.739385029 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c9aaedde-8f0d-4cf9-8210-f6d41d5a9138-metrics-tls") pod "dns-default-4dq86" (UID: "c9aaedde-8f0d-4cf9-8210-f6d41d5a9138") : secret "dns-default-metrics-tls" not found
Jul 27 21:05:12.423419 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:12.423398    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jk8jc"]
Jul 27 21:05:12.427777 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:05:12.427745    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8931191d_6ec9_492d_a5e1_3293972c77bf.slice/crio-d94f8efa29755e74ea828b187ca582740e737f87d8e638dfcccad7aca9ceb4eb WatchSource:0}: Error finding container d94f8efa29755e74ea828b187ca582740e737f87d8e638dfcccad7aca9ceb4eb: Status 404 returned error can't find the container with id d94f8efa29755e74ea828b187ca582740e737f87d8e638dfcccad7aca9ceb4eb
Jul 27 21:05:13.130115 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:13.130083    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb7n2"
Jul 27 21:05:13.130981 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:13.130879    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nj9np"
Jul 27 21:05:13.133794 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:13.133503    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vs64j\""
Jul 27 21:05:13.133913 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:13.133791    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 27 21:05:13.134336 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:13.134251    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7jdqc\""
Jul 27 21:05:13.134336 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:13.134289    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 27 21:05:13.135045 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:13.135020    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 27 21:05:13.169059 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:13.169040    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:13.305184 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:13.305146    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jk8jc" event={"ID":"8931191d-6ec9-492d-a5e1-3293972c77bf","Type":"ContainerStarted","Data":"d94f8efa29755e74ea828b187ca582740e737f87d8e638dfcccad7aca9ceb4eb"}
Jul 27 21:05:13.308155 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:13.308128    2577 generic.go:356] "Generic (PLEG): container finished" podID="fd05d22e-5585-4f39-8597-4fc6978df793" containerID="26ba86a28f621fc3533517e7c8ebf6353a3049741fcb3a917080ee0607e411ac" exitCode=0
Jul 27 21:05:13.308285 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:13.308171    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-667ww" event={"ID":"fd05d22e-5585-4f39-8597-4fc6978df793","Type":"ContainerDied","Data":"26ba86a28f621fc3533517e7c8ebf6353a3049741fcb3a917080ee0607e411ac"}
Jul 27 21:05:13.379740 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:13.379719    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c9aaedde-8f0d-4cf9-8210-f6d41d5a9138-metrics-tls\") pod \"dns-default-4dq86\" (UID: \"c9aaedde-8f0d-4cf9-8210-f6d41d5a9138\") " pod="openshift-dns/dns-default-4dq86"
Jul 27 21:05:13.379911 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:13.379891    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 21:05:13.379981 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:13.379961    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9aaedde-8f0d-4cf9-8210-f6d41d5a9138-metrics-tls podName:c9aaedde-8f0d-4cf9-8210-f6d41d5a9138 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:15.379941558 +0000 UTC m=+38.747131415 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c9aaedde-8f0d-4cf9-8210-f6d41d5a9138-metrics-tls") pod "dns-default-4dq86" (UID: "c9aaedde-8f0d-4cf9-8210-f6d41d5a9138") : secret "dns-default-metrics-tls" not found
Jul 27 21:05:14.313950 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:14.313916    2577 generic.go:356] "Generic (PLEG): container finished" podID="fd05d22e-5585-4f39-8597-4fc6978df793" containerID="b39466efe78b72f80fa854feee9e6fa4f8b0d790acad4db89d9358ab25df4fa5" exitCode=0
Jul 27 21:05:14.314370 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:14.313960    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-667ww" event={"ID":"fd05d22e-5585-4f39-8597-4fc6978df793","Type":"ContainerDied","Data":"b39466efe78b72f80fa854feee9e6fa4f8b0d790acad4db89d9358ab25df4fa5"}
Jul 27 21:05:15.318231 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:15.318019    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jk8jc" event={"ID":"8931191d-6ec9-492d-a5e1-3293972c77bf","Type":"ContainerStarted","Data":"61c939f619b4d5749954e8bbe2c28d2ee396d707e70715b26b78fe09a32ef92e"}
Jul 27 21:05:15.321010 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:15.320987    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-667ww" event={"ID":"fd05d22e-5585-4f39-8597-4fc6978df793","Type":"ContainerStarted","Data":"e43f90fb05e1903aebb56e7e32ba2abf13a18148f148b27fd5de1c9105fd706d"}
Jul 27 21:05:15.336045 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:15.336007    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jk8jc" podStartSLOduration=2.1709622729999998 podStartE2EDuration="4.335994638s" totalImagesPullingTime="2.165032365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:05:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:05:15.335496457 +0000 UTC m=+38.702686332" watchObservedRunningTime="2026-07-27 21:05:15.335994638 +0000 UTC m=+38.703184503"
Jul 27 21:05:15.358873 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:15.358831    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-667ww" podStartSLOduration=9.212375658 podStartE2EDuration="38.358819413s" totalImagesPullingTime="29.146443755s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:04:37 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:05:15.357717457 +0000 UTC m=+38.724907323" watchObservedRunningTime="2026-07-27 21:05:15.358819413 +0000 UTC m=+38.726009277"
Jul 27 21:05:15.395589 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:15.395570    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c9aaedde-8f0d-4cf9-8210-f6d41d5a9138-metrics-tls\") pod \"dns-default-4dq86\" (UID: \"c9aaedde-8f0d-4cf9-8210-f6d41d5a9138\") " pod="openshift-dns/dns-default-4dq86"
Jul 27 21:05:15.395682 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:15.395670    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 21:05:15.395728 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:15.395718    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9aaedde-8f0d-4cf9-8210-f6d41d5a9138-metrics-tls podName:c9aaedde-8f0d-4cf9-8210-f6d41d5a9138 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:19.395706041 +0000 UTC m=+42.762895885 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c9aaedde-8f0d-4cf9-8210-f6d41d5a9138-metrics-tls") pod "dns-default-4dq86" (UID: "c9aaedde-8f0d-4cf9-8210-f6d41d5a9138") : secret "dns-default-metrics-tls" not found
Jul 27 21:05:15.484594 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:15.484529    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:15.595599 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:15.595571    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:16.707865 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:16.707833    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:16.777918 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:16.777866    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:19.423981 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:19.423945    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c9aaedde-8f0d-4cf9-8210-f6d41d5a9138-metrics-tls\") pod \"dns-default-4dq86\" (UID: \"c9aaedde-8f0d-4cf9-8210-f6d41d5a9138\") " pod="openshift-dns/dns-default-4dq86"
Jul 27 21:05:19.424392 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:19.424088    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 21:05:19.424392 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:19.424154    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9aaedde-8f0d-4cf9-8210-f6d41d5a9138-metrics-tls podName:c9aaedde-8f0d-4cf9-8210-f6d41d5a9138 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:27.424137262 +0000 UTC m=+50.791327117 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c9aaedde-8f0d-4cf9-8210-f6d41d5a9138-metrics-tls") pod "dns-default-4dq86" (UID: "c9aaedde-8f0d-4cf9-8210-f6d41d5a9138") : secret "dns-default-metrics-tls" not found
Jul 27 21:05:19.674156 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:19.674052    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:19.750354 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:19.750330    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:20.935257 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:20.935223    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:21.034535 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:21.034489    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:21.386719 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:21.386650    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:21.486164 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:21.486135    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:27.475317 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:27.475278    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c9aaedde-8f0d-4cf9-8210-f6d41d5a9138-metrics-tls\") pod \"dns-default-4dq86\" (UID: \"c9aaedde-8f0d-4cf9-8210-f6d41d5a9138\") " pod="openshift-dns/dns-default-4dq86"
Jul 27 21:05:27.475848 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:27.475396    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 21:05:27.475848 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:27.475450    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9aaedde-8f0d-4cf9-8210-f6d41d5a9138-metrics-tls podName:c9aaedde-8f0d-4cf9-8210-f6d41d5a9138 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:43.475435195 +0000 UTC m=+66.842625041 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c9aaedde-8f0d-4cf9-8210-f6d41d5a9138-metrics-tls") pod "dns-default-4dq86" (UID: "c9aaedde-8f0d-4cf9-8210-f6d41d5a9138") : secret "dns-default-metrics-tls" not found
Jul 27 21:05:32.458053 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:32.458022    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:32.552788 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:32.552740    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:36.300302 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:36.300275    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pftzz"
Jul 27 21:05:36.474634 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:36.474602    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:36.566197 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:36.566125    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:41.870690 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:41.870624    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:41.914273 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:41.914243    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-2cnw9"]
Jul 27 21:05:41.916756 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:41.916737    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-2cnw9"
Jul 27 21:05:41.920083 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:41.920052    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 27 21:05:41.920181 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:41.920153    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 27 21:05:41.920241 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:41.920187    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 27 21:05:41.920401 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:41.920385    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 27 21:05:41.920775 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:41.920751    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-7h2fp\""
Jul 27 21:05:41.939366 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:41.939348    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 27 21:05:41.951664 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:41.951646    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-2cnw9"]
Jul 27 21:05:41.970295 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:41.970269    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59d66d60-9124-4133-b539-5d4649ba0eef-tmp\") pod \"insights-operator-5885fdb759-2cnw9\" (UID: \"59d66d60-9124-4133-b539-5d4649ba0eef\") " pod="openshift-insights/insights-operator-5885fdb759-2cnw9"
Jul 27 21:05:41.970295 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:41.970293    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/59d66d60-9124-4133-b539-5d4649ba0eef-snapshots\") pod \"insights-operator-5885fdb759-2cnw9\" (UID: \"59d66d60-9124-4133-b539-5d4649ba0eef\") " pod="openshift-insights/insights-operator-5885fdb759-2cnw9"
Jul 27 21:05:41.970400 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:41.970314    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59d66d60-9124-4133-b539-5d4649ba0eef-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-2cnw9\" (UID: \"59d66d60-9124-4133-b539-5d4649ba0eef\") " pod="openshift-insights/insights-operator-5885fdb759-2cnw9"
Jul 27 21:05:41.970400 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:41.970330    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfxpp\" (UniqueName: \"kubernetes.io/projected/59d66d60-9124-4133-b539-5d4649ba0eef-kube-api-access-nfxpp\") pod \"insights-operator-5885fdb759-2cnw9\" (UID: \"59d66d60-9124-4133-b539-5d4649ba0eef\") " pod="openshift-insights/insights-operator-5885fdb759-2cnw9"
Jul 27 21:05:41.970400 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:41.970379    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/59d66d60-9124-4133-b539-5d4649ba0eef-serving-cert\") pod \"insights-operator-5885fdb759-2cnw9\" (UID: \"59d66d60-9124-4133-b539-5d4649ba0eef\") " pod="openshift-insights/insights-operator-5885fdb759-2cnw9"
Jul 27 21:05:41.970489 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:41.970416    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59d66d60-9124-4133-b539-5d4649ba0eef-service-ca-bundle\") pod \"insights-operator-5885fdb759-2cnw9\" (UID: \"59d66d60-9124-4133-b539-5d4649ba0eef\") " pod="openshift-insights/insights-operator-5885fdb759-2cnw9"
Jul 27 21:05:42.008586 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:42.008567    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:42.071433 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:42.071410    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59d66d60-9124-4133-b539-5d4649ba0eef-tmp\") pod \"insights-operator-5885fdb759-2cnw9\" (UID: \"59d66d60-9124-4133-b539-5d4649ba0eef\") " pod="openshift-insights/insights-operator-5885fdb759-2cnw9"
Jul 27 21:05:42.071516 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:42.071435    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/59d66d60-9124-4133-b539-5d4649ba0eef-snapshots\") pod \"insights-operator-5885fdb759-2cnw9\" (UID: \"59d66d60-9124-4133-b539-5d4649ba0eef\") " pod="openshift-insights/insights-operator-5885fdb759-2cnw9"
Jul 27 21:05:42.071516 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:42.071457    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59d66d60-9124-4133-b539-5d4649ba0eef-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-2cnw9\" (UID: \"59d66d60-9124-4133-b539-5d4649ba0eef\") " pod="openshift-insights/insights-operator-5885fdb759-2cnw9"
Jul 27 21:05:42.071516 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:42.071473    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfxpp\" (UniqueName: \"kubernetes.io/projected/59d66d60-9124-4133-b539-5d4649ba0eef-kube-api-access-nfxpp\") pod \"insights-operator-5885fdb759-2cnw9\" (UID: \"59d66d60-9124-4133-b539-5d4649ba0eef\") " pod="openshift-insights/insights-operator-5885fdb759-2cnw9"
Jul 27 21:05:42.071516 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:42.071491    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/59d66d60-9124-4133-b539-5d4649ba0eef-serving-cert\") pod \"insights-operator-5885fdb759-2cnw9\" (UID: \"59d66d60-9124-4133-b539-5d4649ba0eef\") " pod="openshift-insights/insights-operator-5885fdb759-2cnw9"
Jul 27 21:05:42.071516 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:42.071515    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59d66d60-9124-4133-b539-5d4649ba0eef-service-ca-bundle\") pod \"insights-operator-5885fdb759-2cnw9\" (UID: \"59d66d60-9124-4133-b539-5d4649ba0eef\") " pod="openshift-insights/insights-operator-5885fdb759-2cnw9"
Jul 27 21:05:42.071802 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:42.071781    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59d66d60-9124-4133-b539-5d4649ba0eef-tmp\") pod \"insights-operator-5885fdb759-2cnw9\" (UID: \"59d66d60-9124-4133-b539-5d4649ba0eef\") " pod="openshift-insights/insights-operator-5885fdb759-2cnw9"
Jul 27 21:05:42.072064 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:42.072044    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59d66d60-9124-4133-b539-5d4649ba0eef-service-ca-bundle\") pod \"insights-operator-5885fdb759-2cnw9\" (UID: \"59d66d60-9124-4133-b539-5d4649ba0eef\") " pod="openshift-insights/insights-operator-5885fdb759-2cnw9"
Jul 27 21:05:42.072127 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:42.072111    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/59d66d60-9124-4133-b539-5d4649ba0eef-snapshots\") pod \"insights-operator-5885fdb759-2cnw9\" (UID: \"59d66d60-9124-4133-b539-5d4649ba0eef\") " pod="openshift-insights/insights-operator-5885fdb759-2cnw9"
Jul 27 21:05:42.072222 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:42.072207    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59d66d60-9124-4133-b539-5d4649ba0eef-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-2cnw9\" (UID: \"59d66d60-9124-4133-b539-5d4649ba0eef\") " pod="openshift-insights/insights-operator-5885fdb759-2cnw9"
Jul 27 21:05:42.075235 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:42.075217    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/59d66d60-9124-4133-b539-5d4649ba0eef-serving-cert\") pod \"insights-operator-5885fdb759-2cnw9\" (UID: \"59d66d60-9124-4133-b539-5d4649ba0eef\") " pod="openshift-insights/insights-operator-5885fdb759-2cnw9"
Jul 27 21:05:42.081256 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:42.081238    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfxpp\" (UniqueName: \"kubernetes.io/projected/59d66d60-9124-4133-b539-5d4649ba0eef-kube-api-access-nfxpp\") pod \"insights-operator-5885fdb759-2cnw9\" (UID: \"59d66d60-9124-4133-b539-5d4649ba0eef\") " pod="openshift-insights/insights-operator-5885fdb759-2cnw9"
Jul 27 21:05:42.116060 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:42.116040    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-79799f5b69-7cqcc"]
Jul 27 21:05:42.118085 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:42.118071    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-79799f5b69-7cqcc"
Jul 27 21:05:42.122231 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:42.122186    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 27 21:05:42.122231 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:42.122216    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 27 21:05:42.122362 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:42.122270    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 27 21:05:42.122483 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:42.122467    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 27 21:05:42.122551 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:42.122539    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 27 21:05:42.122847 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:42.122833    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 27 21:05:42.122950 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:42.122926    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-6ssl5\""
Jul 27 21:05:42.136939 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:42.136919    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-79799f5b69-7cqcc"]
Jul 27 21:05:42.172407 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:42.172387    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/13976f4a-be6e-4e64-bada-c4e12029fcf9-default-certificate\") pod \"router-default-79799f5b69-7cqcc\" (UID: \"13976f4a-be6e-4e64-bada-c4e12029fcf9\") " pod="openshift-ingress/router-default-79799f5b69-7cqcc"
Jul 27 21:05:42.172512 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:42.172416    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13976f4a-be6e-4e64-bada-c4e12029fcf9-metrics-certs\") pod \"router-default-79799f5b69-7cqcc\" (UID: \"13976f4a-be6e-4e64-bada-c4e12029fcf9\") " pod="openshift-ingress/router-default-79799f5b69-7cqcc"
Jul 27 21:05:42.172512 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:42.172436    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gcbr\" (UniqueName: \"kubernetes.io/projected/13976f4a-be6e-4e64-bada-c4e12029fcf9-kube-api-access-6gcbr\") pod \"router-default-79799f5b69-7cqcc\" (UID: \"13976f4a-be6e-4e64-bada-c4e12029fcf9\") " pod="openshift-ingress/router-default-79799f5b69-7cqcc"
Jul 27 21:05:42.172512 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:42.172458    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13976f4a-be6e-4e64-bada-c4e12029fcf9-service-ca-bundle\") pod \"router-default-79799f5b69-7cqcc\" (UID: \"13976f4a-be6e-4e64-bada-c4e12029fcf9\") " pod="openshift-ingress/router-default-79799f5b69-7cqcc"
Jul 27 21:05:42.172606 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:42.172556    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/13976f4a-be6e-4e64-bada-c4e12029fcf9-stats-auth\") pod \"router-default-79799f5b69-7cqcc\" (UID: \"13976f4a-be6e-4e64-bada-c4e12029fcf9\") " pod="openshift-ingress/router-default-79799f5b69-7cqcc"
Jul 27 21:05:42.225075 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:42.225056    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-2cnw9"
Jul 27 21:05:42.273463 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:42.273428    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/13976f4a-be6e-4e64-bada-c4e12029fcf9-default-certificate\") pod \"router-default-79799f5b69-7cqcc\" (UID: \"13976f4a-be6e-4e64-bada-c4e12029fcf9\") " pod="openshift-ingress/router-default-79799f5b69-7cqcc"
Jul 27 21:05:42.273593 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:42.273468    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13976f4a-be6e-4e64-bada-c4e12029fcf9-metrics-certs\") pod \"router-default-79799f5b69-7cqcc\" (UID: \"13976f4a-be6e-4e64-bada-c4e12029fcf9\") " pod="openshift-ingress/router-default-79799f5b69-7cqcc"
Jul 27 21:05:42.273593 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:42.273496    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gcbr\" (UniqueName: \"kubernetes.io/projected/13976f4a-be6e-4e64-bada-c4e12029fcf9-kube-api-access-6gcbr\") pod \"router-default-79799f5b69-7cqcc\" (UID: \"13976f4a-be6e-4e64-bada-c4e12029fcf9\") " pod="openshift-ingress/router-default-79799f5b69-7cqcc"
Jul 27 21:05:42.273593 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:42.273531    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13976f4a-be6e-4e64-bada-c4e12029fcf9-service-ca-bundle\") pod \"router-default-79799f5b69-7cqcc\" (UID: \"13976f4a-be6e-4e64-bada-c4e12029fcf9\") " pod="openshift-ingress/router-default-79799f5b69-7cqcc"
Jul 27 21:05:42.273748 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:42.273623    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 27 21:05:42.273748 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:42.273694    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/13976f4a-be6e-4e64-bada-c4e12029fcf9-stats-auth\") pod \"router-default-79799f5b69-7cqcc\" (UID: \"13976f4a-be6e-4e64-bada-c4e12029fcf9\") " pod="openshift-ingress/router-default-79799f5b69-7cqcc"
Jul 27 21:05:42.273748 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:42.273740    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13976f4a-be6e-4e64-bada-c4e12029fcf9-metrics-certs podName:13976f4a-be6e-4e64-bada-c4e12029fcf9 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:42.773717529 +0000 UTC m=+66.140907384 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13976f4a-be6e-4e64-bada-c4e12029fcf9-metrics-certs") pod "router-default-79799f5b69-7cqcc" (UID: "13976f4a-be6e-4e64-bada-c4e12029fcf9") : secret "router-metrics-certs-default" not found
Jul 27 21:05:42.273919 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:42.273897    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/13976f4a-be6e-4e64-bada-c4e12029fcf9-service-ca-bundle podName:13976f4a-be6e-4e64-bada-c4e12029fcf9 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:42.773875901 +0000 UTC m=+66.141065747 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/13976f4a-be6e-4e64-bada-c4e12029fcf9-service-ca-bundle") pod "router-default-79799f5b69-7cqcc" (UID: "13976f4a-be6e-4e64-bada-c4e12029fcf9") : configmap references non-existent config key: service-ca.crt
Jul 27 21:05:42.276640 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:42.276592    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/13976f4a-be6e-4e64-bada-c4e12029fcf9-stats-auth\") pod \"router-default-79799f5b69-7cqcc\" (UID: \"13976f4a-be6e-4e64-bada-c4e12029fcf9\") " pod="openshift-ingress/router-default-79799f5b69-7cqcc"
Jul 27 21:05:42.276640 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:42.276588    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/13976f4a-be6e-4e64-bada-c4e12029fcf9-default-certificate\") pod \"router-default-79799f5b69-7cqcc\" (UID: \"13976f4a-be6e-4e64-bada-c4e12029fcf9\") " pod="openshift-ingress/router-default-79799f5b69-7cqcc"
Jul 27 21:05:42.296023 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:42.295971    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gcbr\" (UniqueName: \"kubernetes.io/projected/13976f4a-be6e-4e64-bada-c4e12029fcf9-kube-api-access-6gcbr\") pod \"router-default-79799f5b69-7cqcc\" (UID: \"13976f4a-be6e-4e64-bada-c4e12029fcf9\") " pod="openshift-ingress/router-default-79799f5b69-7cqcc"
Jul 27 21:05:42.341161 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:42.341123    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:42.365330 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:42.365306    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-2cnw9"]
Jul 27 21:05:42.368618 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:05:42.368596    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59d66d60_9124_4133_b539_5d4649ba0eef.slice/crio-6e057137e2fd588ded8d58cf178ce10c79ad2afcc10c97aab06e6c3bb6f61f20 WatchSource:0}: Error finding container 6e057137e2fd588ded8d58cf178ce10c79ad2afcc10c97aab06e6c3bb6f61f20: Status 404 returned error can't find the container with id 6e057137e2fd588ded8d58cf178ce10c79ad2afcc10c97aab06e6c3bb6f61f20
Jul 27 21:05:42.380312 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:42.380286    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-2cnw9" event={"ID":"59d66d60-9124-4133-b539-5d4649ba0eef","Type":"ContainerStarted","Data":"6e057137e2fd588ded8d58cf178ce10c79ad2afcc10c97aab06e6c3bb6f61f20"}
Jul 27 21:05:42.442538 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:42.442513    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:42.777733 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:42.777708    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3af7400b-f39f-4f41-a492-586171130c30-metrics-certs\") pod \"network-metrics-daemon-wb7n2\" (UID: \"3af7400b-f39f-4f41-a492-586171130c30\") " pod="openshift-multus/network-metrics-daemon-wb7n2"
Jul 27 21:05:42.777889 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:42.777806    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13976f4a-be6e-4e64-bada-c4e12029fcf9-metrics-certs\") pod \"router-default-79799f5b69-7cqcc\" (UID: \"13976f4a-be6e-4e64-bada-c4e12029fcf9\") " pod="openshift-ingress/router-default-79799f5b69-7cqcc"
Jul 27 21:05:42.777889 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:42.777839    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13976f4a-be6e-4e64-bada-c4e12029fcf9-service-ca-bundle\") pod \"router-default-79799f5b69-7cqcc\" (UID: \"13976f4a-be6e-4e64-bada-c4e12029fcf9\") " pod="openshift-ingress/router-default-79799f5b69-7cqcc"
Jul 27 21:05:42.778007 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:42.777904    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a4af1379-e40a-471d-a011-d50ab76b7f51-original-pull-secret\") pod \"global-pull-secret-syncer-5d7vf\" (UID: \"a4af1379-e40a-471d-a011-d50ab76b7f51\") " pod="kube-system/global-pull-secret-syncer-5d7vf"
Jul 27 21:05:42.778007 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:42.777938    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 27 21:05:42.778007 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:42.778002    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13976f4a-be6e-4e64-bada-c4e12029fcf9-metrics-certs podName:13976f4a-be6e-4e64-bada-c4e12029fcf9 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:43.777986957 +0000 UTC m=+67.145176803 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13976f4a-be6e-4e64-bada-c4e12029fcf9-metrics-certs") pod "router-default-79799f5b69-7cqcc" (UID: "13976f4a-be6e-4e64-bada-c4e12029fcf9") : secret "router-metrics-certs-default" not found
Jul 27 21:05:42.778158 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:42.778015    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/13976f4a-be6e-4e64-bada-c4e12029fcf9-service-ca-bundle podName:13976f4a-be6e-4e64-bada-c4e12029fcf9 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:43.778009611 +0000 UTC m=+67.145199455 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/13976f4a-be6e-4e64-bada-c4e12029fcf9-service-ca-bundle") pod "router-default-79799f5b69-7cqcc" (UID: "13976f4a-be6e-4e64-bada-c4e12029fcf9") : configmap references non-existent config key: service-ca.crt
Jul 27 21:05:42.780502 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:42.780486    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 27 21:05:42.780596 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:42.780503    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 27 21:05:42.788119 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:42.788102    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 27 21:05:42.788172 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:42.788152    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3af7400b-f39f-4f41-a492-586171130c30-metrics-certs podName:3af7400b-f39f-4f41-a492-586171130c30 nodeName:}" failed. No retries permitted until 2026-07-27 21:06:46.788140144 +0000 UTC m=+130.155329987 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3af7400b-f39f-4f41-a492-586171130c30-metrics-certs") pod "network-metrics-daemon-wb7n2" (UID: "3af7400b-f39f-4f41-a492-586171130c30") : secret "metrics-daemon-secret" not found
Jul 27 21:05:42.789989 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:42.789972    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a4af1379-e40a-471d-a011-d50ab76b7f51-original-pull-secret\") pod \"global-pull-secret-syncer-5d7vf\" (UID: \"a4af1379-e40a-471d-a011-d50ab76b7f51\") " pod="kube-system/global-pull-secret-syncer-5d7vf"
Jul 27 21:05:42.878606 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:42.878584    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmvcb\" (UniqueName: \"kubernetes.io/projected/c5915867-c603-4d51-8417-42aada6603ef-kube-api-access-mmvcb\") pod \"network-check-target-nj9np\" (UID: \"c5915867-c603-4d51-8417-42aada6603ef\") " pod="openshift-network-diagnostics/network-check-target-nj9np"
Jul 27 21:05:42.881475 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:42.881459    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 27 21:05:42.891360 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:42.891344    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 27 21:05:42.901434 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:42.901412    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmvcb\" (UniqueName: \"kubernetes.io/projected/c5915867-c603-4d51-8417-42aada6603ef-kube-api-access-mmvcb\") pod \"network-check-target-nj9np\" (UID: \"c5915867-c603-4d51-8417-42aada6603ef\") " pod="openshift-network-diagnostics/network-check-target-nj9np"
Jul 27 21:05:43.040056 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:43.039984    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5d7vf"
Jul 27 21:05:43.153633 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:43.153607    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7jdqc\""
Jul 27 21:05:43.158173 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:43.158150    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nj9np"
Jul 27 21:05:43.196965 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:43.196883    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5d7vf"]
Jul 27 21:05:43.200445 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:05:43.200398    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4af1379_e40a_471d_a011_d50ab76b7f51.slice/crio-83b9b36d28c9692d420bc89fe4e6ede5e6b2c24df044ad7cb4f39361dae6d34e WatchSource:0}: Error finding container 83b9b36d28c9692d420bc89fe4e6ede5e6b2c24df044ad7cb4f39361dae6d34e: Status 404 returned error can't find the container with id 83b9b36d28c9692d420bc89fe4e6ede5e6b2c24df044ad7cb4f39361dae6d34e
Jul 27 21:05:43.301777 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:43.301684    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nj9np"]
Jul 27 21:05:43.304825 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:05:43.304793    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5915867_c603_4d51_8417_42aada6603ef.slice/crio-556246ac0aefcc5e073818f8f4736ca2f948fe011d450e63ae2ed0e759ccbea4 WatchSource:0}: Error finding container 556246ac0aefcc5e073818f8f4736ca2f948fe011d450e63ae2ed0e759ccbea4: Status 404 returned error can't find the container with id 556246ac0aefcc5e073818f8f4736ca2f948fe011d450e63ae2ed0e759ccbea4
Jul 27 21:05:43.384252 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:43.384222    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nj9np" event={"ID":"c5915867-c603-4d51-8417-42aada6603ef","Type":"ContainerStarted","Data":"556246ac0aefcc5e073818f8f4736ca2f948fe011d450e63ae2ed0e759ccbea4"}
Jul 27 21:05:43.385478 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:43.385447    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5d7vf" event={"ID":"a4af1379-e40a-471d-a011-d50ab76b7f51","Type":"ContainerStarted","Data":"83b9b36d28c9692d420bc89fe4e6ede5e6b2c24df044ad7cb4f39361dae6d34e"}
Jul 27 21:05:43.484854 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:43.484810    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c9aaedde-8f0d-4cf9-8210-f6d41d5a9138-metrics-tls\") pod \"dns-default-4dq86\" (UID: \"c9aaedde-8f0d-4cf9-8210-f6d41d5a9138\") " pod="openshift-dns/dns-default-4dq86"
Jul 27 21:05:43.485045 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:43.484953    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 21:05:43.488037 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:43.485355    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9aaedde-8f0d-4cf9-8210-f6d41d5a9138-metrics-tls podName:c9aaedde-8f0d-4cf9-8210-f6d41d5a9138 nodeName:}" failed. No retries permitted until 2026-07-27 21:06:15.48532941 +0000 UTC m=+98.852519269 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c9aaedde-8f0d-4cf9-8210-f6d41d5a9138-metrics-tls") pod "dns-default-4dq86" (UID: "c9aaedde-8f0d-4cf9-8210-f6d41d5a9138") : secret "dns-default-metrics-tls" not found
Jul 27 21:05:43.786946 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:43.786907    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13976f4a-be6e-4e64-bada-c4e12029fcf9-metrics-certs\") pod \"router-default-79799f5b69-7cqcc\" (UID: \"13976f4a-be6e-4e64-bada-c4e12029fcf9\") " pod="openshift-ingress/router-default-79799f5b69-7cqcc"
Jul 27 21:05:43.787125 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:43.786960    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13976f4a-be6e-4e64-bada-c4e12029fcf9-service-ca-bundle\") pod \"router-default-79799f5b69-7cqcc\" (UID: \"13976f4a-be6e-4e64-bada-c4e12029fcf9\") " pod="openshift-ingress/router-default-79799f5b69-7cqcc"
Jul 27 21:05:43.787125 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:43.787121    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/13976f4a-be6e-4e64-bada-c4e12029fcf9-service-ca-bundle podName:13976f4a-be6e-4e64-bada-c4e12029fcf9 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:45.78710152 +0000 UTC m=+69.154291378 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/13976f4a-be6e-4e64-bada-c4e12029fcf9-service-ca-bundle") pod "router-default-79799f5b69-7cqcc" (UID: "13976f4a-be6e-4e64-bada-c4e12029fcf9") : configmap references non-existent config key: service-ca.crt
Jul 27 21:05:43.787256 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:43.787141    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 27 21:05:43.787303 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:43.787263    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13976f4a-be6e-4e64-bada-c4e12029fcf9-metrics-certs podName:13976f4a-be6e-4e64-bada-c4e12029fcf9 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:45.787239433 +0000 UTC m=+69.154429280 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13976f4a-be6e-4e64-bada-c4e12029fcf9-metrics-certs") pod "router-default-79799f5b69-7cqcc" (UID: "13976f4a-be6e-4e64-bada-c4e12029fcf9") : secret "router-metrics-certs-default" not found
Jul 27 21:05:43.903444 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:43.903393    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:45.394704 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:45.394665    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-2cnw9" event={"ID":"59d66d60-9124-4133-b539-5d4649ba0eef","Type":"ContainerStarted","Data":"4af3ed4bb28e826156c72efb64ee41a28af0cd632dd58442a13fd6e301e0e77f"}
Jul 27 21:05:45.423023 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:45.422494    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-2cnw9" podStartSLOduration=1.8969749710000001 podStartE2EDuration="4.422473981s" totalImagesPullingTime="2.52549901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:05:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:05:45.420700768 +0000 UTC m=+68.787890672" watchObservedRunningTime="2026-07-27 21:05:45.422473981 +0000 UTC m=+68.789663846"
Jul 27 21:05:45.517814 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:45.517729    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:45.575716 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:45.575679    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:45.607171 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:45.607130    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-vlktg"]
Jul 27 21:05:45.640646 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:45.640610    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-vlktg"]
Jul 27 21:05:45.640814 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:45.640719    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-vlktg"
Jul 27 21:05:45.643413 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:45.643391    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-m629t\""
Jul 27 21:05:45.702608 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:45.702577    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssxm9\" (UniqueName: \"kubernetes.io/projected/835846a4-5fb1-4f89-b451-3aeee176a34d-kube-api-access-ssxm9\") pod \"network-check-source-7d88cb55d8-vlktg\" (UID: \"835846a4-5fb1-4f89-b451-3aeee176a34d\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-vlktg"
Jul 27 21:05:45.771457 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:45.771425    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4kjk9"]
Jul 27 21:05:45.780644 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:45.780613    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4kjk9"
Jul 27 21:05:45.783327 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:45.783106    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 27 21:05:45.783327 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:45.783133    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 27 21:05:45.783327 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:45.783170    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tk2sn\""
Jul 27 21:05:45.792602 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:45.792511    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4kjk9"]
Jul 27 21:05:45.795939 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:45.795865    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:45.803286 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:45.803258    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13976f4a-be6e-4e64-bada-c4e12029fcf9-service-ca-bundle\") pod \"router-default-79799f5b69-7cqcc\" (UID: \"13976f4a-be6e-4e64-bada-c4e12029fcf9\") " pod="openshift-ingress/router-default-79799f5b69-7cqcc"
Jul 27 21:05:45.803377 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:45.803296    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssxm9\" (UniqueName: \"kubernetes.io/projected/835846a4-5fb1-4f89-b451-3aeee176a34d-kube-api-access-ssxm9\") pod \"network-check-source-7d88cb55d8-vlktg\" (UID: \"835846a4-5fb1-4f89-b451-3aeee176a34d\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-vlktg"
Jul 27 21:05:45.803377 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:45.803326    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/828f730d-81bd-4759-ab08-0959bdf45455-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4kjk9\" (UID: \"828f730d-81bd-4759-ab08-0959bdf45455\") " pod="openshift-insights/insights-runtime-extractor-4kjk9"
Jul 27 21:05:45.803377 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:45.803357    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/828f730d-81bd-4759-ab08-0959bdf45455-crio-socket\") pod \"insights-runtime-extractor-4kjk9\" (UID: \"828f730d-81bd-4759-ab08-0959bdf45455\") " pod="openshift-insights/insights-runtime-extractor-4kjk9"
Jul 27 21:05:45.803538 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:45.803438    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/828f730d-81bd-4759-ab08-0959bdf45455-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4kjk9\" (UID: \"828f730d-81bd-4759-ab08-0959bdf45455\") " pod="openshift-insights/insights-runtime-extractor-4kjk9"
Jul 27 21:05:45.803538 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:45.803461    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr6kl\" (UniqueName: \"kubernetes.io/projected/828f730d-81bd-4759-ab08-0959bdf45455-kube-api-access-sr6kl\") pod \"insights-runtime-extractor-4kjk9\" (UID: \"828f730d-81bd-4759-ab08-0959bdf45455\") " pod="openshift-insights/insights-runtime-extractor-4kjk9"
Jul 27 21:05:45.803538 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:45.803486    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/828f730d-81bd-4759-ab08-0959bdf45455-data-volume\") pod \"insights-runtime-extractor-4kjk9\" (UID: \"828f730d-81bd-4759-ab08-0959bdf45455\") " pod="openshift-insights/insights-runtime-extractor-4kjk9"
Jul 27 21:05:45.803538 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:45.803520    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13976f4a-be6e-4e64-bada-c4e12029fcf9-metrics-certs\") pod \"router-default-79799f5b69-7cqcc\" (UID: \"13976f4a-be6e-4e64-bada-c4e12029fcf9\") " pod="openshift-ingress/router-default-79799f5b69-7cqcc"
Jul 27 21:05:45.803724 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:45.803620    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 27 21:05:45.803724 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:45.803669    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13976f4a-be6e-4e64-bada-c4e12029fcf9-metrics-certs podName:13976f4a-be6e-4e64-bada-c4e12029fcf9 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:49.803653222 +0000 UTC m=+73.170843067 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13976f4a-be6e-4e64-bada-c4e12029fcf9-metrics-certs") pod "router-default-79799f5b69-7cqcc" (UID: "13976f4a-be6e-4e64-bada-c4e12029fcf9") : secret "router-metrics-certs-default" not found
Jul 27 21:05:45.804069 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:45.804046    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/13976f4a-be6e-4e64-bada-c4e12029fcf9-service-ca-bundle podName:13976f4a-be6e-4e64-bada-c4e12029fcf9 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:49.804030656 +0000 UTC m=+73.171220505 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/13976f4a-be6e-4e64-bada-c4e12029fcf9-service-ca-bundle") pod "router-default-79799f5b69-7cqcc" (UID: "13976f4a-be6e-4e64-bada-c4e12029fcf9") : configmap references non-existent config key: service-ca.crt
Jul 27 21:05:45.815907 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:45.815870    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssxm9\" (UniqueName: \"kubernetes.io/projected/835846a4-5fb1-4f89-b451-3aeee176a34d-kube-api-access-ssxm9\") pod \"network-check-source-7d88cb55d8-vlktg\" (UID: \"835846a4-5fb1-4f89-b451-3aeee176a34d\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-vlktg"
Jul 27 21:05:45.903882 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:45.903803    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/828f730d-81bd-4759-ab08-0959bdf45455-data-volume\") pod \"insights-runtime-extractor-4kjk9\" (UID: \"828f730d-81bd-4759-ab08-0959bdf45455\") " pod="openshift-insights/insights-runtime-extractor-4kjk9"
Jul 27 21:05:45.903882 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:45.903875    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/828f730d-81bd-4759-ab08-0959bdf45455-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4kjk9\" (UID: \"828f730d-81bd-4759-ab08-0959bdf45455\") " pod="openshift-insights/insights-runtime-extractor-4kjk9"
Jul 27 21:05:45.904080 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:45.903908    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/828f730d-81bd-4759-ab08-0959bdf45455-crio-socket\") pod \"insights-runtime-extractor-4kjk9\" (UID: \"828f730d-81bd-4759-ab08-0959bdf45455\") " pod="openshift-insights/insights-runtime-extractor-4kjk9"
Jul 27 21:05:45.904080 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:45.903986    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/828f730d-81bd-4759-ab08-0959bdf45455-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4kjk9\" (UID: \"828f730d-81bd-4759-ab08-0959bdf45455\") " pod="openshift-insights/insights-runtime-extractor-4kjk9"
Jul 27 21:05:45.904080 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:45.904009    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr6kl\" (UniqueName: \"kubernetes.io/projected/828f730d-81bd-4759-ab08-0959bdf45455-kube-api-access-sr6kl\") pod \"insights-runtime-extractor-4kjk9\" (UID: \"828f730d-81bd-4759-ab08-0959bdf45455\") " pod="openshift-insights/insights-runtime-extractor-4kjk9"
Jul 27 21:05:45.904080 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:45.904013    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 21:05:45.904277 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:45.904083    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/828f730d-81bd-4759-ab08-0959bdf45455-insights-runtime-extractor-tls podName:828f730d-81bd-4759-ab08-0959bdf45455 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:46.404060256 +0000 UTC m=+69.771250117 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/828f730d-81bd-4759-ab08-0959bdf45455-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4kjk9" (UID: "828f730d-81bd-4759-ab08-0959bdf45455") : secret "insights-runtime-extractor-tls" not found
Jul 27 21:05:45.904277 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:45.904163    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/828f730d-81bd-4759-ab08-0959bdf45455-data-volume\") pod \"insights-runtime-extractor-4kjk9\" (UID: \"828f730d-81bd-4759-ab08-0959bdf45455\") " pod="openshift-insights/insights-runtime-extractor-4kjk9"
Jul 27 21:05:45.904277 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:45.904227    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/828f730d-81bd-4759-ab08-0959bdf45455-crio-socket\") pod \"insights-runtime-extractor-4kjk9\" (UID: \"828f730d-81bd-4759-ab08-0959bdf45455\") " pod="openshift-insights/insights-runtime-extractor-4kjk9"
Jul 27 21:05:45.904277 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:45.904234    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 21:05:45.904477 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:45.904282    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/828f730d-81bd-4759-ab08-0959bdf45455-kube-rbac-proxy-cm podName:828f730d-81bd-4759-ab08-0959bdf45455 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:46.404265763 +0000 UTC m=+69.771455610 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/828f730d-81bd-4759-ab08-0959bdf45455-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4kjk9" (UID: "828f730d-81bd-4759-ab08-0959bdf45455") : configmap "kube-rbac-proxy" not found
Jul 27 21:05:45.912463 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:45.912435    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr6kl\" (UniqueName: \"kubernetes.io/projected/828f730d-81bd-4759-ab08-0959bdf45455-kube-api-access-sr6kl\") pod \"insights-runtime-extractor-4kjk9\" (UID: \"828f730d-81bd-4759-ab08-0959bdf45455\") " pod="openshift-insights/insights-runtime-extractor-4kjk9"
Jul 27 21:05:45.917078 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:45.917051    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:45.952449 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:45.952418    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-vlktg"
Jul 27 21:05:46.012182 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:46.012154    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:46.090771 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:46.090729    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:46.194548 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:46.194518    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:46.292148 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:46.292114    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:46.408690 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:46.408661    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/828f730d-81bd-4759-ab08-0959bdf45455-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4kjk9\" (UID: \"828f730d-81bd-4759-ab08-0959bdf45455\") " pod="openshift-insights/insights-runtime-extractor-4kjk9"
Jul 27 21:05:46.409110 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:46.408751    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/828f730d-81bd-4759-ab08-0959bdf45455-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4kjk9\" (UID: \"828f730d-81bd-4759-ab08-0959bdf45455\") " pod="openshift-insights/insights-runtime-extractor-4kjk9"
Jul 27 21:05:46.409110 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:46.408832    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 21:05:46.409110 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:46.408852    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 21:05:46.409110 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:46.408910    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/828f730d-81bd-4759-ab08-0959bdf45455-insights-runtime-extractor-tls podName:828f730d-81bd-4759-ab08-0959bdf45455 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:47.408893078 +0000 UTC m=+70.776082931 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/828f730d-81bd-4759-ab08-0959bdf45455-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4kjk9" (UID: "828f730d-81bd-4759-ab08-0959bdf45455") : secret "insights-runtime-extractor-tls" not found
Jul 27 21:05:46.409110 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:46.408933    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/828f730d-81bd-4759-ab08-0959bdf45455-kube-rbac-proxy-cm podName:828f730d-81bd-4759-ab08-0959bdf45455 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:47.408922745 +0000 UTC m=+70.776112593 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/828f730d-81bd-4759-ab08-0959bdf45455-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4kjk9" (UID: "828f730d-81bd-4759-ab08-0959bdf45455") : configmap "kube-rbac-proxy" not found
Jul 27 21:05:47.416029 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:47.415996    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/828f730d-81bd-4759-ab08-0959bdf45455-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4kjk9\" (UID: \"828f730d-81bd-4759-ab08-0959bdf45455\") " pod="openshift-insights/insights-runtime-extractor-4kjk9"
Jul 27 21:05:47.416480 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:47.416057    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/828f730d-81bd-4759-ab08-0959bdf45455-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4kjk9\" (UID: \"828f730d-81bd-4759-ab08-0959bdf45455\") " pod="openshift-insights/insights-runtime-extractor-4kjk9"
Jul 27 21:05:47.416480 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:47.416118    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 21:05:47.416480 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:47.416181    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 21:05:47.416480 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:47.416201    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/828f730d-81bd-4759-ab08-0959bdf45455-kube-rbac-proxy-cm podName:828f730d-81bd-4759-ab08-0959bdf45455 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:49.416179306 +0000 UTC m=+72.783369151 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/828f730d-81bd-4759-ab08-0959bdf45455-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4kjk9" (UID: "828f730d-81bd-4759-ab08-0959bdf45455") : configmap "kube-rbac-proxy" not found
Jul 27 21:05:47.416480 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:47.416227    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/828f730d-81bd-4759-ab08-0959bdf45455-insights-runtime-extractor-tls podName:828f730d-81bd-4759-ab08-0959bdf45455 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:49.416209752 +0000 UTC m=+72.783399610 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/828f730d-81bd-4759-ab08-0959bdf45455-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4kjk9" (UID: "828f730d-81bd-4759-ab08-0959bdf45455") : secret "insights-runtime-extractor-tls" not found
Jul 27 21:05:47.529168 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:47.529142    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-frmv7_e610bffe-2c4d-4577-8a48-4849b0d3266d/dns-node-resolver/0.log"
Jul 27 21:05:47.803443 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:47.803425    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-vlktg"]
Jul 27 21:05:47.806775 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:05:47.806735    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod835846a4_5fb1_4f89_b451_3aeee176a34d.slice/crio-0f6c206fd2749b5c95019aae5c9ab0d9ec6496df7322f58be001fc32a0d146e3 WatchSource:0}: Error finding container 0f6c206fd2749b5c95019aae5c9ab0d9ec6496df7322f58be001fc32a0d146e3: Status 404 returned error can't find the container with id 0f6c206fd2749b5c95019aae5c9ab0d9ec6496df7322f58be001fc32a0d146e3
Jul 27 21:05:47.822312 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:47.822286    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-2rqzz"]
Jul 27 21:05:47.824947 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:47.824870    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-2rqzz"
Jul 27 21:05:47.827447 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:47.827427    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 27 21:05:47.828395 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:47.828369    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 27 21:05:47.828498 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:47.828415    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 27 21:05:47.828498 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:47.828454    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 27 21:05:47.828611 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:47.828370    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-4gxgk\""
Jul 27 21:05:47.835366 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:47.835346    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-2rqzz"]
Jul 27 21:05:47.920677 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:47.920648    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k88s\" (UniqueName: \"kubernetes.io/projected/ed727d68-59a2-4753-9c36-003add83987f-kube-api-access-8k88s\") pod \"service-ca-658d7b7c78-2rqzz\" (UID: \"ed727d68-59a2-4753-9c36-003add83987f\") " pod="openshift-service-ca/service-ca-658d7b7c78-2rqzz"
Jul 27 21:05:47.920816 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:47.920721    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ed727d68-59a2-4753-9c36-003add83987f-signing-key\") pod \"service-ca-658d7b7c78-2rqzz\" (UID: \"ed727d68-59a2-4753-9c36-003add83987f\") " pod="openshift-service-ca/service-ca-658d7b7c78-2rqzz"
Jul 27 21:05:47.920816 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:47.920754    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ed727d68-59a2-4753-9c36-003add83987f-signing-cabundle\") pod \"service-ca-658d7b7c78-2rqzz\" (UID: \"ed727d68-59a2-4753-9c36-003add83987f\") " pod="openshift-service-ca/service-ca-658d7b7c78-2rqzz"
Jul 27 21:05:47.935394 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:47.935320    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:48.021346 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:48.021322    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k88s\" (UniqueName: \"kubernetes.io/projected/ed727d68-59a2-4753-9c36-003add83987f-kube-api-access-8k88s\") pod \"service-ca-658d7b7c78-2rqzz\" (UID: \"ed727d68-59a2-4753-9c36-003add83987f\") " pod="openshift-service-ca/service-ca-658d7b7c78-2rqzz"
Jul 27 21:05:48.021521 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:48.021491    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ed727d68-59a2-4753-9c36-003add83987f-signing-key\") pod \"service-ca-658d7b7c78-2rqzz\" (UID: \"ed727d68-59a2-4753-9c36-003add83987f\") " pod="openshift-service-ca/service-ca-658d7b7c78-2rqzz"
Jul 27 21:05:48.021564 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:48.021556    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ed727d68-59a2-4753-9c36-003add83987f-signing-cabundle\") pod \"service-ca-658d7b7c78-2rqzz\" (UID: \"ed727d68-59a2-4753-9c36-003add83987f\") " pod="openshift-service-ca/service-ca-658d7b7c78-2rqzz"
Jul 27 21:05:48.022160 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:48.022140    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ed727d68-59a2-4753-9c36-003add83987f-signing-cabundle\") pod \"service-ca-658d7b7c78-2rqzz\" (UID: \"ed727d68-59a2-4753-9c36-003add83987f\") " pod="openshift-service-ca/service-ca-658d7b7c78-2rqzz"
Jul 27 21:05:48.023721 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:48.023701    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ed727d68-59a2-4753-9c36-003add83987f-signing-key\") pod \"service-ca-658d7b7c78-2rqzz\" (UID: \"ed727d68-59a2-4753-9c36-003add83987f\") " pod="openshift-service-ca/service-ca-658d7b7c78-2rqzz"
Jul 27 21:05:48.029848 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:48.029832    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k88s\" (UniqueName: \"kubernetes.io/projected/ed727d68-59a2-4753-9c36-003add83987f-kube-api-access-8k88s\") pod \"service-ca-658d7b7c78-2rqzz\" (UID: \"ed727d68-59a2-4753-9c36-003add83987f\") " pod="openshift-service-ca/service-ca-658d7b7c78-2rqzz"
Jul 27 21:05:48.041572 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:48.041535    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:48.115583 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:48.115561    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:48.159924 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:48.159885    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-2rqzz"
Jul 27 21:05:48.274427 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:48.274332    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-2rqzz"]
Jul 27 21:05:48.276832 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:05:48.276802    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded727d68_59a2_4753_9c36_003add83987f.slice/crio-0ea06dd0a5716979fe16bba4705edd852c86f3c12ee8e28992b4a22ff331b418 WatchSource:0}: Error finding container 0ea06dd0a5716979fe16bba4705edd852c86f3c12ee8e28992b4a22ff331b418: Status 404 returned error can't find the container with id 0ea06dd0a5716979fe16bba4705edd852c86f3c12ee8e28992b4a22ff331b418
Jul 27 21:05:48.327902 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:48.327878    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dtklb_0b84c165-5dfa-416c-bec6-28c9d55197ab/node-ca/0.log"
Jul 27 21:05:48.402577 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:48.402552    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nj9np" event={"ID":"c5915867-c603-4d51-8417-42aada6603ef","Type":"ContainerStarted","Data":"e93c259e5b1ba9b91c7308730a43c66997dc59399b974f2ac63b37be60ed3ce8"}
Jul 27 21:05:48.402685 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:48.402655    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-nj9np"
Jul 27 21:05:48.403668 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:48.403651    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-2rqzz" event={"ID":"ed727d68-59a2-4753-9c36-003add83987f","Type":"ContainerStarted","Data":"0ea06dd0a5716979fe16bba4705edd852c86f3c12ee8e28992b4a22ff331b418"}
Jul 27 21:05:48.405062 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:48.405043    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5d7vf" event={"ID":"a4af1379-e40a-471d-a011-d50ab76b7f51","Type":"ContainerStarted","Data":"0252e010ea9b74f3fc9c474cdcf627265923471b94fb45c5e6fe7d0ffbd6c2bb"}
Jul 27 21:05:48.406443 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:48.406418    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-vlktg" event={"ID":"835846a4-5fb1-4f89-b451-3aeee176a34d","Type":"ContainerStarted","Data":"6c65283bbbe92ee244593afba0e7922e218d61d7b8a828433fe2251691356c53"}
Jul 27 21:05:48.406530 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:48.406456    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-vlktg" event={"ID":"835846a4-5fb1-4f89-b451-3aeee176a34d","Type":"ContainerStarted","Data":"0f6c206fd2749b5c95019aae5c9ab0d9ec6496df7322f58be001fc32a0d146e3"}
Jul 27 21:05:48.425996 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:48.425956    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-nj9np" podStartSLOduration=67.045412964 podStartE2EDuration="1m11.425945949s" totalImagesPullingTime="4.380532985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:04:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:05:48.425505847 +0000 UTC m=+71.792695713" watchObservedRunningTime="2026-07-27 21:05:48.425945949 +0000 UTC m=+71.793135813"
Jul 27 21:05:48.464049 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:48.463981    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-5d7vf" podStartSLOduration=66.973251781 podStartE2EDuration="1m11.463968482s" totalImagesPullingTime="4.490716701s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:04:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:05:48.462684024 +0000 UTC m=+71.829873888" watchObservedRunningTime="2026-07-27 21:05:48.463968482 +0000 UTC m=+71.831158346"
Jul 27 21:05:48.480969 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:48.480933    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-vlktg" podStartSLOduration=3.480922708 podStartE2EDuration="3.480922708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:05:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:05:48.479576028 +0000 UTC m=+71.846765892" watchObservedRunningTime="2026-07-27 21:05:48.480922708 +0000 UTC m=+71.848112573"
Jul 27 21:05:49.099279 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:49.099238    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:49.433195 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:49.433168    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/828f730d-81bd-4759-ab08-0959bdf45455-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4kjk9\" (UID: \"828f730d-81bd-4759-ab08-0959bdf45455\") " pod="openshift-insights/insights-runtime-extractor-4kjk9"
Jul 27 21:05:49.433617 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:49.433236    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/828f730d-81bd-4759-ab08-0959bdf45455-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4kjk9\" (UID: \"828f730d-81bd-4759-ab08-0959bdf45455\") " pod="openshift-insights/insights-runtime-extractor-4kjk9"
Jul 27 21:05:49.433617 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:49.433286    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 21:05:49.433617 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:49.433353    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/828f730d-81bd-4759-ab08-0959bdf45455-kube-rbac-proxy-cm podName:828f730d-81bd-4759-ab08-0959bdf45455 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:53.433335047 +0000 UTC m=+76.800524899 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/828f730d-81bd-4759-ab08-0959bdf45455-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4kjk9" (UID: "828f730d-81bd-4759-ab08-0959bdf45455") : configmap "kube-rbac-proxy" not found
Jul 27 21:05:49.433617 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:49.433367    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 21:05:49.433617 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:49.433441    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/828f730d-81bd-4759-ab08-0959bdf45455-insights-runtime-extractor-tls podName:828f730d-81bd-4759-ab08-0959bdf45455 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:53.433422749 +0000 UTC m=+76.800612607 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/828f730d-81bd-4759-ab08-0959bdf45455-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4kjk9" (UID: "828f730d-81bd-4759-ab08-0959bdf45455") : secret "insights-runtime-extractor-tls" not found
Jul 27 21:05:49.530262 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:49.530233    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jk8jc_8931191d-6ec9-492d-a5e1-3293972c77bf/serve-healthcheck-canary/0.log"
Jul 27 21:05:49.837122 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:49.837029    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13976f4a-be6e-4e64-bada-c4e12029fcf9-metrics-certs\") pod \"router-default-79799f5b69-7cqcc\" (UID: \"13976f4a-be6e-4e64-bada-c4e12029fcf9\") " pod="openshift-ingress/router-default-79799f5b69-7cqcc"
Jul 27 21:05:49.837290 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:49.837127    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13976f4a-be6e-4e64-bada-c4e12029fcf9-service-ca-bundle\") pod \"router-default-79799f5b69-7cqcc\" (UID: \"13976f4a-be6e-4e64-bada-c4e12029fcf9\") " pod="openshift-ingress/router-default-79799f5b69-7cqcc"
Jul 27 21:05:49.837290 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:49.837182    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 27 21:05:49.837290 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:49.837200    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/13976f4a-be6e-4e64-bada-c4e12029fcf9-service-ca-bundle podName:13976f4a-be6e-4e64-bada-c4e12029fcf9 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:57.837180731 +0000 UTC m=+81.204370584 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/13976f4a-be6e-4e64-bada-c4e12029fcf9-service-ca-bundle") pod "router-default-79799f5b69-7cqcc" (UID: "13976f4a-be6e-4e64-bada-c4e12029fcf9") : configmap references non-existent config key: service-ca.crt
Jul 27 21:05:49.837290 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:49.837224    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13976f4a-be6e-4e64-bada-c4e12029fcf9-metrics-certs podName:13976f4a-be6e-4e64-bada-c4e12029fcf9 nodeName:}" failed. No retries permitted until 2026-07-27 21:05:57.83721457 +0000 UTC m=+81.204404426 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13976f4a-be6e-4e64-bada-c4e12029fcf9-metrics-certs") pod "router-default-79799f5b69-7cqcc" (UID: "13976f4a-be6e-4e64-bada-c4e12029fcf9") : secret "router-metrics-certs-default" not found
Jul 27 21:05:50.414628 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:50.414550    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-2rqzz" event={"ID":"ed727d68-59a2-4753-9c36-003add83987f","Type":"ContainerStarted","Data":"07e9ea5fde8ae07fa42c4b0610aa0aee738af4974125bef57650bde7bcf73b29"}
Jul 27 21:05:50.503563 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:50.503512    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-2rqzz" podStartSLOduration=1.70663514 podStartE2EDuration="3.503497149s" totalImagesPullingTime="1.796862009s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:05:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:05:50.500902924 +0000 UTC m=+73.868092790" watchObservedRunningTime="2026-07-27 21:05:50.503497149 +0000 UTC m=+73.870687013"
Jul 27 21:05:51.223316 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:51.223280    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:51.363587 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:51.363551    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:51.464738 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:51.464704    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:51.563897 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:51.563810    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:51.680605 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:51.680558    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:52.133325 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:52.133271    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:52.960947 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:52.960902    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:53.164933 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:53.164901    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:53.397719 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:53.397643    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:53.467327 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:53.467299    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/828f730d-81bd-4759-ab08-0959bdf45455-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4kjk9\" (UID: \"828f730d-81bd-4759-ab08-0959bdf45455\") " pod="openshift-insights/insights-runtime-extractor-4kjk9"
Jul 27 21:05:53.467467 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:53.467348    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/828f730d-81bd-4759-ab08-0959bdf45455-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4kjk9\" (UID: \"828f730d-81bd-4759-ab08-0959bdf45455\") " pod="openshift-insights/insights-runtime-extractor-4kjk9"
Jul 27 21:05:53.467467 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:53.467426    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 21:05:53.467467 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:53.467452    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 21:05:53.467623 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:53.467493    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/828f730d-81bd-4759-ab08-0959bdf45455-kube-rbac-proxy-cm podName:828f730d-81bd-4759-ab08-0959bdf45455 nodeName:}" failed. No retries permitted until 2026-07-27 21:06:01.467474291 +0000 UTC m=+84.834664136 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/828f730d-81bd-4759-ab08-0959bdf45455-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4kjk9" (UID: "828f730d-81bd-4759-ab08-0959bdf45455") : configmap "kube-rbac-proxy" not found
Jul 27 21:05:53.467623 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:05:53.467514    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/828f730d-81bd-4759-ab08-0959bdf45455-insights-runtime-extractor-tls podName:828f730d-81bd-4759-ab08-0959bdf45455 nodeName:}" failed. No retries permitted until 2026-07-27 21:06:01.46750388 +0000 UTC m=+84.834693724 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/828f730d-81bd-4759-ab08-0959bdf45455-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4kjk9" (UID: "828f730d-81bd-4759-ab08-0959bdf45455") : secret "insights-runtime-extractor-tls" not found
Jul 27 21:05:53.525493 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:53.525469    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:53.727553 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:53.727530    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:53.962557 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:53.962532    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:54.330120 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:54.330090    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:54.759245 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:54.759215    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:54.910846 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:54.910794    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:55.032298 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:55.032173    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:55.145019 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:55.144982    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:55.377236 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:55.377139    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:55.473381 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:55.473305    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:55.739720 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:55.739691    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:56.324553 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:56.324523    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:56.923894 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:56.923858    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:57.121604 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:57.121571    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:57.318880 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:57.318804    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:57.457089 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:57.457060    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:57.732264 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:57.732233    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:57.900289 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:57.900261    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13976f4a-be6e-4e64-bada-c4e12029fcf9-metrics-certs\") pod \"router-default-79799f5b69-7cqcc\" (UID: \"13976f4a-be6e-4e64-bada-c4e12029fcf9\") " pod="openshift-ingress/router-default-79799f5b69-7cqcc"
Jul 27 21:05:57.900438 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:57.900294    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13976f4a-be6e-4e64-bada-c4e12029fcf9-service-ca-bundle\") pod \"router-default-79799f5b69-7cqcc\" (UID: \"13976f4a-be6e-4e64-bada-c4e12029fcf9\") " pod="openshift-ingress/router-default-79799f5b69-7cqcc"
Jul 27 21:05:57.900896 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:57.900870    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13976f4a-be6e-4e64-bada-c4e12029fcf9-service-ca-bundle\") pod \"router-default-79799f5b69-7cqcc\" (UID: \"13976f4a-be6e-4e64-bada-c4e12029fcf9\") " pod="openshift-ingress/router-default-79799f5b69-7cqcc"
Jul 27 21:05:57.902624 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:57.902596    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13976f4a-be6e-4e64-bada-c4e12029fcf9-metrics-certs\") pod \"router-default-79799f5b69-7cqcc\" (UID: \"13976f4a-be6e-4e64-bada-c4e12029fcf9\") " pod="openshift-ingress/router-default-79799f5b69-7cqcc"
Jul 27 21:05:58.027061 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:58.026988    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-79799f5b69-7cqcc"
Jul 27 21:05:58.152952 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:58.152922    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-79799f5b69-7cqcc"]
Jul 27 21:05:58.156395 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:05:58.156370    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13976f4a_be6e_4e64_bada_c4e12029fcf9.slice/crio-eacd1ac984c07e889cfbda7b20a9fb664a42e5985b106f2efc59a6b8fe9658d2 WatchSource:0}: Error finding container eacd1ac984c07e889cfbda7b20a9fb664a42e5985b106f2efc59a6b8fe9658d2: Status 404 returned error can't find the container with id eacd1ac984c07e889cfbda7b20a9fb664a42e5985b106f2efc59a6b8fe9658d2
Jul 27 21:05:58.443863 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:58.443830    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-79799f5b69-7cqcc" event={"ID":"13976f4a-be6e-4e64-bada-c4e12029fcf9","Type":"ContainerStarted","Data":"830d8ec8e3919ab264ae60fdf312946c8fd51138408527bff861d415c9ae0bb7"}
Jul 27 21:05:58.444037 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:58.443874    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-79799f5b69-7cqcc" event={"ID":"13976f4a-be6e-4e64-bada-c4e12029fcf9","Type":"ContainerStarted","Data":"eacd1ac984c07e889cfbda7b20a9fb664a42e5985b106f2efc59a6b8fe9658d2"}
Jul 27 21:05:58.463226 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:58.463185    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-79799f5b69-7cqcc" podStartSLOduration=16.463172657 podStartE2EDuration="16.463172657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:05:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:05:58.462723625 +0000 UTC m=+81.829913489" watchObservedRunningTime="2026-07-27 21:05:58.463172657 +0000 UTC m=+81.830362522"
Jul 27 21:05:58.565360 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:58.565269    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:58.730352 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:58.730280    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:58.915632 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:58.915604    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:59.027501 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:59.027442    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-79799f5b69-7cqcc"
Jul 27 21:05:59.030754 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:59.030733    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-79799f5b69-7cqcc"
Jul 27 21:05:59.123581 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:59.123557    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:59.325967 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:59.325898    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:59.446497 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:59.446465    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-79799f5b69-7cqcc"
Jul 27 21:05:59.447668 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:59.447648    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-79799f5b69-7cqcc"
Jul 27 21:05:59.926426 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:05:59.926393    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:00.310800 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:00.310710    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:00.368607 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:00.368576    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:00.567183 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:00.567112    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:00.926112 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:00.926084    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:01.530414 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:01.530382    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/828f730d-81bd-4759-ab08-0959bdf45455-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4kjk9\" (UID: \"828f730d-81bd-4759-ab08-0959bdf45455\") " pod="openshift-insights/insights-runtime-extractor-4kjk9"
Jul 27 21:06:01.530853 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:01.530628    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/828f730d-81bd-4759-ab08-0959bdf45455-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4kjk9\" (UID: \"828f730d-81bd-4759-ab08-0959bdf45455\") " pod="openshift-insights/insights-runtime-extractor-4kjk9"
Jul 27 21:06:01.531097 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:01.531079    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/828f730d-81bd-4759-ab08-0959bdf45455-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4kjk9\" (UID: \"828f730d-81bd-4759-ab08-0959bdf45455\") " pod="openshift-insights/insights-runtime-extractor-4kjk9"
Jul 27 21:06:01.532583 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:01.532566    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/828f730d-81bd-4759-ab08-0959bdf45455-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4kjk9\" (UID: \"828f730d-81bd-4759-ab08-0959bdf45455\") " pod="openshift-insights/insights-runtime-extractor-4kjk9"
Jul 27 21:06:01.547992 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:01.547961    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:01.691852 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:01.691825    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4kjk9"
Jul 27 21:06:01.720709 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:01.720671    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:01.808947 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:01.808884    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4kjk9"]
Jul 27 21:06:01.811509 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:06:01.811482    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod828f730d_81bd_4759_ab08_0959bdf45455.slice/crio-4831c4dab4155908b3e45f0cf648fbba49d7e172a07c1c9c30fd20d17ac1f444 WatchSource:0}: Error finding container 4831c4dab4155908b3e45f0cf648fbba49d7e172a07c1c9c30fd20d17ac1f444: Status 404 returned error can't find the container with id 4831c4dab4155908b3e45f0cf648fbba49d7e172a07c1c9c30fd20d17ac1f444
Jul 27 21:06:01.925515 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:01.925488    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:02.456801 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:02.456751    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4kjk9" event={"ID":"828f730d-81bd-4759-ab08-0959bdf45455","Type":"ContainerStarted","Data":"4831c4dab4155908b3e45f0cf648fbba49d7e172a07c1c9c30fd20d17ac1f444"}
Jul 27 21:06:04.703306 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:04.703274    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:04.807277 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:04.807246    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:06.474818 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:06.474787    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4kjk9" event={"ID":"828f730d-81bd-4759-ab08-0959bdf45455","Type":"ContainerStarted","Data":"4b5eaf978b9a87f3e7fd8775952e5391662b59180e6782c6fef45de1395fca48"}
Jul 27 21:06:07.448520 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:07.448458    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:07.548327 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:07.548276    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:07.724777 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:07.724689    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:07.862366 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:07.862328    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:07.970288 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:07.970262    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:08.271141 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:08.271108    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:08.362264 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:08.362230    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:08.497807 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:08.497776    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:08.580940 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:08.580865    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:09.279156 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:09.279094    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:09.471983 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:09.471944    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:09.588551 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:09.588479    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:10.490284 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:10.490244    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4kjk9" event={"ID":"828f730d-81bd-4759-ab08-0959bdf45455","Type":"ContainerStarted","Data":"5ae74207b4dbd8dfdc3855c303c0013862c208c3a6ef2864823d69352b65560d"}
Jul 27 21:06:10.892595 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:10.892506    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:11.028048 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:11.028016    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:13.501978 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:13.501941    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4kjk9" event={"ID":"828f730d-81bd-4759-ab08-0959bdf45455","Type":"ContainerStarted","Data":"a55de2dea906e9032224ee1112cb43fcacd441485c61def6edd9ab2523ae2209"}
Jul 27 21:06:13.539717 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:13.539672    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-4kjk9" podStartSLOduration=17.382669656 podStartE2EDuration="28.539656697s" totalImagesPullingTime="11.156987041s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:05:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:06:13.539075191 +0000 UTC m=+96.906265065" watchObservedRunningTime="2026-07-27 21:06:13.539656697 +0000 UTC m=+96.906846540"
Jul 27 21:06:13.662580 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:13.662553    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-58b6497ff7-2767s"]
Jul 27 21:06:13.665200 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:13.665178    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58b6497ff7-2767s"
Jul 27 21:06:13.668875 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:13.668843    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 27 21:06:13.668875 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:13.668863    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 27 21:06:13.669088 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:13.668925    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kdvbm\""
Jul 27 21:06:13.669457 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:13.669440    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 27 21:06:13.673240 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:13.673221    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 27 21:06:13.693276 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:13.693236    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-58b6497ff7-2767s"]
Jul 27 21:06:13.779510 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:13.779452    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67cc7d7d4b-6xvtj"]
Jul 27 21:06:13.781206 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:13.781186    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67cc7d7d4b-6xvtj"
Jul 27 21:06:13.786896 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:06:13.786860    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"managed-serviceaccount-dockercfg-rvftk\" is forbidden: User \"system:node:ip-10-0-142-219.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-142-219.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-rvftk\"" type="*v1.Secret"
Jul 27 21:06:13.787027 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:06:13.786998    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"managed-serviceaccount-addon-agent-67cc7d7d4b-6xvtj\" is forbidden: User \"system:node:ip-10-0-142-219.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-142-219.ec2.internal' and this object" podUID="8591af08-3d08-4ecf-8565-1bb0679b0c2a" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67cc7d7d4b-6xvtj"
Jul 27 21:06:13.787123 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:06:13.787104    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"open-cluster-management-image-pull-credentials\" is forbidden: User \"system:node:ip-10-0-142-219.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-142-219.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" type="*v1.Secret"
Jul 27 21:06:13.788625 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:06:13.788588    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"managed-serviceaccount-hub-kubeconfig\" is forbidden: User \"system:node:ip-10-0-142-219.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-142-219.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" type="*v1.Secret"
Jul 27 21:06:13.792615 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:06:13.792587    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-142-219.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-142-219.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Jul 27 21:06:13.792729 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:06:13.792708    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-142-219.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-142-219.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Jul 27 21:06:13.805314 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:13.805296    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-58b6497ff7-2767s"]
Jul 27 21:06:13.805521 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:06:13.805483    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[bound-sa-token ca-trust-extracted image-registry-private-configuration installation-pull-secrets kube-api-access-9rvkg registry-certificates registry-tls trusted-ca], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-58b6497ff7-2767s" podUID="b1f951d6-7682-49a7-a688-1961c7dd4997"
Jul 27 21:06:13.806151 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:13.806130    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67cc7d7d4b-6xvtj"]
Jul 27 21:06:13.831555 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:13.831532    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b1f951d6-7682-49a7-a688-1961c7dd4997-registry-tls\") pod \"image-registry-58b6497ff7-2767s\" (UID: \"b1f951d6-7682-49a7-a688-1961c7dd4997\") " pod="openshift-image-registry/image-registry-58b6497ff7-2767s"
Jul 27 21:06:13.831639 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:13.831556    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b1f951d6-7682-49a7-a688-1961c7dd4997-installation-pull-secrets\") pod \"image-registry-58b6497ff7-2767s\" (UID: \"b1f951d6-7682-49a7-a688-1961c7dd4997\") " pod="openshift-image-registry/image-registry-58b6497ff7-2767s"
Jul 27 21:06:13.831639 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:13.831578    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rvkg\" (UniqueName: \"kubernetes.io/projected/b1f951d6-7682-49a7-a688-1961c7dd4997-kube-api-access-9rvkg\") pod \"image-registry-58b6497ff7-2767s\" (UID: \"b1f951d6-7682-49a7-a688-1961c7dd4997\") " pod="openshift-image-registry/image-registry-58b6497ff7-2767s"
Jul 27 21:06:13.831714 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:13.831665    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b1f951d6-7682-49a7-a688-1961c7dd4997-bound-sa-token\") pod \"image-registry-58b6497ff7-2767s\" (UID: \"b1f951d6-7682-49a7-a688-1961c7dd4997\") " pod="openshift-image-registry/image-registry-58b6497ff7-2767s"
Jul 27 21:06:13.831714 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:13.831709    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b1f951d6-7682-49a7-a688-1961c7dd4997-trusted-ca\") pod \"image-registry-58b6497ff7-2767s\" (UID: \"b1f951d6-7682-49a7-a688-1961c7dd4997\") " pod="openshift-image-registry/image-registry-58b6497ff7-2767s"
Jul 27 21:06:13.831804 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:13.831727    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b1f951d6-7682-49a7-a688-1961c7dd4997-image-registry-private-configuration\") pod \"image-registry-58b6497ff7-2767s\" (UID: \"b1f951d6-7682-49a7-a688-1961c7dd4997\") " pod="openshift-image-registry/image-registry-58b6497ff7-2767s"
Jul 27 21:06:13.831804 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:13.831746    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b1f951d6-7682-49a7-a688-1961c7dd4997-ca-trust-extracted\") pod \"image-registry-58b6497ff7-2767s\" (UID: \"b1f951d6-7682-49a7-a688-1961c7dd4997\") " pod="openshift-image-registry/image-registry-58b6497ff7-2767s"
Jul 27 21:06:13.831804 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:13.831785    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b1f951d6-7682-49a7-a688-1961c7dd4997-registry-certificates\") pod \"image-registry-58b6497ff7-2767s\" (UID: \"b1f951d6-7682-49a7-a688-1961c7dd4997\") " pod="openshift-image-registry/image-registry-58b6497ff7-2767s"
Jul 27 21:06:13.866978 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:13.866958    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:13.933002 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:13.932975    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8591af08-3d08-4ecf-8565-1bb0679b0c2a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-67cc7d7d4b-6xvtj\" (UID: \"8591af08-3d08-4ecf-8565-1bb0679b0c2a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67cc7d7d4b-6xvtj"
Jul 27 21:06:13.933082 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:13.933025    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b1f951d6-7682-49a7-a688-1961c7dd4997-bound-sa-token\") pod \"image-registry-58b6497ff7-2767s\" (UID: \"b1f951d6-7682-49a7-a688-1961c7dd4997\") " pod="openshift-image-registry/image-registry-58b6497ff7-2767s"
Jul 27 21:06:13.933082 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:13.933052    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v4f5\" (UniqueName: \"kubernetes.io/projected/8591af08-3d08-4ecf-8565-1bb0679b0c2a-kube-api-access-5v4f5\") pod \"managed-serviceaccount-addon-agent-67cc7d7d4b-6xvtj\" (UID: \"8591af08-3d08-4ecf-8565-1bb0679b0c2a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67cc7d7d4b-6xvtj"
Jul 27 21:06:13.933158 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:13.933088    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b1f951d6-7682-49a7-a688-1961c7dd4997-trusted-ca\") pod \"image-registry-58b6497ff7-2767s\" (UID: \"b1f951d6-7682-49a7-a688-1961c7dd4997\") " pod="openshift-image-registry/image-registry-58b6497ff7-2767s"
Jul 27 21:06:13.933158 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:13.933104    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b1f951d6-7682-49a7-a688-1961c7dd4997-image-registry-private-configuration\") pod \"image-registry-58b6497ff7-2767s\" (UID: \"b1f951d6-7682-49a7-a688-1961c7dd4997\") " pod="openshift-image-registry/image-registry-58b6497ff7-2767s"
Jul 27 21:06:13.933158 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:13.933130    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b1f951d6-7682-49a7-a688-1961c7dd4997-ca-trust-extracted\") pod \"image-registry-58b6497ff7-2767s\" (UID: \"b1f951d6-7682-49a7-a688-1961c7dd4997\") " pod="openshift-image-registry/image-registry-58b6497ff7-2767s"
Jul 27 21:06:13.933275 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:13.933156    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b1f951d6-7682-49a7-a688-1961c7dd4997-registry-certificates\") pod \"image-registry-58b6497ff7-2767s\" (UID: \"b1f951d6-7682-49a7-a688-1961c7dd4997\") " pod="openshift-image-registry/image-registry-58b6497ff7-2767s"
Jul 27 21:06:13.933275 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:13.933187    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b1f951d6-7682-49a7-a688-1961c7dd4997-registry-tls\") pod \"image-registry-58b6497ff7-2767s\" (UID: \"b1f951d6-7682-49a7-a688-1961c7dd4997\") " pod="openshift-image-registry/image-registry-58b6497ff7-2767s"
Jul 27 21:06:13.933275 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:13.933210    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b1f951d6-7682-49a7-a688-1961c7dd4997-installation-pull-secrets\") pod \"image-registry-58b6497ff7-2767s\" (UID: \"b1f951d6-7682-49a7-a688-1961c7dd4997\") " pod="openshift-image-registry/image-registry-58b6497ff7-2767s"
Jul 27 21:06:13.933275 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:13.933243    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rvkg\" (UniqueName: \"kubernetes.io/projected/b1f951d6-7682-49a7-a688-1961c7dd4997-kube-api-access-9rvkg\") pod \"image-registry-58b6497ff7-2767s\" (UID: \"b1f951d6-7682-49a7-a688-1961c7dd4997\") " pod="openshift-image-registry/image-registry-58b6497ff7-2767s"
Jul 27 21:06:13.933643 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:13.933615    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b1f951d6-7682-49a7-a688-1961c7dd4997-ca-trust-extracted\") pod \"image-registry-58b6497ff7-2767s\" (UID: \"b1f951d6-7682-49a7-a688-1961c7dd4997\") " pod="openshift-image-registry/image-registry-58b6497ff7-2767s"
Jul 27 21:06:13.934046 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:13.934027    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b1f951d6-7682-49a7-a688-1961c7dd4997-trusted-ca\") pod \"image-registry-58b6497ff7-2767s\" (UID: \"b1f951d6-7682-49a7-a688-1961c7dd4997\") " pod="openshift-image-registry/image-registry-58b6497ff7-2767s"
Jul 27 21:06:13.934322 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:13.934085    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b1f951d6-7682-49a7-a688-1961c7dd4997-registry-certificates\") pod \"image-registry-58b6497ff7-2767s\" (UID: \"b1f951d6-7682-49a7-a688-1961c7dd4997\") " pod="openshift-image-registry/image-registry-58b6497ff7-2767s"
Jul 27 21:06:13.935527 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:13.935507    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b1f951d6-7682-49a7-a688-1961c7dd4997-image-registry-private-configuration\") pod \"image-registry-58b6497ff7-2767s\" (UID: \"b1f951d6-7682-49a7-a688-1961c7dd4997\") " pod="openshift-image-registry/image-registry-58b6497ff7-2767s"
Jul 27 21:06:13.935647 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:13.935628    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b1f951d6-7682-49a7-a688-1961c7dd4997-installation-pull-secrets\") pod \"image-registry-58b6497ff7-2767s\" (UID: \"b1f951d6-7682-49a7-a688-1961c7dd4997\") " pod="openshift-image-registry/image-registry-58b6497ff7-2767s"
Jul 27 21:06:13.935842 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:13.935827    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b1f951d6-7682-49a7-a688-1961c7dd4997-registry-tls\") pod \"image-registry-58b6497ff7-2767s\" (UID: \"b1f951d6-7682-49a7-a688-1961c7dd4997\") " pod="openshift-image-registry/image-registry-58b6497ff7-2767s"
Jul 27 21:06:13.946974 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:13.946955    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rvkg\" (UniqueName: \"kubernetes.io/projected/b1f951d6-7682-49a7-a688-1961c7dd4997-kube-api-access-9rvkg\") pod \"image-registry-58b6497ff7-2767s\" (UID: \"b1f951d6-7682-49a7-a688-1961c7dd4997\") " pod="openshift-image-registry/image-registry-58b6497ff7-2767s"
Jul 27 21:06:13.949156 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:13.949134    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b1f951d6-7682-49a7-a688-1961c7dd4997-bound-sa-token\") pod \"image-registry-58b6497ff7-2767s\" (UID: \"b1f951d6-7682-49a7-a688-1961c7dd4997\") " pod="openshift-image-registry/image-registry-58b6497ff7-2767s"
Jul 27 21:06:13.981374 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:13.981353    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:14.034234 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:14.034182    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v4f5\" (UniqueName: \"kubernetes.io/projected/8591af08-3d08-4ecf-8565-1bb0679b0c2a-kube-api-access-5v4f5\") pod \"managed-serviceaccount-addon-agent-67cc7d7d4b-6xvtj\" (UID: \"8591af08-3d08-4ecf-8565-1bb0679b0c2a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67cc7d7d4b-6xvtj"
Jul 27 21:06:14.034323 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:14.034240    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8591af08-3d08-4ecf-8565-1bb0679b0c2a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-67cc7d7d4b-6xvtj\" (UID: \"8591af08-3d08-4ecf-8565-1bb0679b0c2a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67cc7d7d4b-6xvtj"
Jul 27 21:06:14.505286 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:14.505263    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58b6497ff7-2767s"
Jul 27 21:06:14.511666 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:14.511645    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58b6497ff7-2767s"
Jul 27 21:06:14.601890 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:14.601865    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 27 21:06:14.607379 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:14.607358    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8591af08-3d08-4ecf-8565-1bb0679b0c2a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-67cc7d7d4b-6xvtj\" (UID: \"8591af08-3d08-4ecf-8565-1bb0679b0c2a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67cc7d7d4b-6xvtj"
Jul 27 21:06:14.638589 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:14.638572    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b1f951d6-7682-49a7-a688-1961c7dd4997-trusted-ca\") pod \"b1f951d6-7682-49a7-a688-1961c7dd4997\" (UID: \"b1f951d6-7682-49a7-a688-1961c7dd4997\") "
Jul 27 21:06:14.638682 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:14.638617    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b1f951d6-7682-49a7-a688-1961c7dd4997-installation-pull-secrets\") pod \"b1f951d6-7682-49a7-a688-1961c7dd4997\" (UID: \"b1f951d6-7682-49a7-a688-1961c7dd4997\") "
Jul 27 21:06:14.638682 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:14.638652    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b1f951d6-7682-49a7-a688-1961c7dd4997-ca-trust-extracted\") pod \"b1f951d6-7682-49a7-a688-1961c7dd4997\" (UID: \"b1f951d6-7682-49a7-a688-1961c7dd4997\") "
Jul 27 21:06:14.638745 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:14.638679    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b1f951d6-7682-49a7-a688-1961c7dd4997-bound-sa-token\") pod \"b1f951d6-7682-49a7-a688-1961c7dd4997\" (UID: \"b1f951d6-7682-49a7-a688-1961c7dd4997\") "
Jul 27 21:06:14.638745 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:14.638706    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rvkg\" (UniqueName: \"kubernetes.io/projected/b1f951d6-7682-49a7-a688-1961c7dd4997-kube-api-access-9rvkg\") pod \"b1f951d6-7682-49a7-a688-1961c7dd4997\" (UID: \"b1f951d6-7682-49a7-a688-1961c7dd4997\") "
Jul 27 21:06:14.638745 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:14.638738    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b1f951d6-7682-49a7-a688-1961c7dd4997-registry-certificates\") pod \"b1f951d6-7682-49a7-a688-1961c7dd4997\" (UID: \"b1f951d6-7682-49a7-a688-1961c7dd4997\") "
Jul 27 21:06:14.638875 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:14.638806    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b1f951d6-7682-49a7-a688-1961c7dd4997-registry-tls\") pod \"b1f951d6-7682-49a7-a688-1961c7dd4997\" (UID: \"b1f951d6-7682-49a7-a688-1961c7dd4997\") "
Jul 27 21:06:14.638875 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:14.638837    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b1f951d6-7682-49a7-a688-1961c7dd4997-image-registry-private-configuration\") pod \"b1f951d6-7682-49a7-a688-1961c7dd4997\" (UID: \"b1f951d6-7682-49a7-a688-1961c7dd4997\") "
Jul 27 21:06:14.639210 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:14.638998    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1f951d6-7682-49a7-a688-1961c7dd4997-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "b1f951d6-7682-49a7-a688-1961c7dd4997" (UID: "b1f951d6-7682-49a7-a688-1961c7dd4997"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:06:14.639210 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:14.639012    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1f951d6-7682-49a7-a688-1961c7dd4997-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "b1f951d6-7682-49a7-a688-1961c7dd4997" (UID: "b1f951d6-7682-49a7-a688-1961c7dd4997"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:06:14.639339 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:14.639205    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1f951d6-7682-49a7-a688-1961c7dd4997-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "b1f951d6-7682-49a7-a688-1961c7dd4997" (UID: "b1f951d6-7682-49a7-a688-1961c7dd4997"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:06:14.639339 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:14.639273    2577 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b1f951d6-7682-49a7-a688-1961c7dd4997-ca-trust-extracted\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:06:14.639339 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:14.639289    2577 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b1f951d6-7682-49a7-a688-1961c7dd4997-registry-certificates\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:06:14.639339 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:14.639317    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b1f951d6-7682-49a7-a688-1961c7dd4997-trusted-ca\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:06:14.640707 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:14.640685    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1f951d6-7682-49a7-a688-1961c7dd4997-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "b1f951d6-7682-49a7-a688-1961c7dd4997" (UID: "b1f951d6-7682-49a7-a688-1961c7dd4997"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:06:14.640995 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:14.640968    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1f951d6-7682-49a7-a688-1961c7dd4997-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "b1f951d6-7682-49a7-a688-1961c7dd4997" (UID: "b1f951d6-7682-49a7-a688-1961c7dd4997"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:06:14.641088 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:14.641011    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1f951d6-7682-49a7-a688-1961c7dd4997-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "b1f951d6-7682-49a7-a688-1961c7dd4997" (UID: "b1f951d6-7682-49a7-a688-1961c7dd4997"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:06:14.641088 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:14.641062    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1f951d6-7682-49a7-a688-1961c7dd4997-kube-api-access-9rvkg" (OuterVolumeSpecName: "kube-api-access-9rvkg") pod "b1f951d6-7682-49a7-a688-1961c7dd4997" (UID: "b1f951d6-7682-49a7-a688-1961c7dd4997"). InnerVolumeSpecName "kube-api-access-9rvkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:06:14.641088 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:14.641072    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1f951d6-7682-49a7-a688-1961c7dd4997-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "b1f951d6-7682-49a7-a688-1961c7dd4997" (UID: "b1f951d6-7682-49a7-a688-1961c7dd4997"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:06:14.740075 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:14.740041    2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b1f951d6-7682-49a7-a688-1961c7dd4997-registry-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:06:14.740075 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:14.740072    2577 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b1f951d6-7682-49a7-a688-1961c7dd4997-image-registry-private-configuration\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:06:14.740075 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:14.740084    2577 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b1f951d6-7682-49a7-a688-1961c7dd4997-installation-pull-secrets\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:06:14.740241 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:14.740093    2577 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b1f951d6-7682-49a7-a688-1961c7dd4997-bound-sa-token\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:06:14.740241 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:14.740101    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rvkg\" (UniqueName: \"kubernetes.io/projected/b1f951d6-7682-49a7-a688-1961c7dd4997-kube-api-access-9rvkg\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:06:14.992607 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:14.992581    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 27 21:06:15.054060 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:15.054030    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-rvftk\""
Jul 27 21:06:15.060888 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:15.060872    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 27 21:06:15.311200 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:15.311146    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 27 21:06:15.315382 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:15.315351    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v4f5\" (UniqueName: \"kubernetes.io/projected/8591af08-3d08-4ecf-8565-1bb0679b0c2a-kube-api-access-5v4f5\") pod \"managed-serviceaccount-addon-agent-67cc7d7d4b-6xvtj\" (UID: \"8591af08-3d08-4ecf-8565-1bb0679b0c2a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67cc7d7d4b-6xvtj"
Jul 27 21:06:15.508124 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:15.508094    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58b6497ff7-2767s"
Jul 27 21:06:15.539009 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:15.538981    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-58b6497ff7-2767s"]
Jul 27 21:06:15.543553 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:15.543532    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-58b6497ff7-2767s"]
Jul 27 21:06:15.544876 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:15.544859    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c9aaedde-8f0d-4cf9-8210-f6d41d5a9138-metrics-tls\") pod \"dns-default-4dq86\" (UID: \"c9aaedde-8f0d-4cf9-8210-f6d41d5a9138\") " pod="openshift-dns/dns-default-4dq86"
Jul 27 21:06:15.546947 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:15.546927    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c9aaedde-8f0d-4cf9-8210-f6d41d5a9138-metrics-tls\") pod \"dns-default-4dq86\" (UID: \"c9aaedde-8f0d-4cf9-8210-f6d41d5a9138\") " pod="openshift-dns/dns-default-4dq86"
Jul 27 21:06:15.606082 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:15.606021    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67cc7d7d4b-6xvtj"
Jul 27 21:06:15.724349 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:15.724196    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67cc7d7d4b-6xvtj"]
Jul 27 21:06:15.726693 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:06:15.726663    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8591af08_3d08_4ecf_8565_1bb0679b0c2a.slice/crio-625c1950ee88c60c00d4a460eac6b22d0805be55c61e4bd29c69838b0667e49e WatchSource:0}: Error finding container 625c1950ee88c60c00d4a460eac6b22d0805be55c61e4bd29c69838b0667e49e: Status 404 returned error can't find the container with id 625c1950ee88c60c00d4a460eac6b22d0805be55c61e4bd29c69838b0667e49e
Jul 27 21:06:15.802962 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:15.802933    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6vzp2\""
Jul 27 21:06:15.811137 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:15.811118    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4dq86"
Jul 27 21:06:15.927996 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:15.927965    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4dq86"]
Jul 27 21:06:15.930403 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:06:15.930382    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9aaedde_8f0d_4cf9_8210_f6d41d5a9138.slice/crio-de4a4950af56ae8d9f7565563ef255a0f41f08d6dbaeb55d4243bb24d6b813a2 WatchSource:0}: Error finding container de4a4950af56ae8d9f7565563ef255a0f41f08d6dbaeb55d4243bb24d6b813a2: Status 404 returned error can't find the container with id de4a4950af56ae8d9f7565563ef255a0f41f08d6dbaeb55d4243bb24d6b813a2
Jul 27 21:06:16.513538 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:16.513461    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67cc7d7d4b-6xvtj" event={"ID":"8591af08-3d08-4ecf-8565-1bb0679b0c2a","Type":"ContainerStarted","Data":"625c1950ee88c60c00d4a460eac6b22d0805be55c61e4bd29c69838b0667e49e"}
Jul 27 21:06:16.515357 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:16.515330    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4dq86" event={"ID":"c9aaedde-8f0d-4cf9-8210-f6d41d5a9138","Type":"ContainerStarted","Data":"de4a4950af56ae8d9f7565563ef255a0f41f08d6dbaeb55d4243bb24d6b813a2"}
Jul 27 21:06:16.693163 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:16.693123    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:16.800483 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:16.800401    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:17.134456 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:17.134373    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1f951d6-7682-49a7-a688-1961c7dd4997" path="/var/lib/kubelet/pods/b1f951d6-7682-49a7-a688-1961c7dd4997/volumes"
Jul 27 21:06:18.325907 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:18.325869    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:18.479552 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:18.479518    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:18.488386 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:18.488361    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2959c"]
Jul 27 21:06:18.509437 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:18.509404    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2959c"]
Jul 27 21:06:18.509646 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:18.509539    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2959c"
Jul 27 21:06:18.512083 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:18.512058    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 27 21:06:18.512179 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:18.512113    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-9dwkf\""
Jul 27 21:06:18.668865 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:18.668791    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f87d5ad2-d8c3-4bd5-8ac4-722d19ce0ef0-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-2959c\" (UID: \"f87d5ad2-d8c3-4bd5-8ac4-722d19ce0ef0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2959c"
Jul 27 21:06:18.737579 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:18.737541    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:18.769627 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:18.769592    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f87d5ad2-d8c3-4bd5-8ac4-722d19ce0ef0-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-2959c\" (UID: \"f87d5ad2-d8c3-4bd5-8ac4-722d19ce0ef0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2959c"
Jul 27 21:06:18.771931 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:18.771903    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f87d5ad2-d8c3-4bd5-8ac4-722d19ce0ef0-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-2959c\" (UID: \"f87d5ad2-d8c3-4bd5-8ac4-722d19ce0ef0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2959c"
Jul 27 21:06:18.815611 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:18.815587    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:18.820605 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:18.820588    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2959c"
Jul 27 21:06:18.994563 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:18.994538    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2959c"]
Jul 27 21:06:18.997943 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:06:18.997920    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf87d5ad2_d8c3_4bd5_8ac4_722d19ce0ef0.slice/crio-03771d9a705bb63a1ff580eedd464bd00ee4dc0c10f2b10dd1c19ad4d3e37cf0 WatchSource:0}: Error finding container 03771d9a705bb63a1ff580eedd464bd00ee4dc0c10f2b10dd1c19ad4d3e37cf0: Status 404 returned error can't find the container with id 03771d9a705bb63a1ff580eedd464bd00ee4dc0c10f2b10dd1c19ad4d3e37cf0
Jul 27 21:06:19.300617 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:19.300550    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:19.402501 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:19.402470    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:19.412195 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:19.412173    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-nj9np"
Jul 27 21:06:19.526626 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:19.526551    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2959c" event={"ID":"f87d5ad2-d8c3-4bd5-8ac4-722d19ce0ef0","Type":"ContainerStarted","Data":"03771d9a705bb63a1ff580eedd464bd00ee4dc0c10f2b10dd1c19ad4d3e37cf0"}
Jul 27 21:06:19.528191 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:19.528166    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4dq86" event={"ID":"c9aaedde-8f0d-4cf9-8210-f6d41d5a9138","Type":"ContainerStarted","Data":"cdd3acf32a24b7c9cfbb7c59336eb2815ee1655a1762b86354f368602e705224"}
Jul 27 21:06:19.528329 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:19.528206    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4dq86" event={"ID":"c9aaedde-8f0d-4cf9-8210-f6d41d5a9138","Type":"ContainerStarted","Data":"674a469974cbef18be8f94ac1679b7f4bc04ea85d1b4fb6009afbfae638bc0b7"}
Jul 27 21:06:19.528329 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:19.528274    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4dq86"
Jul 27 21:06:19.529590 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:19.529569    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67cc7d7d4b-6xvtj" event={"ID":"8591af08-3d08-4ecf-8565-1bb0679b0c2a","Type":"ContainerStarted","Data":"780780b1a629ecf2c5183c9646ef0dfdea994b35b3f49b95dd29d5697e377568"}
Jul 27 21:06:19.547371 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:19.547336    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-4dq86" podStartSLOduration=65.625587895 podStartE2EDuration="1m8.547323871s" totalImagesPullingTime="2.921735976s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:05:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:06:19.54590188 +0000 UTC m=+102.913091747" watchObservedRunningTime="2026-07-27 21:06:19.547323871 +0000 UTC m=+102.914513736"
Jul 27 21:06:19.567310 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:19.567249    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67cc7d7d4b-6xvtj" podStartSLOduration=3.06511306 podStartE2EDuration="6.567231575s" totalImagesPullingTime="3.502118515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:06:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:06:19.566361449 +0000 UTC m=+102.933551316" watchObservedRunningTime="2026-07-27 21:06:19.567231575 +0000 UTC m=+102.934421441"
Jul 27 21:06:20.534456 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:20.534423    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2959c" event={"ID":"f87d5ad2-d8c3-4bd5-8ac4-722d19ce0ef0","Type":"ContainerStarted","Data":"01e40dfa420020f6f918ad3c12ac0aa18bbd3957cec53d2e86e2f747475e8935"}
Jul 27 21:06:20.534863 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:20.534781    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2959c"
Jul 27 21:06:20.539088 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:20.539069    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2959c"
Jul 27 21:06:20.551613 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:20.551575    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2959c" podStartSLOduration=1.143687962 podStartE2EDuration="2.551562054s" totalImagesPullingTime="1.407874092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:06:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:06:20.550585102 +0000 UTC m=+103.917774967" watchObservedRunningTime="2026-07-27 21:06:20.551562054 +0000 UTC m=+103.918751931"
Jul 27 21:06:21.517445 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:21.517414    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:21.560579 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:21.560547    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-qt8p9"]
Jul 27 21:06:21.580576 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:21.580549    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-qt8p9"]
Jul 27 21:06:21.580727 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:21.580678    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-qt8p9"
Jul 27 21:06:21.582999 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:21.582978    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 27 21:06:21.583902 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:21.583882    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 27 21:06:21.583988 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:21.583889    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-frg5s\""
Jul 27 21:06:21.584039 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:21.583988    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 27 21:06:21.584039 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:21.584031    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 27 21:06:21.584128 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:21.583909    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 27 21:06:21.602044 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:21.602021    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:21.691929 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:21.691890    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4a316693-388f-41e7-bc94-2dce39838f9a-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-qt8p9\" (UID: \"4a316693-388f-41e7-bc94-2dce39838f9a\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-qt8p9"
Jul 27 21:06:21.692103 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:21.691983    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a316693-388f-41e7-bc94-2dce39838f9a-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-qt8p9\" (UID: \"4a316693-388f-41e7-bc94-2dce39838f9a\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-qt8p9"
Jul 27 21:06:21.692103 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:21.692022    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a316693-388f-41e7-bc94-2dce39838f9a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-qt8p9\" (UID: \"4a316693-388f-41e7-bc94-2dce39838f9a\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-qt8p9"
Jul 27 21:06:21.692103 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:21.692055    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w6pl\" (UniqueName: \"kubernetes.io/projected/4a316693-388f-41e7-bc94-2dce39838f9a-kube-api-access-7w6pl\") pod \"prometheus-operator-84d69f794d-qt8p9\" (UID: \"4a316693-388f-41e7-bc94-2dce39838f9a\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-qt8p9"
Jul 27 21:06:21.793035 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:21.792955    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a316693-388f-41e7-bc94-2dce39838f9a-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-qt8p9\" (UID: \"4a316693-388f-41e7-bc94-2dce39838f9a\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-qt8p9"
Jul 27 21:06:21.793035 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:21.792996    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a316693-388f-41e7-bc94-2dce39838f9a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-qt8p9\" (UID: \"4a316693-388f-41e7-bc94-2dce39838f9a\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-qt8p9"
Jul 27 21:06:21.793035 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:21.793023    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w6pl\" (UniqueName: \"kubernetes.io/projected/4a316693-388f-41e7-bc94-2dce39838f9a-kube-api-access-7w6pl\") pod \"prometheus-operator-84d69f794d-qt8p9\" (UID: \"4a316693-388f-41e7-bc94-2dce39838f9a\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-qt8p9"
Jul 27 21:06:21.793294 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:21.793057    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4a316693-388f-41e7-bc94-2dce39838f9a-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-qt8p9\" (UID: \"4a316693-388f-41e7-bc94-2dce39838f9a\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-qt8p9"
Jul 27 21:06:21.793294 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:06:21.793193    2577 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 27 21:06:21.793294 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:06:21.793267    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a316693-388f-41e7-bc94-2dce39838f9a-prometheus-operator-tls podName:4a316693-388f-41e7-bc94-2dce39838f9a nodeName:}" failed. No retries permitted until 2026-07-27 21:06:22.293247741 +0000 UTC m=+105.660437592 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/4a316693-388f-41e7-bc94-2dce39838f9a-prometheus-operator-tls") pod "prometheus-operator-84d69f794d-qt8p9" (UID: "4a316693-388f-41e7-bc94-2dce39838f9a") : secret "prometheus-operator-tls" not found
Jul 27 21:06:21.793837 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:21.793815    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a316693-388f-41e7-bc94-2dce39838f9a-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-qt8p9\" (UID: \"4a316693-388f-41e7-bc94-2dce39838f9a\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-qt8p9"
Jul 27 21:06:21.795363 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:21.795341    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a316693-388f-41e7-bc94-2dce39838f9a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-qt8p9\" (UID: \"4a316693-388f-41e7-bc94-2dce39838f9a\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-qt8p9"
Jul 27 21:06:21.804934 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:21.804905    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w6pl\" (UniqueName: \"kubernetes.io/projected/4a316693-388f-41e7-bc94-2dce39838f9a-kube-api-access-7w6pl\") pod \"prometheus-operator-84d69f794d-qt8p9\" (UID: \"4a316693-388f-41e7-bc94-2dce39838f9a\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-qt8p9"
Jul 27 21:06:21.889896 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:21.889862    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:21.990932 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:21.990895    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:22.297051 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:22.297018    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4a316693-388f-41e7-bc94-2dce39838f9a-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-qt8p9\" (UID: \"4a316693-388f-41e7-bc94-2dce39838f9a\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-qt8p9"
Jul 27 21:06:22.299242 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:22.299214    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4a316693-388f-41e7-bc94-2dce39838f9a-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-qt8p9\" (UID: \"4a316693-388f-41e7-bc94-2dce39838f9a\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-qt8p9"
Jul 27 21:06:22.489728 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:22.489695    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-qt8p9"
Jul 27 21:06:22.624289 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:22.624258    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-qt8p9"]
Jul 27 21:06:22.626831 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:06:22.626805    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a316693_388f_41e7_bc94_2dce39838f9a.slice/crio-7a02ca1b839b73abb8a7c64620eb1c6a2611758e2ce894e003b780c017a3df0f WatchSource:0}: Error finding container 7a02ca1b839b73abb8a7c64620eb1c6a2611758e2ce894e003b780c017a3df0f: Status 404 returned error can't find the container with id 7a02ca1b839b73abb8a7c64620eb1c6a2611758e2ce894e003b780c017a3df0f
Jul 27 21:06:23.550467 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:23.550431    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-qt8p9" event={"ID":"4a316693-388f-41e7-bc94-2dce39838f9a","Type":"ContainerStarted","Data":"7a02ca1b839b73abb8a7c64620eb1c6a2611758e2ce894e003b780c017a3df0f"}
Jul 27 21:06:23.937196 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:23.937164    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:24.318802 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:24.318690    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:24.428861 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:24.428826    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:24.556293 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:24.556260    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-qt8p9" event={"ID":"4a316693-388f-41e7-bc94-2dce39838f9a","Type":"ContainerStarted","Data":"f7ef6b047da0e58460bb3f76a176f81c33bdadcee877f9d33c688a8dd4168a43"}
Jul 27 21:06:24.556441 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:24.556311    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-qt8p9" event={"ID":"4a316693-388f-41e7-bc94-2dce39838f9a","Type":"ContainerStarted","Data":"86ea2720e046c061cb7f591ef5e851abb6ba8fb2667959a914f26548743e552c"}
Jul 27 21:06:26.841776 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:26.841740    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:27.046460 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.046423    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:27.079467 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.079412    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-qt8p9" podStartSLOduration=5.046401311 podStartE2EDuration="6.079393466s" totalImagesPullingTime="1.032992155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:06:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:06:24.593088713 +0000 UTC m=+107.960278579" watchObservedRunningTime="2026-07-27 21:06:27.079393466 +0000 UTC m=+110.446583332"
Jul 27 21:06:27.080119 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.080101    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-dmpfr"]
Jul 27 21:06:27.088756 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.088736    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dmpfr"
Jul 27 21:06:27.091941 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.091887    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-swh5h\""
Jul 27 21:06:27.092092 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.091887    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 27 21:06:27.092154 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.092032    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 27 21:06:27.107699 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.107677    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-dmpfr"]
Jul 27 21:06:27.136157 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.136135    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c65aca0-1763-49ac-a497-5a9176651765-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-dmpfr\" (UID: \"8c65aca0-1763-49ac-a497-5a9176651765\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dmpfr"
Jul 27 21:06:27.136323 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.136171    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c65aca0-1763-49ac-a497-5a9176651765-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-dmpfr\" (UID: \"8c65aca0-1763-49ac-a497-5a9176651765\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dmpfr"
Jul 27 21:06:27.136323 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.136206    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8c65aca0-1763-49ac-a497-5a9176651765-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-dmpfr\" (UID: \"8c65aca0-1763-49ac-a497-5a9176651765\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dmpfr"
Jul 27 21:06:27.136428 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.136314    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vxw6\" (UniqueName: \"kubernetes.io/projected/8c65aca0-1763-49ac-a497-5a9176651765-kube-api-access-5vxw6\") pod \"openshift-state-metrics-b8f49f5c6-dmpfr\" (UID: \"8c65aca0-1763-49ac-a497-5a9176651765\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dmpfr"
Jul 27 21:06:27.156720 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.156678    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:27.237371 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.237336    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c65aca0-1763-49ac-a497-5a9176651765-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-dmpfr\" (UID: \"8c65aca0-1763-49ac-a497-5a9176651765\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dmpfr"
Jul 27 21:06:27.237552 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.237388    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c65aca0-1763-49ac-a497-5a9176651765-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-dmpfr\" (UID: \"8c65aca0-1763-49ac-a497-5a9176651765\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dmpfr"
Jul 27 21:06:27.237552 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.237430    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8c65aca0-1763-49ac-a497-5a9176651765-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-dmpfr\" (UID: \"8c65aca0-1763-49ac-a497-5a9176651765\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dmpfr"
Jul 27 21:06:27.237552 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.237463    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vxw6\" (UniqueName: \"kubernetes.io/projected/8c65aca0-1763-49ac-a497-5a9176651765-kube-api-access-5vxw6\") pod \"openshift-state-metrics-b8f49f5c6-dmpfr\" (UID: \"8c65aca0-1763-49ac-a497-5a9176651765\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dmpfr"
Jul 27 21:06:27.238201 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.238139    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c65aca0-1763-49ac-a497-5a9176651765-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-dmpfr\" (UID: \"8c65aca0-1763-49ac-a497-5a9176651765\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dmpfr"
Jul 27 21:06:27.239818 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.239800    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8c65aca0-1763-49ac-a497-5a9176651765-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-dmpfr\" (UID: \"8c65aca0-1763-49ac-a497-5a9176651765\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dmpfr"
Jul 27 21:06:27.239912 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.239894    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c65aca0-1763-49ac-a497-5a9176651765-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-dmpfr\" (UID: \"8c65aca0-1763-49ac-a497-5a9176651765\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dmpfr"
Jul 27 21:06:27.242601 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.242576    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-glj4s"]
Jul 27 21:06:27.245292 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.245266    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-glj4s"
Jul 27 21:06:27.248794 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:06:27.248748    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"node-exporter-dockercfg-6qqmv\" is forbidden: User \"system:node:ip-10-0-142-219.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-142-219.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6qqmv\"" type="*v1.Secret"
Jul 27 21:06:27.248894 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:06:27.248851    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"node-exporter-glj4s\" is forbidden: User \"system:node:ip-10-0-142-219.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-142-219.ec2.internal' and this object" podUID="b5d1ed58-e125-4d48-beba-c93d6cb7d5b7" pod="openshift-monitoring/node-exporter-glj4s"
Jul 27 21:06:27.249508 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.249491    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 27 21:06:27.249836 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.249818    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 27 21:06:27.249956 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.249855    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 27 21:06:27.263512 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.263484    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vxw6\" (UniqueName: \"kubernetes.io/projected/8c65aca0-1763-49ac-a497-5a9176651765-kube-api-access-5vxw6\") pod \"openshift-state-metrics-b8f49f5c6-dmpfr\" (UID: \"8c65aca0-1763-49ac-a497-5a9176651765\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dmpfr"
Jul 27 21:06:27.321582 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.321553    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:27.338277 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.338239    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5d1ed58-e125-4d48-beba-c93d6cb7d5b7-metrics-client-ca\") pod \"node-exporter-glj4s\" (UID: \"b5d1ed58-e125-4d48-beba-c93d6cb7d5b7\") " pod="openshift-monitoring/node-exporter-glj4s"
Jul 27 21:06:27.338483 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.338282    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b5d1ed58-e125-4d48-beba-c93d6cb7d5b7-node-exporter-textfile\") pod \"node-exporter-glj4s\" (UID: \"b5d1ed58-e125-4d48-beba-c93d6cb7d5b7\") " pod="openshift-monitoring/node-exporter-glj4s"
Jul 27 21:06:27.338483 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.338351    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5d1ed58-e125-4d48-beba-c93d6cb7d5b7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-glj4s\" (UID: \"b5d1ed58-e125-4d48-beba-c93d6cb7d5b7\") " pod="openshift-monitoring/node-exporter-glj4s"
Jul 27 21:06:27.338483 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.338375    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9lqw\" (UniqueName: \"kubernetes.io/projected/b5d1ed58-e125-4d48-beba-c93d6cb7d5b7-kube-api-access-f9lqw\") pod \"node-exporter-glj4s\" (UID: \"b5d1ed58-e125-4d48-beba-c93d6cb7d5b7\") " pod="openshift-monitoring/node-exporter-glj4s"
Jul 27 21:06:27.338483 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.338396    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b5d1ed58-e125-4d48-beba-c93d6cb7d5b7-root\") pod \"node-exporter-glj4s\" (UID: \"b5d1ed58-e125-4d48-beba-c93d6cb7d5b7\") " pod="openshift-monitoring/node-exporter-glj4s"
Jul 27 21:06:27.338483 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.338419    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5d1ed58-e125-4d48-beba-c93d6cb7d5b7-sys\") pod \"node-exporter-glj4s\" (UID: \"b5d1ed58-e125-4d48-beba-c93d6cb7d5b7\") " pod="openshift-monitoring/node-exporter-glj4s"
Jul 27 21:06:27.338483 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.338442    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b5d1ed58-e125-4d48-beba-c93d6cb7d5b7-node-exporter-wtmp\") pod \"node-exporter-glj4s\" (UID: \"b5d1ed58-e125-4d48-beba-c93d6cb7d5b7\") " pod="openshift-monitoring/node-exporter-glj4s"
Jul 27 21:06:27.338696 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.338513    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b5d1ed58-e125-4d48-beba-c93d6cb7d5b7-node-exporter-tls\") pod \"node-exporter-glj4s\" (UID: \"b5d1ed58-e125-4d48-beba-c93d6cb7d5b7\") " pod="openshift-monitoring/node-exporter-glj4s"
Jul 27 21:06:27.338696 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.338540    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b5d1ed58-e125-4d48-beba-c93d6cb7d5b7-node-exporter-accelerators-collector-config\") pod \"node-exporter-glj4s\" (UID: \"b5d1ed58-e125-4d48-beba-c93d6cb7d5b7\") " pod="openshift-monitoring/node-exporter-glj4s"
Jul 27 21:06:27.398333 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.398257    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dmpfr"
Jul 27 21:06:27.422391 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.422360    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:27.439727 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.439692    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5d1ed58-e125-4d48-beba-c93d6cb7d5b7-sys\") pod \"node-exporter-glj4s\" (UID: \"b5d1ed58-e125-4d48-beba-c93d6cb7d5b7\") " pod="openshift-monitoring/node-exporter-glj4s"
Jul 27 21:06:27.439877 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.439740    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b5d1ed58-e125-4d48-beba-c93d6cb7d5b7-node-exporter-wtmp\") pod \"node-exporter-glj4s\" (UID: \"b5d1ed58-e125-4d48-beba-c93d6cb7d5b7\") " pod="openshift-monitoring/node-exporter-glj4s"
Jul 27 21:06:27.439877 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.439818    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5d1ed58-e125-4d48-beba-c93d6cb7d5b7-sys\") pod \"node-exporter-glj4s\" (UID: \"b5d1ed58-e125-4d48-beba-c93d6cb7d5b7\") " pod="openshift-monitoring/node-exporter-glj4s"
Jul 27 21:06:27.439877 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.439828    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b5d1ed58-e125-4d48-beba-c93d6cb7d5b7-node-exporter-tls\") pod \"node-exporter-glj4s\" (UID: \"b5d1ed58-e125-4d48-beba-c93d6cb7d5b7\") " pod="openshift-monitoring/node-exporter-glj4s"
Jul 27 21:06:27.439877 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.439853    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b5d1ed58-e125-4d48-beba-c93d6cb7d5b7-node-exporter-accelerators-collector-config\") pod \"node-exporter-glj4s\" (UID: \"b5d1ed58-e125-4d48-beba-c93d6cb7d5b7\") " pod="openshift-monitoring/node-exporter-glj4s"
Jul 27 21:06:27.439877 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.439873    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b5d1ed58-e125-4d48-beba-c93d6cb7d5b7-node-exporter-wtmp\") pod \"node-exporter-glj4s\" (UID: \"b5d1ed58-e125-4d48-beba-c93d6cb7d5b7\") " pod="openshift-monitoring/node-exporter-glj4s"
Jul 27 21:06:27.440130 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.439891    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5d1ed58-e125-4d48-beba-c93d6cb7d5b7-metrics-client-ca\") pod \"node-exporter-glj4s\" (UID: \"b5d1ed58-e125-4d48-beba-c93d6cb7d5b7\") " pod="openshift-monitoring/node-exporter-glj4s"
Jul 27 21:06:27.440130 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.439919    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b5d1ed58-e125-4d48-beba-c93d6cb7d5b7-node-exporter-textfile\") pod \"node-exporter-glj4s\" (UID: \"b5d1ed58-e125-4d48-beba-c93d6cb7d5b7\") " pod="openshift-monitoring/node-exporter-glj4s"
Jul 27 21:06:27.440130 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.439962    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5d1ed58-e125-4d48-beba-c93d6cb7d5b7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-glj4s\" (UID: \"b5d1ed58-e125-4d48-beba-c93d6cb7d5b7\") " pod="openshift-monitoring/node-exporter-glj4s"
Jul 27 21:06:27.440130 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.439988    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9lqw\" (UniqueName: \"kubernetes.io/projected/b5d1ed58-e125-4d48-beba-c93d6cb7d5b7-kube-api-access-f9lqw\") pod \"node-exporter-glj4s\" (UID: \"b5d1ed58-e125-4d48-beba-c93d6cb7d5b7\") " pod="openshift-monitoring/node-exporter-glj4s"
Jul 27 21:06:27.440130 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.440012    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b5d1ed58-e125-4d48-beba-c93d6cb7d5b7-root\") pod \"node-exporter-glj4s\" (UID: \"b5d1ed58-e125-4d48-beba-c93d6cb7d5b7\") " pod="openshift-monitoring/node-exporter-glj4s"
Jul 27 21:06:27.440130 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.440086    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b5d1ed58-e125-4d48-beba-c93d6cb7d5b7-root\") pod \"node-exporter-glj4s\" (UID: \"b5d1ed58-e125-4d48-beba-c93d6cb7d5b7\") " pod="openshift-monitoring/node-exporter-glj4s"
Jul 27 21:06:27.440423 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.440402    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b5d1ed58-e125-4d48-beba-c93d6cb7d5b7-node-exporter-textfile\") pod \"node-exporter-glj4s\" (UID: \"b5d1ed58-e125-4d48-beba-c93d6cb7d5b7\") " pod="openshift-monitoring/node-exporter-glj4s"
Jul 27 21:06:27.440482 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.440451    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b5d1ed58-e125-4d48-beba-c93d6cb7d5b7-node-exporter-accelerators-collector-config\") pod \"node-exporter-glj4s\" (UID: \"b5d1ed58-e125-4d48-beba-c93d6cb7d5b7\") " pod="openshift-monitoring/node-exporter-glj4s"
Jul 27 21:06:27.440482 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.440468    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5d1ed58-e125-4d48-beba-c93d6cb7d5b7-metrics-client-ca\") pod \"node-exporter-glj4s\" (UID: \"b5d1ed58-e125-4d48-beba-c93d6cb7d5b7\") " pod="openshift-monitoring/node-exporter-glj4s"
Jul 27 21:06:27.442476 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.442455    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b5d1ed58-e125-4d48-beba-c93d6cb7d5b7-node-exporter-tls\") pod \"node-exporter-glj4s\" (UID: \"b5d1ed58-e125-4d48-beba-c93d6cb7d5b7\") " pod="openshift-monitoring/node-exporter-glj4s"
Jul 27 21:06:27.442549 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.442494    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5d1ed58-e125-4d48-beba-c93d6cb7d5b7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-glj4s\" (UID: \"b5d1ed58-e125-4d48-beba-c93d6cb7d5b7\") " pod="openshift-monitoring/node-exporter-glj4s"
Jul 27 21:06:27.453968 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.453949    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9lqw\" (UniqueName: \"kubernetes.io/projected/b5d1ed58-e125-4d48-beba-c93d6cb7d5b7-kube-api-access-f9lqw\") pod \"node-exporter-glj4s\" (UID: \"b5d1ed58-e125-4d48-beba-c93d6cb7d5b7\") " pod="openshift-monitoring/node-exporter-glj4s"
Jul 27 21:06:27.526129 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.526100    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:27.532556 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.532534    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-dmpfr"]
Jul 27 21:06:27.535071 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:06:27.535047    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c65aca0_1763_49ac_a497_5a9176651765.slice/crio-c477f4ed687f125a33d7b51c054f641d578429e6cb02a4f7510dfdafdc07b093 WatchSource:0}: Error finding container c477f4ed687f125a33d7b51c054f641d578429e6cb02a4f7510dfdafdc07b093: Status 404 returned error can't find the container with id c477f4ed687f125a33d7b51c054f641d578429e6cb02a4f7510dfdafdc07b093
Jul 27 21:06:27.568021 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.567998    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dmpfr" event={"ID":"8c65aca0-1763-49ac-a497-5a9176651765","Type":"ContainerStarted","Data":"c477f4ed687f125a33d7b51c054f641d578429e6cb02a4f7510dfdafdc07b093"}
Jul 27 21:06:27.649847 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.649782    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:27.833362 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:27.833324    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:28.006298 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.006218    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:28.132825 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.132789    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 21:06:28.135648 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.135623    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:28.140117 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.140093    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 27 21:06:28.142843 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.142466    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 27 21:06:28.142843 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.142482    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 27 21:06:28.143097 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.143075    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 27 21:06:28.143798 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.143538    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 27 21:06:28.143798 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.143746    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 27 21:06:28.144174 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.143964    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 27 21:06:28.144174 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.144142    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 27 21:06:28.144992 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.144977    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 27 21:06:28.146716 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.146693    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7e963cdc-42c5-4717-8d40-a30efe01c299-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7e963cdc-42c5-4717-8d40-a30efe01c299\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:28.146843 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.146804    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e963cdc-42c5-4717-8d40-a30efe01c299-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7e963cdc-42c5-4717-8d40-a30efe01c299\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:28.146843 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.146837    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7e963cdc-42c5-4717-8d40-a30efe01c299-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7e963cdc-42c5-4717-8d40-a30efe01c299\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:28.146965 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.146871    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjcbf\" (UniqueName: \"kubernetes.io/projected/7e963cdc-42c5-4717-8d40-a30efe01c299-kube-api-access-tjcbf\") pod \"alertmanager-main-0\" (UID: \"7e963cdc-42c5-4717-8d40-a30efe01c299\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:28.146965 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.146915    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7e963cdc-42c5-4717-8d40-a30efe01c299-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7e963cdc-42c5-4717-8d40-a30efe01c299\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:28.146965 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.146939    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7e963cdc-42c5-4717-8d40-a30efe01c299-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7e963cdc-42c5-4717-8d40-a30efe01c299\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:28.147066 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.147011    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7e963cdc-42c5-4717-8d40-a30efe01c299-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7e963cdc-42c5-4717-8d40-a30efe01c299\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:28.147066 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.147044    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e963cdc-42c5-4717-8d40-a30efe01c299-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7e963cdc-42c5-4717-8d40-a30efe01c299\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:28.147190 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.147080    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7e963cdc-42c5-4717-8d40-a30efe01c299-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7e963cdc-42c5-4717-8d40-a30efe01c299\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:28.147190 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.147148    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7e963cdc-42c5-4717-8d40-a30efe01c299-config-out\") pod \"alertmanager-main-0\" (UID: \"7e963cdc-42c5-4717-8d40-a30efe01c299\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:28.147190 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.147182    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7e963cdc-42c5-4717-8d40-a30efe01c299-config-volume\") pod \"alertmanager-main-0\" (UID: \"7e963cdc-42c5-4717-8d40-a30efe01c299\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:28.147379 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.147218    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7e963cdc-42c5-4717-8d40-a30efe01c299-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7e963cdc-42c5-4717-8d40-a30efe01c299\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:28.147379 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.147313    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7e963cdc-42c5-4717-8d40-a30efe01c299-web-config\") pod \"alertmanager-main-0\" (UID: \"7e963cdc-42c5-4717-8d40-a30efe01c299\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:28.150129 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.150079    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:28.151741 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.151720    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-qbqch\""
Jul 27 21:06:28.175642 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.175620    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 21:06:28.248495 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.248461    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7e963cdc-42c5-4717-8d40-a30efe01c299-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7e963cdc-42c5-4717-8d40-a30efe01c299\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:28.248653 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.248500    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7e963cdc-42c5-4717-8d40-a30efe01c299-web-config\") pod \"alertmanager-main-0\" (UID: \"7e963cdc-42c5-4717-8d40-a30efe01c299\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:28.248653 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.248541    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7e963cdc-42c5-4717-8d40-a30efe01c299-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7e963cdc-42c5-4717-8d40-a30efe01c299\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:28.248653 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.248591    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e963cdc-42c5-4717-8d40-a30efe01c299-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7e963cdc-42c5-4717-8d40-a30efe01c299\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:28.248653 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.248617    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7e963cdc-42c5-4717-8d40-a30efe01c299-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7e963cdc-42c5-4717-8d40-a30efe01c299\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:28.248653 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.248644    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjcbf\" (UniqueName: \"kubernetes.io/projected/7e963cdc-42c5-4717-8d40-a30efe01c299-kube-api-access-tjcbf\") pod \"alertmanager-main-0\" (UID: \"7e963cdc-42c5-4717-8d40-a30efe01c299\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:28.248935 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.248672    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7e963cdc-42c5-4717-8d40-a30efe01c299-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7e963cdc-42c5-4717-8d40-a30efe01c299\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:28.248935 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.248695    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7e963cdc-42c5-4717-8d40-a30efe01c299-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7e963cdc-42c5-4717-8d40-a30efe01c299\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:28.248935 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.248726    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7e963cdc-42c5-4717-8d40-a30efe01c299-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7e963cdc-42c5-4717-8d40-a30efe01c299\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:28.248935 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.248751    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e963cdc-42c5-4717-8d40-a30efe01c299-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7e963cdc-42c5-4717-8d40-a30efe01c299\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:28.248935 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.248795    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7e963cdc-42c5-4717-8d40-a30efe01c299-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7e963cdc-42c5-4717-8d40-a30efe01c299\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:28.248935 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.248829    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7e963cdc-42c5-4717-8d40-a30efe01c299-config-out\") pod \"alertmanager-main-0\" (UID: \"7e963cdc-42c5-4717-8d40-a30efe01c299\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:28.248935 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.248855    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7e963cdc-42c5-4717-8d40-a30efe01c299-config-volume\") pod \"alertmanager-main-0\" (UID: \"7e963cdc-42c5-4717-8d40-a30efe01c299\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:28.250019 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.249965    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e963cdc-42c5-4717-8d40-a30efe01c299-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7e963cdc-42c5-4717-8d40-a30efe01c299\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:28.250532 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.250505    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e963cdc-42c5-4717-8d40-a30efe01c299-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7e963cdc-42c5-4717-8d40-a30efe01c299\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:28.252110 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.250752    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7e963cdc-42c5-4717-8d40-a30efe01c299-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7e963cdc-42c5-4717-8d40-a30efe01c299\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:28.252110 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:06:28.251836    2577 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Jul 27 21:06:28.252110 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.251959    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7e963cdc-42c5-4717-8d40-a30efe01c299-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7e963cdc-42c5-4717-8d40-a30efe01c299\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:28.252110 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:06:28.251971    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e963cdc-42c5-4717-8d40-a30efe01c299-secret-alertmanager-main-tls podName:7e963cdc-42c5-4717-8d40-a30efe01c299 nodeName:}" failed. No retries permitted until 2026-07-27 21:06:28.751913607 +0000 UTC m=+112.119103465 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/7e963cdc-42c5-4717-8d40-a30efe01c299-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "7e963cdc-42c5-4717-8d40-a30efe01c299") : secret "alertmanager-main-tls" not found
Jul 27 21:06:28.252110 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.252098    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7e963cdc-42c5-4717-8d40-a30efe01c299-config-volume\") pod \"alertmanager-main-0\" (UID: \"7e963cdc-42c5-4717-8d40-a30efe01c299\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:28.253103 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.252726    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7e963cdc-42c5-4717-8d40-a30efe01c299-web-config\") pod \"alertmanager-main-0\" (UID: \"7e963cdc-42c5-4717-8d40-a30efe01c299\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:28.253330 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.253268    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7e963cdc-42c5-4717-8d40-a30efe01c299-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7e963cdc-42c5-4717-8d40-a30efe01c299\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:28.253820 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.253757    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7e963cdc-42c5-4717-8d40-a30efe01c299-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7e963cdc-42c5-4717-8d40-a30efe01c299\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:28.254330 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.254298    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7e963cdc-42c5-4717-8d40-a30efe01c299-config-out\") pod \"alertmanager-main-0\" (UID: \"7e963cdc-42c5-4717-8d40-a30efe01c299\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:28.255468 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.255438    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7e963cdc-42c5-4717-8d40-a30efe01c299-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7e963cdc-42c5-4717-8d40-a30efe01c299\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:28.256172 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.256153    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7e963cdc-42c5-4717-8d40-a30efe01c299-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7e963cdc-42c5-4717-8d40-a30efe01c299\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:28.261267 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.261197    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjcbf\" (UniqueName: \"kubernetes.io/projected/7e963cdc-42c5-4717-8d40-a30efe01c299-kube-api-access-tjcbf\") pod \"alertmanager-main-0\" (UID: \"7e963cdc-42c5-4717-8d40-a30efe01c299\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:28.506995 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.506966    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:28.559093 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.559059    2577 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-monitoring/node-exporter-glj4s" secret="" err="failed to sync secret cache: timed out waiting for the condition"
Jul 27 21:06:28.559211 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.559116    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-glj4s"
Jul 27 21:06:28.567910 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:06:28.567884    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5d1ed58_e125_4d48_beba_c93d6cb7d5b7.slice/crio-0059656a3ec20a21b98efeebd582c2f1f678cc096b6ac1e370f4bb371d23e447 WatchSource:0}: Error finding container 0059656a3ec20a21b98efeebd582c2f1f678cc096b6ac1e370f4bb371d23e447: Status 404 returned error can't find the container with id 0059656a3ec20a21b98efeebd582c2f1f678cc096b6ac1e370f4bb371d23e447
Jul 27 21:06:28.582611 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.582427    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dmpfr" event={"ID":"8c65aca0-1763-49ac-a497-5a9176651765","Type":"ContainerStarted","Data":"3269b4eba50aef905b2048fcd2f7edfc7086d1344925275fbf20b9b8e0208c66"}
Jul 27 21:06:28.582611 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.582525    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dmpfr" event={"ID":"8c65aca0-1763-49ac-a497-5a9176651765","Type":"ContainerStarted","Data":"ca80045f9621ce4afe9285fe72aaf966fc51d672bbe6f9932d6bf47ecb381dad"}
Jul 27 21:06:28.582611 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.582555    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dmpfr" event={"ID":"8c65aca0-1763-49ac-a497-5a9176651765","Type":"ContainerStarted","Data":"66f087f238d70b2ad17b593d683100ab988ad0f40f0cbebfc275465d1292bca4"}
Jul 27 21:06:28.725364 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.725312    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-dmpfr" podStartSLOduration=0.856881508 podStartE2EDuration="1.725295493s" totalImagesPullingTime="868.413985ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:06:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:06:28.622949261 +0000 UTC m=+111.990139125" watchObservedRunningTime="2026-07-27 21:06:28.725295493 +0000 UTC m=+112.092485357"
Jul 27 21:06:28.725651 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.725637    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6dcd58f88c-2wdl2"]
Jul 27 21:06:28.728399 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.728385    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dcd58f88c-2wdl2"
Jul 27 21:06:28.732739 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.732710    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 27 21:06:28.732924 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.732755    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 27 21:06:28.732924 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.732755    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 27 21:06:28.733105 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.732940    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 27 21:06:28.733298 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.733284    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 27 21:06:28.733496 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.733479    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-pzcm6\""
Jul 27 21:06:28.733810 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.733793    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 27 21:06:28.733879 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.733798    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 27 21:06:28.740105 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.740088    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 27 21:06:28.748015 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.747998    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6dcd58f88c-2wdl2"]
Jul 27 21:06:28.748664 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.748645    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6qqmv\""
Jul 27 21:06:28.752088 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.752066    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f2aba92c-19d7-4f7b-8da0-c6d960e57111-oauth-serving-cert\") pod \"console-6dcd58f88c-2wdl2\" (UID: \"f2aba92c-19d7-4f7b-8da0-c6d960e57111\") " pod="openshift-console/console-6dcd58f88c-2wdl2"
Jul 27 21:06:28.752167 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.752100    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7e963cdc-42c5-4717-8d40-a30efe01c299-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7e963cdc-42c5-4717-8d40-a30efe01c299\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:28.752167 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.752129    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f2aba92c-19d7-4f7b-8da0-c6d960e57111-service-ca\") pod \"console-6dcd58f88c-2wdl2\" (UID: \"f2aba92c-19d7-4f7b-8da0-c6d960e57111\") " pod="openshift-console/console-6dcd58f88c-2wdl2"
Jul 27 21:06:28.752279 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.752171    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2aba92c-19d7-4f7b-8da0-c6d960e57111-trusted-ca-bundle\") pod \"console-6dcd58f88c-2wdl2\" (UID: \"f2aba92c-19d7-4f7b-8da0-c6d960e57111\") " pod="openshift-console/console-6dcd58f88c-2wdl2"
Jul 27 21:06:28.752279 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.752200    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f2aba92c-19d7-4f7b-8da0-c6d960e57111-console-oauth-config\") pod \"console-6dcd58f88c-2wdl2\" (UID: \"f2aba92c-19d7-4f7b-8da0-c6d960e57111\") " pod="openshift-console/console-6dcd58f88c-2wdl2"
Jul 27 21:06:28.752279 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.752218    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f2aba92c-19d7-4f7b-8da0-c6d960e57111-console-config\") pod \"console-6dcd58f88c-2wdl2\" (UID: \"f2aba92c-19d7-4f7b-8da0-c6d960e57111\") " pod="openshift-console/console-6dcd58f88c-2wdl2"
Jul 27 21:06:28.752279 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.752255    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6qln\" (UniqueName: \"kubernetes.io/projected/f2aba92c-19d7-4f7b-8da0-c6d960e57111-kube-api-access-h6qln\") pod \"console-6dcd58f88c-2wdl2\" (UID: \"f2aba92c-19d7-4f7b-8da0-c6d960e57111\") " pod="openshift-console/console-6dcd58f88c-2wdl2"
Jul 27 21:06:28.752279 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.752274    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2aba92c-19d7-4f7b-8da0-c6d960e57111-console-serving-cert\") pod \"console-6dcd58f88c-2wdl2\" (UID: \"f2aba92c-19d7-4f7b-8da0-c6d960e57111\") " pod="openshift-console/console-6dcd58f88c-2wdl2"
Jul 27 21:06:28.754345 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.754325    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7e963cdc-42c5-4717-8d40-a30efe01c299-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7e963cdc-42c5-4717-8d40-a30efe01c299\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:28.853270 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.853208    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2aba92c-19d7-4f7b-8da0-c6d960e57111-trusted-ca-bundle\") pod \"console-6dcd58f88c-2wdl2\" (UID: \"f2aba92c-19d7-4f7b-8da0-c6d960e57111\") " pod="openshift-console/console-6dcd58f88c-2wdl2"
Jul 27 21:06:28.853270 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.853247    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f2aba92c-19d7-4f7b-8da0-c6d960e57111-console-oauth-config\") pod \"console-6dcd58f88c-2wdl2\" (UID: \"f2aba92c-19d7-4f7b-8da0-c6d960e57111\") " pod="openshift-console/console-6dcd58f88c-2wdl2"
Jul 27 21:06:28.853270 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.853262    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f2aba92c-19d7-4f7b-8da0-c6d960e57111-console-config\") pod \"console-6dcd58f88c-2wdl2\" (UID: \"f2aba92c-19d7-4f7b-8da0-c6d960e57111\") " pod="openshift-console/console-6dcd58f88c-2wdl2"
Jul 27 21:06:28.853505 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.853283    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6qln\" (UniqueName: \"kubernetes.io/projected/f2aba92c-19d7-4f7b-8da0-c6d960e57111-kube-api-access-h6qln\") pod \"console-6dcd58f88c-2wdl2\" (UID: \"f2aba92c-19d7-4f7b-8da0-c6d960e57111\") " pod="openshift-console/console-6dcd58f88c-2wdl2"
Jul 27 21:06:28.853505 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.853390    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2aba92c-19d7-4f7b-8da0-c6d960e57111-console-serving-cert\") pod \"console-6dcd58f88c-2wdl2\" (UID: \"f2aba92c-19d7-4f7b-8da0-c6d960e57111\") " pod="openshift-console/console-6dcd58f88c-2wdl2"
Jul 27 21:06:28.853505 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.853467    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f2aba92c-19d7-4f7b-8da0-c6d960e57111-oauth-serving-cert\") pod \"console-6dcd58f88c-2wdl2\" (UID: \"f2aba92c-19d7-4f7b-8da0-c6d960e57111\") " pod="openshift-console/console-6dcd58f88c-2wdl2"
Jul 27 21:06:28.853654 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.853542    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f2aba92c-19d7-4f7b-8da0-c6d960e57111-service-ca\") pod \"console-6dcd58f88c-2wdl2\" (UID: \"f2aba92c-19d7-4f7b-8da0-c6d960e57111\") " pod="openshift-console/console-6dcd58f88c-2wdl2"
Jul 27 21:06:28.854015 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.853992    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f2aba92c-19d7-4f7b-8da0-c6d960e57111-console-config\") pod \"console-6dcd58f88c-2wdl2\" (UID: \"f2aba92c-19d7-4f7b-8da0-c6d960e57111\") " pod="openshift-console/console-6dcd58f88c-2wdl2"
Jul 27 21:06:28.854219 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.854198    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f2aba92c-19d7-4f7b-8da0-c6d960e57111-oauth-serving-cert\") pod \"console-6dcd58f88c-2wdl2\" (UID: \"f2aba92c-19d7-4f7b-8da0-c6d960e57111\") " pod="openshift-console/console-6dcd58f88c-2wdl2"
Jul 27 21:06:28.854335 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.854318    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f2aba92c-19d7-4f7b-8da0-c6d960e57111-service-ca\") pod \"console-6dcd58f88c-2wdl2\" (UID: \"f2aba92c-19d7-4f7b-8da0-c6d960e57111\") " pod="openshift-console/console-6dcd58f88c-2wdl2"
Jul 27 21:06:28.854522 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.854506    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2aba92c-19d7-4f7b-8da0-c6d960e57111-trusted-ca-bundle\") pod \"console-6dcd58f88c-2wdl2\" (UID: \"f2aba92c-19d7-4f7b-8da0-c6d960e57111\") " pod="openshift-console/console-6dcd58f88c-2wdl2"
Jul 27 21:06:28.856121 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.856104    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f2aba92c-19d7-4f7b-8da0-c6d960e57111-console-oauth-config\") pod \"console-6dcd58f88c-2wdl2\" (UID: \"f2aba92c-19d7-4f7b-8da0-c6d960e57111\") " pod="openshift-console/console-6dcd58f88c-2wdl2"
Jul 27 21:06:28.856203 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.856188    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2aba92c-19d7-4f7b-8da0-c6d960e57111-console-serving-cert\") pod \"console-6dcd58f88c-2wdl2\" (UID: \"f2aba92c-19d7-4f7b-8da0-c6d960e57111\") " pod="openshift-console/console-6dcd58f88c-2wdl2"
Jul 27 21:06:28.863016 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:28.862994    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6qln\" (UniqueName: \"kubernetes.io/projected/f2aba92c-19d7-4f7b-8da0-c6d960e57111-kube-api-access-h6qln\") pod \"console-6dcd58f88c-2wdl2\" (UID: \"f2aba92c-19d7-4f7b-8da0-c6d960e57111\") " pod="openshift-console/console-6dcd58f88c-2wdl2"
Jul 27 21:06:29.038914 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:29.038882    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dcd58f88c-2wdl2"
Jul 27 21:06:29.047983 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:29.047888    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:06:29.164506 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:29.164459    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:29.188664 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:29.188628    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6dcd58f88c-2wdl2"]
Jul 27 21:06:29.192183 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:06:29.192155    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2aba92c_19d7_4f7b_8da0_c6d960e57111.slice/crio-6f12955299e282a071e7c454ff054bee4650fafde2d87afda437ceb3b2344503 WatchSource:0}: Error finding container 6f12955299e282a071e7c454ff054bee4650fafde2d87afda437ceb3b2344503: Status 404 returned error can't find the container with id 6f12955299e282a071e7c454ff054bee4650fafde2d87afda437ceb3b2344503
Jul 27 21:06:29.217952 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:29.217928    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 21:06:29.221410 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:06:29.221380    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e963cdc_42c5_4717_8d40_a30efe01c299.slice/crio-063d86ee37794b22ccb4f70af9e17c8712909baffef2520f782d57867a1deaa7 WatchSource:0}: Error finding container 063d86ee37794b22ccb4f70af9e17c8712909baffef2520f782d57867a1deaa7: Status 404 returned error can't find the container with id 063d86ee37794b22ccb4f70af9e17c8712909baffef2520f782d57867a1deaa7
Jul 27 21:06:29.291441 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:29.291409    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:29.537295 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:29.537274    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4dq86"
Jul 27 21:06:29.588200 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:29.588134    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dcd58f88c-2wdl2" event={"ID":"f2aba92c-19d7-4f7b-8da0-c6d960e57111","Type":"ContainerStarted","Data":"6f12955299e282a071e7c454ff054bee4650fafde2d87afda437ceb3b2344503"}
Jul 27 21:06:29.590487 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:29.590464    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-glj4s" event={"ID":"b5d1ed58-e125-4d48-beba-c93d6cb7d5b7","Type":"ContainerStarted","Data":"0059656a3ec20a21b98efeebd582c2f1f678cc096b6ac1e370f4bb371d23e447"}
Jul 27 21:06:29.592966 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:29.592938    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7e963cdc-42c5-4717-8d40-a30efe01c299","Type":"ContainerStarted","Data":"063d86ee37794b22ccb4f70af9e17c8712909baffef2520f782d57867a1deaa7"}
Jul 27 21:06:30.098825 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:30.098599    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:30.177048 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:30.177011    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:30.383381 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:30.383345    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:30.534795 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:30.534740    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:30.598360 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:30.598328    2577 generic.go:356] "Generic (PLEG): container finished" podID="b5d1ed58-e125-4d48-beba-c93d6cb7d5b7" containerID="4e1ae9bb3f67cd81178be68ce1edf43ae28ee92b2e0bc00a9c99e754be832240" exitCode=0
Jul 27 21:06:30.598535 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:30.598382    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-glj4s" event={"ID":"b5d1ed58-e125-4d48-beba-c93d6cb7d5b7","Type":"ContainerDied","Data":"4e1ae9bb3f67cd81178be68ce1edf43ae28ee92b2e0bc00a9c99e754be832240"}
Jul 27 21:06:30.600201 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:30.600072    2577 generic.go:356] "Generic (PLEG): container finished" podID="7e963cdc-42c5-4717-8d40-a30efe01c299" containerID="5218f39d38b7ec718af11d34923cec8ec078b5d3e9879eda7fe40a59eedcb6dd" exitCode=0
Jul 27 21:06:30.600201 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:30.600120    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7e963cdc-42c5-4717-8d40-a30efe01c299","Type":"ContainerDied","Data":"5218f39d38b7ec718af11d34923cec8ec078b5d3e9879eda7fe40a59eedcb6dd"}
Jul 27 21:06:31.607644 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:31.607604    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-glj4s" event={"ID":"b5d1ed58-e125-4d48-beba-c93d6cb7d5b7","Type":"ContainerStarted","Data":"f84f48fbd918351801b1e947100429b23bf577fe6470c19d08fbb50ceb7c0525"}
Jul 27 21:06:31.608078 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:31.607668    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-glj4s" event={"ID":"b5d1ed58-e125-4d48-beba-c93d6cb7d5b7","Type":"ContainerStarted","Data":"e04db4522843d00d09f8e05e46642ba75ac2082fdcb31d33a466ef398a1e5fdc"}
Jul 27 21:06:31.632685 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:31.632634    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-glj4s" podStartSLOduration=3.585482684 podStartE2EDuration="4.632617942s" totalImagesPullingTime="959.954835ms" totalInitContainerRuntime="87.180423ms" isStatefulPod=true podCreationTimestamp="2026-07-27 21:06:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:06:31.63083538 +0000 UTC m=+114.998025247" watchObservedRunningTime="2026-07-27 21:06:31.632617942 +0000 UTC m=+114.999807811"
Jul 27 21:06:31.759410 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:31.759376    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:31.868285 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:31.868203    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:32.078914 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:32.078867    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:33.360884 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.360846    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:33.369329 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.368332    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 21:06:33.373081 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.373054    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.375742 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.375716    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 27 21:06:33.379485 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.379461    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 27 21:06:33.379595 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.379564    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 27 21:06:33.379668 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.379617    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-l4lcr\""
Jul 27 21:06:33.379668 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.379631    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 27 21:06:33.379797 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.379679    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 27 21:06:33.379851 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.379824    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 27 21:06:33.379996 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.379977    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 27 21:06:33.380707 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.380176    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 27 21:06:33.380707 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.380211    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 27 21:06:33.380707 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.380221    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 27 21:06:33.380707 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.380269    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fl5nch1nqsboi\""
Jul 27 21:06:33.380707 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.380346    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 27 21:06:33.380707 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.380371    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 27 21:06:33.380707 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.380457    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 27 21:06:33.380707 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.380458    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 27 21:06:33.380707 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.380607    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 27 21:06:33.389554 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.389522    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 21:06:33.395869 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.395846    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f0104c14-0836-4133-a002-da6d3e8201f0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.396003 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.395882    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5lgt\" (UniqueName: \"kubernetes.io/projected/f0104c14-0836-4133-a002-da6d3e8201f0-kube-api-access-k5lgt\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.396003 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.395909    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f0104c14-0836-4133-a002-da6d3e8201f0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.396003 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.395937    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0104c14-0836-4133-a002-da6d3e8201f0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.396003 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.395976    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f0104c14-0836-4133-a002-da6d3e8201f0-config-out\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.396003 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.395990    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0104c14-0836-4133-a002-da6d3e8201f0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.396246 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.396015    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f0104c14-0836-4133-a002-da6d3e8201f0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.396246 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.396038    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f0104c14-0836-4133-a002-da6d3e8201f0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.396246 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.396085    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f0104c14-0836-4133-a002-da6d3e8201f0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.396246 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.396102    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f0104c14-0836-4133-a002-da6d3e8201f0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.396246 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.396145    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f0104c14-0836-4133-a002-da6d3e8201f0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.396246 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.396164    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f0104c14-0836-4133-a002-da6d3e8201f0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.396246 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.396186    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0104c14-0836-4133-a002-da6d3e8201f0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.396246 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.396235    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f0104c14-0836-4133-a002-da6d3e8201f0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.396615 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.396259    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f0104c14-0836-4133-a002-da6d3e8201f0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.396615 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.396275    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f0104c14-0836-4133-a002-da6d3e8201f0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.396615 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.396370    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f0104c14-0836-4133-a002-da6d3e8201f0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.396615 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.396401    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f0104c14-0836-4133-a002-da6d3e8201f0-config\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.396615 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.396435    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f0104c14-0836-4133-a002-da6d3e8201f0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.396615 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.396458    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f0104c14-0836-4133-a002-da6d3e8201f0-web-config\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.497912 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.497882    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f0104c14-0836-4133-a002-da6d3e8201f0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.498092 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.497929    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f0104c14-0836-4133-a002-da6d3e8201f0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.498092 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.497964    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f0104c14-0836-4133-a002-da6d3e8201f0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.498092 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.497991    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f0104c14-0836-4133-a002-da6d3e8201f0-config\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.498092 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.498025    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f0104c14-0836-4133-a002-da6d3e8201f0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.498092 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.498053    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f0104c14-0836-4133-a002-da6d3e8201f0-web-config\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.498092 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.498084    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f0104c14-0836-4133-a002-da6d3e8201f0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.498387 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.498116    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5lgt\" (UniqueName: \"kubernetes.io/projected/f0104c14-0836-4133-a002-da6d3e8201f0-kube-api-access-k5lgt\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.498387 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.498146    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f0104c14-0836-4133-a002-da6d3e8201f0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.498387 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.498176    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0104c14-0836-4133-a002-da6d3e8201f0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.498387 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.498215    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f0104c14-0836-4133-a002-da6d3e8201f0-config-out\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.498387 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.498240    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0104c14-0836-4133-a002-da6d3e8201f0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.498387 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.498274    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f0104c14-0836-4133-a002-da6d3e8201f0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.498387 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.498297    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f0104c14-0836-4133-a002-da6d3e8201f0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.498387 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.498356    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f0104c14-0836-4133-a002-da6d3e8201f0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.498387 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.498383    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f0104c14-0836-4133-a002-da6d3e8201f0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.498833 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.498413    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f0104c14-0836-4133-a002-da6d3e8201f0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.498833 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.498431    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f0104c14-0836-4133-a002-da6d3e8201f0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.498833 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.498446    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0104c14-0836-4133-a002-da6d3e8201f0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.498833 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.498498    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f0104c14-0836-4133-a002-da6d3e8201f0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.498833 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.498568    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f0104c14-0836-4133-a002-da6d3e8201f0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.499448 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.499422    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0104c14-0836-4133-a002-da6d3e8201f0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.502970 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.499877    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f0104c14-0836-4133-a002-da6d3e8201f0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.502970 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.500977    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f0104c14-0836-4133-a002-da6d3e8201f0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.502970 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.501037    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f0104c14-0836-4133-a002-da6d3e8201f0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.502970 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.501473    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f0104c14-0836-4133-a002-da6d3e8201f0-config\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.502970 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.501535    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f0104c14-0836-4133-a002-da6d3e8201f0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.502970 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.502138    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0104c14-0836-4133-a002-da6d3e8201f0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.502970 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.502919    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f0104c14-0836-4133-a002-da6d3e8201f0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.503392 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.503010    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0104c14-0836-4133-a002-da6d3e8201f0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.503636 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.503610    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f0104c14-0836-4133-a002-da6d3e8201f0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.504143 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.504116    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f0104c14-0836-4133-a002-da6d3e8201f0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.504345 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.504320    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f0104c14-0836-4133-a002-da6d3e8201f0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.504544 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.504522    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f0104c14-0836-4133-a002-da6d3e8201f0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.504720 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.504694    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f0104c14-0836-4133-a002-da6d3e8201f0-web-config\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.504815 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.504803    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f0104c14-0836-4133-a002-da6d3e8201f0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.505717 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.505677    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f0104c14-0836-4133-a002-da6d3e8201f0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.505874 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.505850    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f0104c14-0836-4133-a002-da6d3e8201f0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.506693 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.506655    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f0104c14-0836-4133-a002-da6d3e8201f0-config-out\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.507330 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.507312    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5lgt\" (UniqueName: \"kubernetes.io/projected/f0104c14-0836-4133-a002-da6d3e8201f0-kube-api-access-k5lgt\") pod \"prometheus-k8s-0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.622836 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.622801    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7e963cdc-42c5-4717-8d40-a30efe01c299","Type":"ContainerStarted","Data":"959e20d8985878927b64feb5b96562450e4167b5eae5cd6dcefec5cfeecc2c6b"}
Jul 27 21:06:33.622926 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.622888    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7e963cdc-42c5-4717-8d40-a30efe01c299","Type":"ContainerStarted","Data":"8490a420afbde38f1318ed19a4601e3e232354dfcb7279a8d7f4ac58431895fa"}
Jul 27 21:06:33.622978 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.622933    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7e963cdc-42c5-4717-8d40-a30efe01c299","Type":"ContainerStarted","Data":"f481c1e635220c79af7b8b2cd35fddf481a62f7d05b56fbf7892c838ff41eedd"}
Jul 27 21:06:33.622978 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.622971    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7e963cdc-42c5-4717-8d40-a30efe01c299","Type":"ContainerStarted","Data":"29ad99c7a1aea8e8f38b95b56629c2a3e5638dc518f8dc4ef8459d76932b1088"}
Jul 27 21:06:33.623043 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.623008    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7e963cdc-42c5-4717-8d40-a30efe01c299","Type":"ContainerStarted","Data":"a4e08fde302bf8e8073340bcd02b4bce60c7f413e5b7bfdda72f15c2eef83fe4"}
Jul 27 21:06:33.624349 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.624326    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dcd58f88c-2wdl2" event={"ID":"f2aba92c-19d7-4f7b-8da0-c6d960e57111","Type":"ContainerStarted","Data":"24a6b16511da0e23d78ee9eeb4231c93844feadb8e917f01e782b0584b3489c9"}
Jul 27 21:06:33.643944 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.643893    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6dcd58f88c-2wdl2" podStartSLOduration=2.245823164 podStartE2EDuration="5.64387447s" totalImagesPullingTime="3.398051306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:06:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:06:33.642928759 +0000 UTC m=+117.010118624" watchObservedRunningTime="2026-07-27 21:06:33.64387447 +0000 UTC m=+117.011064334"
Jul 27 21:06:33.684832 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.684800    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:33.836256 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:33.836203    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 21:06:33.841483 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:06:33.841453    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0104c14_0836_4133_a002_da6d3e8201f0.slice/crio-1c6e777359f5b3bc49845086892c596361f3a33a60e72876e11bbb811e258718 WatchSource:0}: Error finding container 1c6e777359f5b3bc49845086892c596361f3a33a60e72876e11bbb811e258718: Status 404 returned error can't find the container with id 1c6e777359f5b3bc49845086892c596361f3a33a60e72876e11bbb811e258718
Jul 27 21:06:34.629416 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:34.629383    2577 generic.go:356] "Generic (PLEG): container finished" podID="f0104c14-0836-4133-a002-da6d3e8201f0" containerID="73dda3777ef18ae905eff003fc8da402d89c46dbc698796069910318c6e6c7fc" exitCode=0
Jul 27 21:06:34.629824 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:34.629474    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f0104c14-0836-4133-a002-da6d3e8201f0","Type":"ContainerDied","Data":"73dda3777ef18ae905eff003fc8da402d89c46dbc698796069910318c6e6c7fc"}
Jul 27 21:06:34.629824 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:34.629558    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f0104c14-0836-4133-a002-da6d3e8201f0","Type":"ContainerStarted","Data":"1c6e777359f5b3bc49845086892c596361f3a33a60e72876e11bbb811e258718"}
Jul 27 21:06:34.632817 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:34.632796    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7e963cdc-42c5-4717-8d40-a30efe01c299","Type":"ContainerStarted","Data":"f226c0b9947e69b2160eb348c0b631d5d07c1a737f554dce1554cd11c84d69e3"}
Jul 27 21:06:34.716782 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:34.716715    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.929171882 podStartE2EDuration="6.716695951s" totalImagesPullingTime="3.771587084s" totalInitContainerRuntime="15.936985ms" isStatefulPod=false podCreationTimestamp="2026-07-27 21:06:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:06:34.714406393 +0000 UTC m=+118.081596255" watchObservedRunningTime="2026-07-27 21:06:34.716695951 +0000 UTC m=+118.083885818"
Jul 27 21:06:35.735125 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:35.735088    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:35.834973 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:35.834936    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:35.985195 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:35.985105    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:36.081283 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:36.081250    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:36.146157 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:36.146124    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:37.651992 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:37.651924    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f0104c14-0836-4133-a002-da6d3e8201f0","Type":"ContainerStarted","Data":"45f8c41fa8367dfd515edb300dfe2fd9e83838743d31767e12cd89c93161fc6b"}
Jul 27 21:06:37.652467 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:37.652050    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f0104c14-0836-4133-a002-da6d3e8201f0","Type":"ContainerStarted","Data":"e96e82d193435c9c366be7df58fc466fa55ef9cf1804b9a25d0ae87c61d8d8f7"}
Jul 27 21:06:37.676304 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:37.676265    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:37.811814 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:37.811725    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:37.915252 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:37.915213    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:39.039975 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:39.039938    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6dcd58f88c-2wdl2"
Jul 27 21:06:39.040383 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:39.039998    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6dcd58f88c-2wdl2"
Jul 27 21:06:39.045497 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:39.045475    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6dcd58f88c-2wdl2"
Jul 27 21:06:39.662652 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:39.662622    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f0104c14-0836-4133-a002-da6d3e8201f0","Type":"ContainerStarted","Data":"03d9007e8240c96b4e4cf654b1a335a1ebf7a44c5e3844f32b9770d595b60e6b"}
Jul 27 21:06:39.666588 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:39.666565    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6dcd58f88c-2wdl2"
Jul 27 21:06:40.044923 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:40.044838    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:40.145007 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:40.144959    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:40.669919 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:40.669886    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f0104c14-0836-4133-a002-da6d3e8201f0","Type":"ContainerStarted","Data":"fddf5ca2dadecb211ff5f5105ea121b5de4e4dbbf0585ecb456e1109225f3515"}
Jul 27 21:06:40.670075 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:40.669951    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f0104c14-0836-4133-a002-da6d3e8201f0","Type":"ContainerStarted","Data":"a9ffac14b0d393831019345f8ea9b4c04b6029d42d27bf3622ac23d8bdd267d1"}
Jul 27 21:06:40.670075 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:40.669978    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f0104c14-0836-4133-a002-da6d3e8201f0","Type":"ContainerStarted","Data":"b557aaec80a751da3e43f6cf3c340e86630b2a74d7357c20e38a349ad148f2ad"}
Jul 27 21:06:40.700369 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:40.700317    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.952978898 podStartE2EDuration="7.70030097s" totalImagesPullingTime="4.730423994s" totalInitContainerRuntime="16.898078ms" isStatefulPod=false podCreationTimestamp="2026-07-27 21:06:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:06:40.698088731 +0000 UTC m=+124.065278600" watchObservedRunningTime="2026-07-27 21:06:40.70030097 +0000 UTC m=+124.067490836"
Jul 27 21:06:42.117371 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:42.117337    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:42.177717 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:42.177683    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:43.277134 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:43.277108    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:43.377959 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:43.377920    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:43.685132 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:43.685098    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:06:44.829684 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:44.829639    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:44.888052 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:44.888012    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:45.784502 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:45.784462    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6dcd58f88c-2wdl2"]
Jul 27 21:06:45.967954 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:45.967924    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:46.068745 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:46.068665    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:46.717809 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:46.717778    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:46.818220 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:46.818187    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:46.827030 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:46.826999    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3af7400b-f39f-4f41-a492-586171130c30-metrics-certs\") pod \"network-metrics-daemon-wb7n2\" (UID: \"3af7400b-f39f-4f41-a492-586171130c30\") " pod="openshift-multus/network-metrics-daemon-wb7n2"
Jul 27 21:06:46.829372 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:46.829344    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3af7400b-f39f-4f41-a492-586171130c30-metrics-certs\") pod \"network-metrics-daemon-wb7n2\" (UID: \"3af7400b-f39f-4f41-a492-586171130c30\") " pod="openshift-multus/network-metrics-daemon-wb7n2"
Jul 27 21:06:47.045417 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:47.045336    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vs64j\""
Jul 27 21:06:47.052598 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:47.052567    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wb7n2"
Jul 27 21:06:47.253261 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:47.253237    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wb7n2"]
Jul 27 21:06:47.256961 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:06:47.256930    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3af7400b_f39f_4f41_a492_586171130c30.slice/crio-f422664fba9abbebe1463bd9bfb72d25a60d9f445bdcea22af88be1c5accb7e0 WatchSource:0}: Error finding container f422664fba9abbebe1463bd9bfb72d25a60d9f445bdcea22af88be1c5accb7e0: Status 404 returned error can't find the container with id f422664fba9abbebe1463bd9bfb72d25a60d9f445bdcea22af88be1c5accb7e0
Jul 27 21:06:47.696525 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:47.696461    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wb7n2" event={"ID":"3af7400b-f39f-4f41-a492-586171130c30","Type":"ContainerStarted","Data":"f422664fba9abbebe1463bd9bfb72d25a60d9f445bdcea22af88be1c5accb7e0"}
Jul 27 21:06:48.701429 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:48.701396    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wb7n2" event={"ID":"3af7400b-f39f-4f41-a492-586171130c30","Type":"ContainerStarted","Data":"4ffe248daaecb5b0e3c68a4a9d40029b63bc1d59bd2fea102eaf93022bacdeb7"}
Jul 27 21:06:48.701868 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:48.701461    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wb7n2" event={"ID":"3af7400b-f39f-4f41-a492-586171130c30","Type":"ContainerStarted","Data":"0854ad7e64abe1a09c899521c7dbcea997b96170e764e0a290fac3e6307c4318"}
Jul 27 21:06:48.734591 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:48.734533    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-wb7n2" podStartSLOduration=130.636414015 podStartE2EDuration="2m11.734518408s" totalImagesPullingTime="1.098104393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:04:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:06:48.732285039 +0000 UTC m=+132.099474896" watchObservedRunningTime="2026-07-27 21:06:48.734518408 +0000 UTC m=+132.101708274"
Jul 27 21:06:48.990232 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:48.990163    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:49.088895 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:49.088863    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:49.687237 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:49.687202    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:49.793166 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:49.793139    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:52.365625 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:52.365583    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:52.466282 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:52.466254    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:54.192452 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:54.192415    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:54.293134 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:54.293068    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:58.445366 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:58.445329    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:58.547165 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:58.547134    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:58.648686 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:06:58.648661    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:07:00.748616 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:00.748576    2577 generic.go:356] "Generic (PLEG): container finished" podID="59d66d60-9124-4133-b539-5d4649ba0eef" containerID="4af3ed4bb28e826156c72efb64ee41a28af0cd632dd58442a13fd6e301e0e77f" exitCode=0
Jul 27 21:07:00.749247 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:00.748644    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-2cnw9" event={"ID":"59d66d60-9124-4133-b539-5d4649ba0eef","Type":"ContainerDied","Data":"4af3ed4bb28e826156c72efb64ee41a28af0cd632dd58442a13fd6e301e0e77f"}
Jul 27 21:07:00.749247 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:00.749030    2577 scope.go:117] "RemoveContainer" containerID="4af3ed4bb28e826156c72efb64ee41a28af0cd632dd58442a13fd6e301e0e77f"
Jul 27 21:07:01.753993 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:01.753962    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-2cnw9" event={"ID":"59d66d60-9124-4133-b539-5d4649ba0eef","Type":"ContainerStarted","Data":"f163020f1f4abb0403f37eda4ea36a418255e2c34b64dd31936c3bd211c8ef6c"}
Jul 27 21:07:04.648240 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:04.648209    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:07:04.703468 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:04.703435    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:07:10.807006 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:10.806948    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6dcd58f88c-2wdl2" podUID="f2aba92c-19d7-4f7b-8da0-c6d960e57111" containerName="console" containerID="cri-o://24a6b16511da0e23d78ee9eeb4231c93844feadb8e917f01e782b0584b3489c9" gracePeriod=15
Jul 27 21:07:11.076932 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:11.076877    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dcd58f88c-2wdl2"
Jul 27 21:07:11.138134 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:11.138101    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2aba92c-19d7-4f7b-8da0-c6d960e57111-trusted-ca-bundle\") pod \"f2aba92c-19d7-4f7b-8da0-c6d960e57111\" (UID: \"f2aba92c-19d7-4f7b-8da0-c6d960e57111\") "
Jul 27 21:07:11.138134 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:11.138135    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2aba92c-19d7-4f7b-8da0-c6d960e57111-console-serving-cert\") pod \"f2aba92c-19d7-4f7b-8da0-c6d960e57111\" (UID: \"f2aba92c-19d7-4f7b-8da0-c6d960e57111\") "
Jul 27 21:07:11.138321 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:11.138181    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f2aba92c-19d7-4f7b-8da0-c6d960e57111-service-ca\") pod \"f2aba92c-19d7-4f7b-8da0-c6d960e57111\" (UID: \"f2aba92c-19d7-4f7b-8da0-c6d960e57111\") "
Jul 27 21:07:11.138375 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:11.138357    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f2aba92c-19d7-4f7b-8da0-c6d960e57111-console-config\") pod \"f2aba92c-19d7-4f7b-8da0-c6d960e57111\" (UID: \"f2aba92c-19d7-4f7b-8da0-c6d960e57111\") "
Jul 27 21:07:11.138432 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:11.138403    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f2aba92c-19d7-4f7b-8da0-c6d960e57111-console-oauth-config\") pod \"f2aba92c-19d7-4f7b-8da0-c6d960e57111\" (UID: \"f2aba92c-19d7-4f7b-8da0-c6d960e57111\") "
Jul 27 21:07:11.138484 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:11.138460    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f2aba92c-19d7-4f7b-8da0-c6d960e57111-oauth-serving-cert\") pod \"f2aba92c-19d7-4f7b-8da0-c6d960e57111\" (UID: \"f2aba92c-19d7-4f7b-8da0-c6d960e57111\") "
Jul 27 21:07:11.138537 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:11.138489    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h6qln\" (UniqueName: \"kubernetes.io/projected/f2aba92c-19d7-4f7b-8da0-c6d960e57111-kube-api-access-h6qln\") pod \"f2aba92c-19d7-4f7b-8da0-c6d960e57111\" (UID: \"f2aba92c-19d7-4f7b-8da0-c6d960e57111\") "
Jul 27 21:07:11.138537 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:11.138500    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2aba92c-19d7-4f7b-8da0-c6d960e57111-service-ca" (OuterVolumeSpecName: "service-ca") pod "f2aba92c-19d7-4f7b-8da0-c6d960e57111" (UID: "f2aba92c-19d7-4f7b-8da0-c6d960e57111"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:07:11.138666 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:11.138529    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2aba92c-19d7-4f7b-8da0-c6d960e57111-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f2aba92c-19d7-4f7b-8da0-c6d960e57111" (UID: "f2aba92c-19d7-4f7b-8da0-c6d960e57111"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:07:11.138732 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:11.138712    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2aba92c-19d7-4f7b-8da0-c6d960e57111-console-config" (OuterVolumeSpecName: "console-config") pod "f2aba92c-19d7-4f7b-8da0-c6d960e57111" (UID: "f2aba92c-19d7-4f7b-8da0-c6d960e57111"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:07:11.138802 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:11.138742    2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f2aba92c-19d7-4f7b-8da0-c6d960e57111-service-ca\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:07:11.138802 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:11.138757    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2aba92c-19d7-4f7b-8da0-c6d960e57111-trusted-ca-bundle\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:07:11.138884 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:11.138837    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2aba92c-19d7-4f7b-8da0-c6d960e57111-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f2aba92c-19d7-4f7b-8da0-c6d960e57111" (UID: "f2aba92c-19d7-4f7b-8da0-c6d960e57111"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:07:11.140445 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:11.140424    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2aba92c-19d7-4f7b-8da0-c6d960e57111-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f2aba92c-19d7-4f7b-8da0-c6d960e57111" (UID: "f2aba92c-19d7-4f7b-8da0-c6d960e57111"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:07:11.140536 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:11.140516    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2aba92c-19d7-4f7b-8da0-c6d960e57111-kube-api-access-h6qln" (OuterVolumeSpecName: "kube-api-access-h6qln") pod "f2aba92c-19d7-4f7b-8da0-c6d960e57111" (UID: "f2aba92c-19d7-4f7b-8da0-c6d960e57111"). InnerVolumeSpecName "kube-api-access-h6qln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:07:11.140574 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:11.140537    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2aba92c-19d7-4f7b-8da0-c6d960e57111-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f2aba92c-19d7-4f7b-8da0-c6d960e57111" (UID: "f2aba92c-19d7-4f7b-8da0-c6d960e57111"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:07:11.239949 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:11.239905    2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f2aba92c-19d7-4f7b-8da0-c6d960e57111-console-serving-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:07:11.239949 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:11.239943    2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f2aba92c-19d7-4f7b-8da0-c6d960e57111-console-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:07:11.239949 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:11.239956    2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f2aba92c-19d7-4f7b-8da0-c6d960e57111-console-oauth-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:07:11.240176 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:11.239968    2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f2aba92c-19d7-4f7b-8da0-c6d960e57111-oauth-serving-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:07:11.240176 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:11.239981    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6qln\" (UniqueName: \"kubernetes.io/projected/f2aba92c-19d7-4f7b-8da0-c6d960e57111-kube-api-access-h6qln\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:07:11.793926 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:11.793891    2577 generic.go:356] "Generic (PLEG): container finished" podID="f2aba92c-19d7-4f7b-8da0-c6d960e57111" containerID="24a6b16511da0e23d78ee9eeb4231c93844feadb8e917f01e782b0584b3489c9" exitCode=0
Jul 27 21:07:11.794111 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:11.793941    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dcd58f88c-2wdl2"
Jul 27 21:07:11.794111 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:11.793943    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dcd58f88c-2wdl2" event={"ID":"f2aba92c-19d7-4f7b-8da0-c6d960e57111","Type":"ContainerDied","Data":"24a6b16511da0e23d78ee9eeb4231c93844feadb8e917f01e782b0584b3489c9"}
Jul 27 21:07:11.794111 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:11.794072    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dcd58f88c-2wdl2" event={"ID":"f2aba92c-19d7-4f7b-8da0-c6d960e57111","Type":"ContainerDied","Data":"6f12955299e282a071e7c454ff054bee4650fafde2d87afda437ceb3b2344503"}
Jul 27 21:07:11.794111 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:11.794103    2577 scope.go:117] "RemoveContainer" containerID="24a6b16511da0e23d78ee9eeb4231c93844feadb8e917f01e782b0584b3489c9"
Jul 27 21:07:11.804975 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:11.804950    2577 scope.go:117] "RemoveContainer" containerID="24a6b16511da0e23d78ee9eeb4231c93844feadb8e917f01e782b0584b3489c9"
Jul 27 21:07:11.805302 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:07:11.805280    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24a6b16511da0e23d78ee9eeb4231c93844feadb8e917f01e782b0584b3489c9\": container with ID starting with 24a6b16511da0e23d78ee9eeb4231c93844feadb8e917f01e782b0584b3489c9 not found: ID does not exist" containerID="24a6b16511da0e23d78ee9eeb4231c93844feadb8e917f01e782b0584b3489c9"
Jul 27 21:07:11.805359 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:11.805311    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24a6b16511da0e23d78ee9eeb4231c93844feadb8e917f01e782b0584b3489c9"} err="failed to get container status \"24a6b16511da0e23d78ee9eeb4231c93844feadb8e917f01e782b0584b3489c9\": rpc error: code = NotFound desc = could not find container \"24a6b16511da0e23d78ee9eeb4231c93844feadb8e917f01e782b0584b3489c9\": container with ID starting with 24a6b16511da0e23d78ee9eeb4231c93844feadb8e917f01e782b0584b3489c9 not found: ID does not exist"
Jul 27 21:07:11.816936 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:11.816903    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6dcd58f88c-2wdl2"]
Jul 27 21:07:11.820605 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:11.820584    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6dcd58f88c-2wdl2"]
Jul 27 21:07:13.139826 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:13.139792    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2aba92c-19d7-4f7b-8da0-c6d960e57111" path="/var/lib/kubelet/pods/f2aba92c-19d7-4f7b-8da0-c6d960e57111/volumes"
Jul 27 21:07:13.574706 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:13.574665    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:07:33.685543 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:33.685453    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:33.701021 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:33.700995    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:33.899272 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:33.899245    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:46.414497 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:46.414459    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:07:46.699537 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:46.699489    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:07:47.483961 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:47.483923    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 21:07:47.484409 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:47.484214    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7e963cdc-42c5-4717-8d40-a30efe01c299" containerName="alertmanager" containerID="cri-o://a4e08fde302bf8e8073340bcd02b4bce60c7f413e5b7bfdda72f15c2eef83fe4" gracePeriod=120
Jul 27 21:07:47.484409 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:47.484295    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7e963cdc-42c5-4717-8d40-a30efe01c299" containerName="kube-rbac-proxy-web" containerID="cri-o://f481c1e635220c79af7b8b2cd35fddf481a62f7d05b56fbf7892c838ff41eedd" gracePeriod=120
Jul 27 21:07:47.484409 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:47.484321    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7e963cdc-42c5-4717-8d40-a30efe01c299" containerName="config-reloader" containerID="cri-o://29ad99c7a1aea8e8f38b95b56629c2a3e5638dc518f8dc4ef8459d76932b1088" gracePeriod=120
Jul 27 21:07:47.484409 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:47.484319    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7e963cdc-42c5-4717-8d40-a30efe01c299" containerName="kube-rbac-proxy" containerID="cri-o://8490a420afbde38f1318ed19a4601e3e232354dfcb7279a8d7f4ac58431895fa" gracePeriod=120
Jul 27 21:07:47.484409 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:47.484347    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7e963cdc-42c5-4717-8d40-a30efe01c299" containerName="prom-label-proxy" containerID="cri-o://f226c0b9947e69b2160eb348c0b631d5d07c1a737f554dce1554cd11c84d69e3" gracePeriod=120
Jul 27 21:07:47.484409 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:47.484295    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7e963cdc-42c5-4717-8d40-a30efe01c299" containerName="kube-rbac-proxy-metric" containerID="cri-o://959e20d8985878927b64feb5b96562450e4167b5eae5cd6dcefec5cfeecc2c6b" gracePeriod=120
Jul 27 21:07:47.527337 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:07:47.527303    2577 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e963cdc_42c5_4717_8d40_a30efe01c299.slice/crio-conmon-a4e08fde302bf8e8073340bcd02b4bce60c7f413e5b7bfdda72f15c2eef83fe4.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e963cdc_42c5_4717_8d40_a30efe01c299.slice/crio-conmon-f226c0b9947e69b2160eb348c0b631d5d07c1a737f554dce1554cd11c84d69e3.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e963cdc_42c5_4717_8d40_a30efe01c299.slice/crio-a4e08fde302bf8e8073340bcd02b4bce60c7f413e5b7bfdda72f15c2eef83fe4.scope\": RecentStats: unable to find data in memory cache]"
Jul 27 21:07:47.940996 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:47.940968    2577 generic.go:356] "Generic (PLEG): container finished" podID="7e963cdc-42c5-4717-8d40-a30efe01c299" containerID="f226c0b9947e69b2160eb348c0b631d5d07c1a737f554dce1554cd11c84d69e3" exitCode=0
Jul 27 21:07:47.940996 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:47.940989    2577 generic.go:356] "Generic (PLEG): container finished" podID="7e963cdc-42c5-4717-8d40-a30efe01c299" containerID="8490a420afbde38f1318ed19a4601e3e232354dfcb7279a8d7f4ac58431895fa" exitCode=0
Jul 27 21:07:47.940996 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:47.940996    2577 generic.go:356] "Generic (PLEG): container finished" podID="7e963cdc-42c5-4717-8d40-a30efe01c299" containerID="29ad99c7a1aea8e8f38b95b56629c2a3e5638dc518f8dc4ef8459d76932b1088" exitCode=0
Jul 27 21:07:47.940996 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:47.941001    2577 generic.go:356] "Generic (PLEG): container finished" podID="7e963cdc-42c5-4717-8d40-a30efe01c299" containerID="a4e08fde302bf8e8073340bcd02b4bce60c7f413e5b7bfdda72f15c2eef83fe4" exitCode=0
Jul 27 21:07:47.941261 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:47.941041    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7e963cdc-42c5-4717-8d40-a30efe01c299","Type":"ContainerDied","Data":"f226c0b9947e69b2160eb348c0b631d5d07c1a737f554dce1554cd11c84d69e3"}
Jul 27 21:07:47.941261 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:47.941100    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7e963cdc-42c5-4717-8d40-a30efe01c299","Type":"ContainerDied","Data":"8490a420afbde38f1318ed19a4601e3e232354dfcb7279a8d7f4ac58431895fa"}
Jul 27 21:07:47.941261 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:47.941126    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7e963cdc-42c5-4717-8d40-a30efe01c299","Type":"ContainerDied","Data":"29ad99c7a1aea8e8f38b95b56629c2a3e5638dc518f8dc4ef8459d76932b1088"}
Jul 27 21:07:47.941261 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:47.941150    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7e963cdc-42c5-4717-8d40-a30efe01c299","Type":"ContainerDied","Data":"a4e08fde302bf8e8073340bcd02b4bce60c7f413e5b7bfdda72f15c2eef83fe4"}
Jul 27 21:07:48.732617 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:48.732533    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:07:48.871723 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:48.871653    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7e963cdc-42c5-4717-8d40-a30efe01c299-tls-assets\") pod \"7e963cdc-42c5-4717-8d40-a30efe01c299\" (UID: \"7e963cdc-42c5-4717-8d40-a30efe01c299\") "
Jul 27 21:07:48.871723 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:48.871686    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjcbf\" (UniqueName: \"kubernetes.io/projected/7e963cdc-42c5-4717-8d40-a30efe01c299-kube-api-access-tjcbf\") pod \"7e963cdc-42c5-4717-8d40-a30efe01c299\" (UID: \"7e963cdc-42c5-4717-8d40-a30efe01c299\") "
Jul 27 21:07:48.871723 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:48.871722    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7e963cdc-42c5-4717-8d40-a30efe01c299-config-out\") pod \"7e963cdc-42c5-4717-8d40-a30efe01c299\" (UID: \"7e963cdc-42c5-4717-8d40-a30efe01c299\") "
Jul 27 21:07:48.872000 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:48.871743    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7e963cdc-42c5-4717-8d40-a30efe01c299-cluster-tls-config\") pod \"7e963cdc-42c5-4717-8d40-a30efe01c299\" (UID: \"7e963cdc-42c5-4717-8d40-a30efe01c299\") "
Jul 27 21:07:48.872000 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:48.871803    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7e963cdc-42c5-4717-8d40-a30efe01c299-secret-alertmanager-kube-rbac-proxy\") pod \"7e963cdc-42c5-4717-8d40-a30efe01c299\" (UID: \"7e963cdc-42c5-4717-8d40-a30efe01c299\") "
Jul 27 21:07:48.872000 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:48.871837    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7e963cdc-42c5-4717-8d40-a30efe01c299-secret-alertmanager-kube-rbac-proxy-metric\") pod \"7e963cdc-42c5-4717-8d40-a30efe01c299\" (UID: \"7e963cdc-42c5-4717-8d40-a30efe01c299\") "
Jul 27 21:07:48.872000 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:48.871861    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7e963cdc-42c5-4717-8d40-a30efe01c299-web-config\") pod \"7e963cdc-42c5-4717-8d40-a30efe01c299\" (UID: \"7e963cdc-42c5-4717-8d40-a30efe01c299\") "
Jul 27 21:07:48.872000 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:48.871895    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e963cdc-42c5-4717-8d40-a30efe01c299-metrics-client-ca\") pod \"7e963cdc-42c5-4717-8d40-a30efe01c299\" (UID: \"7e963cdc-42c5-4717-8d40-a30efe01c299\") "
Jul 27 21:07:48.872000 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:48.871930    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7e963cdc-42c5-4717-8d40-a30efe01c299-config-volume\") pod \"7e963cdc-42c5-4717-8d40-a30efe01c299\" (UID: \"7e963cdc-42c5-4717-8d40-a30efe01c299\") "
Jul 27 21:07:48.872000 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:48.871966    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7e963cdc-42c5-4717-8d40-a30efe01c299-alertmanager-main-db\") pod \"7e963cdc-42c5-4717-8d40-a30efe01c299\" (UID: \"7e963cdc-42c5-4717-8d40-a30efe01c299\") "
Jul 27 21:07:48.872334 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:48.872007    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e963cdc-42c5-4717-8d40-a30efe01c299-alertmanager-trusted-ca-bundle\") pod \"7e963cdc-42c5-4717-8d40-a30efe01c299\" (UID: \"7e963cdc-42c5-4717-8d40-a30efe01c299\") "
Jul 27 21:07:48.872334 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:48.872038    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7e963cdc-42c5-4717-8d40-a30efe01c299-secret-alertmanager-main-tls\") pod \"7e963cdc-42c5-4717-8d40-a30efe01c299\" (UID: \"7e963cdc-42c5-4717-8d40-a30efe01c299\") "
Jul 27 21:07:48.872334 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:48.872070    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7e963cdc-42c5-4717-8d40-a30efe01c299-secret-alertmanager-kube-rbac-proxy-web\") pod \"7e963cdc-42c5-4717-8d40-a30efe01c299\" (UID: \"7e963cdc-42c5-4717-8d40-a30efe01c299\") "
Jul 27 21:07:48.873225 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:48.872729    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e963cdc-42c5-4717-8d40-a30efe01c299-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "7e963cdc-42c5-4717-8d40-a30efe01c299" (UID: "7e963cdc-42c5-4717-8d40-a30efe01c299"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:07:48.873225 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:48.872984    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e963cdc-42c5-4717-8d40-a30efe01c299-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "7e963cdc-42c5-4717-8d40-a30efe01c299" (UID: "7e963cdc-42c5-4717-8d40-a30efe01c299"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:07:48.873512 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:48.873483    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e963cdc-42c5-4717-8d40-a30efe01c299-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "7e963cdc-42c5-4717-8d40-a30efe01c299" (UID: "7e963cdc-42c5-4717-8d40-a30efe01c299"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:07:48.874609 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:48.874585    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e963cdc-42c5-4717-8d40-a30efe01c299-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "7e963cdc-42c5-4717-8d40-a30efe01c299" (UID: "7e963cdc-42c5-4717-8d40-a30efe01c299"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:07:48.874757 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:48.874608    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e963cdc-42c5-4717-8d40-a30efe01c299-kube-api-access-tjcbf" (OuterVolumeSpecName: "kube-api-access-tjcbf") pod "7e963cdc-42c5-4717-8d40-a30efe01c299" (UID: "7e963cdc-42c5-4717-8d40-a30efe01c299"). InnerVolumeSpecName "kube-api-access-tjcbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:07:48.874902 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:48.874630    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e963cdc-42c5-4717-8d40-a30efe01c299-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "7e963cdc-42c5-4717-8d40-a30efe01c299" (UID: "7e963cdc-42c5-4717-8d40-a30efe01c299"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:07:48.875253 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:48.875216    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e963cdc-42c5-4717-8d40-a30efe01c299-config-volume" (OuterVolumeSpecName: "config-volume") pod "7e963cdc-42c5-4717-8d40-a30efe01c299" (UID: "7e963cdc-42c5-4717-8d40-a30efe01c299"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:07:48.875484 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:48.875457    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e963cdc-42c5-4717-8d40-a30efe01c299-config-out" (OuterVolumeSpecName: "config-out") pod "7e963cdc-42c5-4717-8d40-a30efe01c299" (UID: "7e963cdc-42c5-4717-8d40-a30efe01c299"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:07:48.875646 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:48.875622    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e963cdc-42c5-4717-8d40-a30efe01c299-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "7e963cdc-42c5-4717-8d40-a30efe01c299" (UID: "7e963cdc-42c5-4717-8d40-a30efe01c299"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:07:48.876035 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:48.876007    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e963cdc-42c5-4717-8d40-a30efe01c299-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "7e963cdc-42c5-4717-8d40-a30efe01c299" (UID: "7e963cdc-42c5-4717-8d40-a30efe01c299"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:07:48.876035 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:48.876016    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e963cdc-42c5-4717-8d40-a30efe01c299-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "7e963cdc-42c5-4717-8d40-a30efe01c299" (UID: "7e963cdc-42c5-4717-8d40-a30efe01c299"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:07:48.880303 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:48.880182    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e963cdc-42c5-4717-8d40-a30efe01c299-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "7e963cdc-42c5-4717-8d40-a30efe01c299" (UID: "7e963cdc-42c5-4717-8d40-a30efe01c299"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:07:48.886086 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:48.886065    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e963cdc-42c5-4717-8d40-a30efe01c299-web-config" (OuterVolumeSpecName: "web-config") pod "7e963cdc-42c5-4717-8d40-a30efe01c299" (UID: "7e963cdc-42c5-4717-8d40-a30efe01c299"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:07:48.947867 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:48.947838    2577 generic.go:356] "Generic (PLEG): container finished" podID="7e963cdc-42c5-4717-8d40-a30efe01c299" containerID="959e20d8985878927b64feb5b96562450e4167b5eae5cd6dcefec5cfeecc2c6b" exitCode=0
Jul 27 21:07:48.947867 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:48.947861    2577 generic.go:356] "Generic (PLEG): container finished" podID="7e963cdc-42c5-4717-8d40-a30efe01c299" containerID="f481c1e635220c79af7b8b2cd35fddf481a62f7d05b56fbf7892c838ff41eedd" exitCode=0
Jul 27 21:07:48.947996 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:48.947897    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7e963cdc-42c5-4717-8d40-a30efe01c299","Type":"ContainerDied","Data":"959e20d8985878927b64feb5b96562450e4167b5eae5cd6dcefec5cfeecc2c6b"}
Jul 27 21:07:48.947996 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:48.947907    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:07:48.947996 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:48.947951    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7e963cdc-42c5-4717-8d40-a30efe01c299","Type":"ContainerDied","Data":"f481c1e635220c79af7b8b2cd35fddf481a62f7d05b56fbf7892c838ff41eedd"}
Jul 27 21:07:48.947996 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:48.947980    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7e963cdc-42c5-4717-8d40-a30efe01c299","Type":"ContainerDied","Data":"063d86ee37794b22ccb4f70af9e17c8712909baffef2520f782d57867a1deaa7"}
Jul 27 21:07:48.948120 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:48.948009    2577 scope.go:117] "RemoveContainer" containerID="f226c0b9947e69b2160eb348c0b631d5d07c1a737f554dce1554cd11c84d69e3"
Jul 27 21:07:48.956778 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:48.956744    2577 scope.go:117] "RemoveContainer" containerID="959e20d8985878927b64feb5b96562450e4167b5eae5cd6dcefec5cfeecc2c6b"
Jul 27 21:07:48.965296 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:48.965272    2577 scope.go:117] "RemoveContainer" containerID="8490a420afbde38f1318ed19a4601e3e232354dfcb7279a8d7f4ac58431895fa"
Jul 27 21:07:48.971716 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:48.971699    2577 scope.go:117] "RemoveContainer" containerID="f481c1e635220c79af7b8b2cd35fddf481a62f7d05b56fbf7892c838ff41eedd"
Jul 27 21:07:48.974026 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:48.974002    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7e963cdc-42c5-4717-8d40-a30efe01c299-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:07:48.974093 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:48.974032    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7e963cdc-42c5-4717-8d40-a30efe01c299-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:07:48.974093 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:48.974076    2577 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7e963cdc-42c5-4717-8d40-a30efe01c299-web-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:07:48.974166 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:48.974092    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e963cdc-42c5-4717-8d40-a30efe01c299-metrics-client-ca\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:07:48.974166 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:48.974105    2577 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7e963cdc-42c5-4717-8d40-a30efe01c299-config-volume\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:07:48.974166 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:48.974118    2577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7e963cdc-42c5-4717-8d40-a30efe01c299-alertmanager-main-db\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:07:48.974166 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:48.974132    2577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e963cdc-42c5-4717-8d40-a30efe01c299-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:07:48.974166 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:48.974147    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7e963cdc-42c5-4717-8d40-a30efe01c299-secret-alertmanager-main-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:07:48.974166 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:48.974161    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7e963cdc-42c5-4717-8d40-a30efe01c299-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:07:48.974339 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:48.974174    2577 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7e963cdc-42c5-4717-8d40-a30efe01c299-tls-assets\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:07:48.974339 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:48.974188    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjcbf\" (UniqueName: \"kubernetes.io/projected/7e963cdc-42c5-4717-8d40-a30efe01c299-kube-api-access-tjcbf\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:07:48.974339 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:48.974201    2577 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7e963cdc-42c5-4717-8d40-a30efe01c299-config-out\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:07:48.974339 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:48.974215    2577 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7e963cdc-42c5-4717-8d40-a30efe01c299-cluster-tls-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:07:48.978597 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:48.978574    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 21:07:48.979381 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:48.979352    2577 scope.go:117] "RemoveContainer" containerID="29ad99c7a1aea8e8f38b95b56629c2a3e5638dc518f8dc4ef8459d76932b1088"
Jul 27 21:07:48.985872 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:48.985636    2577 scope.go:117] "RemoveContainer" containerID="a4e08fde302bf8e8073340bcd02b4bce60c7f413e5b7bfdda72f15c2eef83fe4"
Jul 27 21:07:48.988740 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:48.988721    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 21:07:48.993005 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:48.992985    2577 scope.go:117] "RemoveContainer" containerID="5218f39d38b7ec718af11d34923cec8ec078b5d3e9879eda7fe40a59eedcb6dd"
Jul 27 21:07:48.999599 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:48.999582    2577 scope.go:117] "RemoveContainer" containerID="f226c0b9947e69b2160eb348c0b631d5d07c1a737f554dce1554cd11c84d69e3"
Jul 27 21:07:48.999894 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:07:48.999875    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f226c0b9947e69b2160eb348c0b631d5d07c1a737f554dce1554cd11c84d69e3\": container with ID starting with f226c0b9947e69b2160eb348c0b631d5d07c1a737f554dce1554cd11c84d69e3 not found: ID does not exist" containerID="f226c0b9947e69b2160eb348c0b631d5d07c1a737f554dce1554cd11c84d69e3"
Jul 27 21:07:48.999982 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:48.999905    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f226c0b9947e69b2160eb348c0b631d5d07c1a737f554dce1554cd11c84d69e3"} err="failed to get container status \"f226c0b9947e69b2160eb348c0b631d5d07c1a737f554dce1554cd11c84d69e3\": rpc error: code = NotFound desc = could not find container \"f226c0b9947e69b2160eb348c0b631d5d07c1a737f554dce1554cd11c84d69e3\": container with ID starting with f226c0b9947e69b2160eb348c0b631d5d07c1a737f554dce1554cd11c84d69e3 not found: ID does not exist"
Jul 27 21:07:48.999982 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:48.999928    2577 scope.go:117] "RemoveContainer" containerID="959e20d8985878927b64feb5b96562450e4167b5eae5cd6dcefec5cfeecc2c6b"
Jul 27 21:07:49.000205 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:07:49.000188    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"959e20d8985878927b64feb5b96562450e4167b5eae5cd6dcefec5cfeecc2c6b\": container with ID starting with 959e20d8985878927b64feb5b96562450e4167b5eae5cd6dcefec5cfeecc2c6b not found: ID does not exist" containerID="959e20d8985878927b64feb5b96562450e4167b5eae5cd6dcefec5cfeecc2c6b"
Jul 27 21:07:49.000256 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:49.000212    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"959e20d8985878927b64feb5b96562450e4167b5eae5cd6dcefec5cfeecc2c6b"} err="failed to get container status \"959e20d8985878927b64feb5b96562450e4167b5eae5cd6dcefec5cfeecc2c6b\": rpc error: code = NotFound desc = could not find container \"959e20d8985878927b64feb5b96562450e4167b5eae5cd6dcefec5cfeecc2c6b\": container with ID starting with 959e20d8985878927b64feb5b96562450e4167b5eae5cd6dcefec5cfeecc2c6b not found: ID does not exist"
Jul 27 21:07:49.000256 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:49.000229    2577 scope.go:117] "RemoveContainer" containerID="8490a420afbde38f1318ed19a4601e3e232354dfcb7279a8d7f4ac58431895fa"
Jul 27 21:07:49.000448 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:07:49.000430    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8490a420afbde38f1318ed19a4601e3e232354dfcb7279a8d7f4ac58431895fa\": container with ID starting with 8490a420afbde38f1318ed19a4601e3e232354dfcb7279a8d7f4ac58431895fa not found: ID does not exist" containerID="8490a420afbde38f1318ed19a4601e3e232354dfcb7279a8d7f4ac58431895fa"
Jul 27 21:07:49.000496 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:49.000454    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8490a420afbde38f1318ed19a4601e3e232354dfcb7279a8d7f4ac58431895fa"} err="failed to get container status \"8490a420afbde38f1318ed19a4601e3e232354dfcb7279a8d7f4ac58431895fa\": rpc error: code = NotFound desc = could not find container \"8490a420afbde38f1318ed19a4601e3e232354dfcb7279a8d7f4ac58431895fa\": container with ID starting with 8490a420afbde38f1318ed19a4601e3e232354dfcb7279a8d7f4ac58431895fa not found: ID does not exist"
Jul 27 21:07:49.000496 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:49.000470    2577 scope.go:117] "RemoveContainer" containerID="f481c1e635220c79af7b8b2cd35fddf481a62f7d05b56fbf7892c838ff41eedd"
Jul 27 21:07:49.000663 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:07:49.000651    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f481c1e635220c79af7b8b2cd35fddf481a62f7d05b56fbf7892c838ff41eedd\": container with ID starting with f481c1e635220c79af7b8b2cd35fddf481a62f7d05b56fbf7892c838ff41eedd not found: ID does not exist" containerID="f481c1e635220c79af7b8b2cd35fddf481a62f7d05b56fbf7892c838ff41eedd"
Jul 27 21:07:49.000701 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:49.000666    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f481c1e635220c79af7b8b2cd35fddf481a62f7d05b56fbf7892c838ff41eedd"} err="failed to get container status \"f481c1e635220c79af7b8b2cd35fddf481a62f7d05b56fbf7892c838ff41eedd\": rpc error: code = NotFound desc = could not find container \"f481c1e635220c79af7b8b2cd35fddf481a62f7d05b56fbf7892c838ff41eedd\": container with ID starting with f481c1e635220c79af7b8b2cd35fddf481a62f7d05b56fbf7892c838ff41eedd not found: ID does not exist"
Jul 27 21:07:49.000701 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:49.000678    2577 scope.go:117] "RemoveContainer" containerID="29ad99c7a1aea8e8f38b95b56629c2a3e5638dc518f8dc4ef8459d76932b1088"
Jul 27 21:07:49.000884 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:07:49.000863    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29ad99c7a1aea8e8f38b95b56629c2a3e5638dc518f8dc4ef8459d76932b1088\": container with ID starting with 29ad99c7a1aea8e8f38b95b56629c2a3e5638dc518f8dc4ef8459d76932b1088 not found: ID does not exist" containerID="29ad99c7a1aea8e8f38b95b56629c2a3e5638dc518f8dc4ef8459d76932b1088"
Jul 27 21:07:49.000930 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:49.000889    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29ad99c7a1aea8e8f38b95b56629c2a3e5638dc518f8dc4ef8459d76932b1088"} err="failed to get container status \"29ad99c7a1aea8e8f38b95b56629c2a3e5638dc518f8dc4ef8459d76932b1088\": rpc error: code = NotFound desc = could not find container \"29ad99c7a1aea8e8f38b95b56629c2a3e5638dc518f8dc4ef8459d76932b1088\": container with ID starting with 29ad99c7a1aea8e8f38b95b56629c2a3e5638dc518f8dc4ef8459d76932b1088 not found: ID does not exist"
Jul 27 21:07:49.000930 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:49.000902    2577 scope.go:117] "RemoveContainer" containerID="a4e08fde302bf8e8073340bcd02b4bce60c7f413e5b7bfdda72f15c2eef83fe4"
Jul 27 21:07:49.001109 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:07:49.001094    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4e08fde302bf8e8073340bcd02b4bce60c7f413e5b7bfdda72f15c2eef83fe4\": container with ID starting with a4e08fde302bf8e8073340bcd02b4bce60c7f413e5b7bfdda72f15c2eef83fe4 not found: ID does not exist" containerID="a4e08fde302bf8e8073340bcd02b4bce60c7f413e5b7bfdda72f15c2eef83fe4"
Jul 27 21:07:49.001164 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:49.001116    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4e08fde302bf8e8073340bcd02b4bce60c7f413e5b7bfdda72f15c2eef83fe4"} err="failed to get container status \"a4e08fde302bf8e8073340bcd02b4bce60c7f413e5b7bfdda72f15c2eef83fe4\": rpc error: code = NotFound desc = could not find container \"a4e08fde302bf8e8073340bcd02b4bce60c7f413e5b7bfdda72f15c2eef83fe4\": container with ID starting with a4e08fde302bf8e8073340bcd02b4bce60c7f413e5b7bfdda72f15c2eef83fe4 not found: ID does not exist"
Jul 27 21:07:49.001164 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:49.001136    2577 scope.go:117] "RemoveContainer" containerID="5218f39d38b7ec718af11d34923cec8ec078b5d3e9879eda7fe40a59eedcb6dd"
Jul 27 21:07:49.001346 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:07:49.001332    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5218f39d38b7ec718af11d34923cec8ec078b5d3e9879eda7fe40a59eedcb6dd\": container with ID starting with 5218f39d38b7ec718af11d34923cec8ec078b5d3e9879eda7fe40a59eedcb6dd not found: ID does not exist" containerID="5218f39d38b7ec718af11d34923cec8ec078b5d3e9879eda7fe40a59eedcb6dd"
Jul 27 21:07:49.001384 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:49.001351    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5218f39d38b7ec718af11d34923cec8ec078b5d3e9879eda7fe40a59eedcb6dd"} err="failed to get container status \"5218f39d38b7ec718af11d34923cec8ec078b5d3e9879eda7fe40a59eedcb6dd\": rpc error: code = NotFound desc = could not find container \"5218f39d38b7ec718af11d34923cec8ec078b5d3e9879eda7fe40a59eedcb6dd\": container with ID starting with 5218f39d38b7ec718af11d34923cec8ec078b5d3e9879eda7fe40a59eedcb6dd not found: ID does not exist"
Jul 27 21:07:49.001384 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:49.001363    2577 scope.go:117] "RemoveContainer" containerID="f226c0b9947e69b2160eb348c0b631d5d07c1a737f554dce1554cd11c84d69e3"
Jul 27 21:07:49.001574 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:49.001558    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f226c0b9947e69b2160eb348c0b631d5d07c1a737f554dce1554cd11c84d69e3"} err="failed to get container status \"f226c0b9947e69b2160eb348c0b631d5d07c1a737f554dce1554cd11c84d69e3\": rpc error: code = NotFound desc = could not find container \"f226c0b9947e69b2160eb348c0b631d5d07c1a737f554dce1554cd11c84d69e3\": container with ID starting with f226c0b9947e69b2160eb348c0b631d5d07c1a737f554dce1554cd11c84d69e3 not found: ID does not exist"
Jul 27 21:07:49.001636 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:49.001576    2577 scope.go:117] "RemoveContainer" containerID="959e20d8985878927b64feb5b96562450e4167b5eae5cd6dcefec5cfeecc2c6b"
Jul 27 21:07:49.001842 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:49.001824    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"959e20d8985878927b64feb5b96562450e4167b5eae5cd6dcefec5cfeecc2c6b"} err="failed to get container status \"959e20d8985878927b64feb5b96562450e4167b5eae5cd6dcefec5cfeecc2c6b\": rpc error: code = NotFound desc = could not find container \"959e20d8985878927b64feb5b96562450e4167b5eae5cd6dcefec5cfeecc2c6b\": container with ID starting with 959e20d8985878927b64feb5b96562450e4167b5eae5cd6dcefec5cfeecc2c6b not found: ID does not exist"
Jul 27 21:07:49.001886 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:49.001843    2577 scope.go:117] "RemoveContainer" containerID="8490a420afbde38f1318ed19a4601e3e232354dfcb7279a8d7f4ac58431895fa"
Jul 27 21:07:49.002037 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:49.002022    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8490a420afbde38f1318ed19a4601e3e232354dfcb7279a8d7f4ac58431895fa"} err="failed to get container status \"8490a420afbde38f1318ed19a4601e3e232354dfcb7279a8d7f4ac58431895fa\": rpc error: code = NotFound desc = could not find container \"8490a420afbde38f1318ed19a4601e3e232354dfcb7279a8d7f4ac58431895fa\": container with ID starting with 8490a420afbde38f1318ed19a4601e3e232354dfcb7279a8d7f4ac58431895fa not found: ID does not exist"
Jul 27 21:07:49.002083 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:49.002037    2577 scope.go:117] "RemoveContainer" containerID="f481c1e635220c79af7b8b2cd35fddf481a62f7d05b56fbf7892c838ff41eedd"
Jul 27 21:07:49.002245 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:49.002227    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f481c1e635220c79af7b8b2cd35fddf481a62f7d05b56fbf7892c838ff41eedd"} err="failed to get container status \"f481c1e635220c79af7b8b2cd35fddf481a62f7d05b56fbf7892c838ff41eedd\": rpc error: code = NotFound desc = could not find container \"f481c1e635220c79af7b8b2cd35fddf481a62f7d05b56fbf7892c838ff41eedd\": container with ID starting with f481c1e635220c79af7b8b2cd35fddf481a62f7d05b56fbf7892c838ff41eedd not found: ID does not exist"
Jul 27 21:07:49.002303 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:49.002246    2577 scope.go:117] "RemoveContainer" containerID="29ad99c7a1aea8e8f38b95b56629c2a3e5638dc518f8dc4ef8459d76932b1088"
Jul 27 21:07:49.002439 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:49.002424    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29ad99c7a1aea8e8f38b95b56629c2a3e5638dc518f8dc4ef8459d76932b1088"} err="failed to get container status \"29ad99c7a1aea8e8f38b95b56629c2a3e5638dc518f8dc4ef8459d76932b1088\": rpc error: code = NotFound desc = could not find container \"29ad99c7a1aea8e8f38b95b56629c2a3e5638dc518f8dc4ef8459d76932b1088\": container with ID starting with 29ad99c7a1aea8e8f38b95b56629c2a3e5638dc518f8dc4ef8459d76932b1088 not found: ID does not exist"
Jul 27 21:07:49.002483 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:49.002439    2577 scope.go:117] "RemoveContainer" containerID="a4e08fde302bf8e8073340bcd02b4bce60c7f413e5b7bfdda72f15c2eef83fe4"
Jul 27 21:07:49.002631 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:49.002613    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4e08fde302bf8e8073340bcd02b4bce60c7f413e5b7bfdda72f15c2eef83fe4"} err="failed to get container status \"a4e08fde302bf8e8073340bcd02b4bce60c7f413e5b7bfdda72f15c2eef83fe4\": rpc error: code = NotFound desc = could not find container \"a4e08fde302bf8e8073340bcd02b4bce60c7f413e5b7bfdda72f15c2eef83fe4\": container with ID starting with a4e08fde302bf8e8073340bcd02b4bce60c7f413e5b7bfdda72f15c2eef83fe4 not found: ID does not exist"
Jul 27 21:07:49.002696 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:49.002632    2577 scope.go:117] "RemoveContainer" containerID="5218f39d38b7ec718af11d34923cec8ec078b5d3e9879eda7fe40a59eedcb6dd"
Jul 27 21:07:49.002846 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:49.002826    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5218f39d38b7ec718af11d34923cec8ec078b5d3e9879eda7fe40a59eedcb6dd"} err="failed to get container status \"5218f39d38b7ec718af11d34923cec8ec078b5d3e9879eda7fe40a59eedcb6dd\": rpc error: code = NotFound desc = could not find container \"5218f39d38b7ec718af11d34923cec8ec078b5d3e9879eda7fe40a59eedcb6dd\": container with ID starting with 5218f39d38b7ec718af11d34923cec8ec078b5d3e9879eda7fe40a59eedcb6dd not found: ID does not exist"
Jul 27 21:07:49.135440 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:49.135367    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e963cdc-42c5-4717-8d40-a30efe01c299" path="/var/lib/kubelet/pods/7e963cdc-42c5-4717-8d40-a30efe01c299/volumes"
Jul 27 21:07:51.788313 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:51.788278    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 21:07:51.788824 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:51.788665    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f0104c14-0836-4133-a002-da6d3e8201f0" containerName="prometheus" containerID="cri-o://e96e82d193435c9c366be7df58fc466fa55ef9cf1804b9a25d0ae87c61d8d8f7" gracePeriod=600
Jul 27 21:07:51.788824 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:51.788693    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f0104c14-0836-4133-a002-da6d3e8201f0" containerName="thanos-sidecar" containerID="cri-o://03d9007e8240c96b4e4cf654b1a335a1ebf7a44c5e3844f32b9770d595b60e6b" gracePeriod=600
Jul 27 21:07:51.788956 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:51.788832    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f0104c14-0836-4133-a002-da6d3e8201f0" containerName="config-reloader" containerID="cri-o://45f8c41fa8367dfd515edb300dfe2fd9e83838743d31767e12cd89c93161fc6b" gracePeriod=600
Jul 27 21:07:51.788956 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:51.788866    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f0104c14-0836-4133-a002-da6d3e8201f0" containerName="kube-rbac-proxy-web" containerID="cri-o://b557aaec80a751da3e43f6cf3c340e86630b2a74d7357c20e38a349ad148f2ad" gracePeriod=600
Jul 27 21:07:51.789048 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:51.788971    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f0104c14-0836-4133-a002-da6d3e8201f0" containerName="kube-rbac-proxy-thanos" containerID="cri-o://fddf5ca2dadecb211ff5f5105ea121b5de4e4dbbf0585ecb456e1109225f3515" gracePeriod=600
Jul 27 21:07:51.789048 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:51.789015    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f0104c14-0836-4133-a002-da6d3e8201f0" containerName="kube-rbac-proxy" containerID="cri-o://a9ffac14b0d393831019345f8ea9b4c04b6029d42d27bf3622ac23d8bdd267d1" gracePeriod=600
Jul 27 21:07:51.964852 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:51.964817    2577 generic.go:356] "Generic (PLEG): container finished" podID="f0104c14-0836-4133-a002-da6d3e8201f0" containerID="fddf5ca2dadecb211ff5f5105ea121b5de4e4dbbf0585ecb456e1109225f3515" exitCode=0
Jul 27 21:07:51.964852 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:51.964850    2577 generic.go:356] "Generic (PLEG): container finished" podID="f0104c14-0836-4133-a002-da6d3e8201f0" containerID="a9ffac14b0d393831019345f8ea9b4c04b6029d42d27bf3622ac23d8bdd267d1" exitCode=0
Jul 27 21:07:51.965244 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:51.964860    2577 generic.go:356] "Generic (PLEG): container finished" podID="f0104c14-0836-4133-a002-da6d3e8201f0" containerID="b557aaec80a751da3e43f6cf3c340e86630b2a74d7357c20e38a349ad148f2ad" exitCode=0
Jul 27 21:07:51.965244 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:51.964869    2577 generic.go:356] "Generic (PLEG): container finished" podID="f0104c14-0836-4133-a002-da6d3e8201f0" containerID="03d9007e8240c96b4e4cf654b1a335a1ebf7a44c5e3844f32b9770d595b60e6b" exitCode=0
Jul 27 21:07:51.965244 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:51.964877    2577 generic.go:356] "Generic (PLEG): container finished" podID="f0104c14-0836-4133-a002-da6d3e8201f0" containerID="45f8c41fa8367dfd515edb300dfe2fd9e83838743d31767e12cd89c93161fc6b" exitCode=0
Jul 27 21:07:51.965244 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:51.964891    2577 generic.go:356] "Generic (PLEG): container finished" podID="f0104c14-0836-4133-a002-da6d3e8201f0" containerID="e96e82d193435c9c366be7df58fc466fa55ef9cf1804b9a25d0ae87c61d8d8f7" exitCode=0
Jul 27 21:07:51.965244 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:51.964898    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f0104c14-0836-4133-a002-da6d3e8201f0","Type":"ContainerDied","Data":"fddf5ca2dadecb211ff5f5105ea121b5de4e4dbbf0585ecb456e1109225f3515"}
Jul 27 21:07:51.965244 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:51.964983    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f0104c14-0836-4133-a002-da6d3e8201f0","Type":"ContainerDied","Data":"a9ffac14b0d393831019345f8ea9b4c04b6029d42d27bf3622ac23d8bdd267d1"}
Jul 27 21:07:51.965244 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:51.965026    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f0104c14-0836-4133-a002-da6d3e8201f0","Type":"ContainerDied","Data":"b557aaec80a751da3e43f6cf3c340e86630b2a74d7357c20e38a349ad148f2ad"}
Jul 27 21:07:51.965244 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:51.965065    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f0104c14-0836-4133-a002-da6d3e8201f0","Type":"ContainerDied","Data":"03d9007e8240c96b4e4cf654b1a335a1ebf7a44c5e3844f32b9770d595b60e6b"}
Jul 27 21:07:51.965244 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:51.965097    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f0104c14-0836-4133-a002-da6d3e8201f0","Type":"ContainerDied","Data":"45f8c41fa8367dfd515edb300dfe2fd9e83838743d31767e12cd89c93161fc6b"}
Jul 27 21:07:51.965244 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:51.965134    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f0104c14-0836-4133-a002-da6d3e8201f0","Type":"ContainerDied","Data":"e96e82d193435c9c366be7df58fc466fa55ef9cf1804b9a25d0ae87c61d8d8f7"}
Jul 27 21:07:52.044430 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.044377    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:52.094400 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.094373    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f0104c14-0836-4133-a002-da6d3e8201f0-prometheus-k8s-rulefiles-0\") pod \"f0104c14-0836-4133-a002-da6d3e8201f0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") "
Jul 27 21:07:52.094548 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.094404    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f0104c14-0836-4133-a002-da6d3e8201f0-configmap-metrics-client-ca\") pod \"f0104c14-0836-4133-a002-da6d3e8201f0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") "
Jul 27 21:07:52.094548 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.094433    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f0104c14-0836-4133-a002-da6d3e8201f0-secret-grpc-tls\") pod \"f0104c14-0836-4133-a002-da6d3e8201f0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") "
Jul 27 21:07:52.094548 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.094480    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f0104c14-0836-4133-a002-da6d3e8201f0-tls-assets\") pod \"f0104c14-0836-4133-a002-da6d3e8201f0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") "
Jul 27 21:07:52.094548 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.094510    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f0104c14-0836-4133-a002-da6d3e8201f0-secret-metrics-client-certs\") pod \"f0104c14-0836-4133-a002-da6d3e8201f0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") "
Jul 27 21:07:52.094725 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.094622    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f0104c14-0836-4133-a002-da6d3e8201f0-prometheus-k8s-rulefiles-1\") pod \"f0104c14-0836-4133-a002-da6d3e8201f0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") "
Jul 27 21:07:52.094725 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.094663    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f0104c14-0836-4133-a002-da6d3e8201f0-prometheus-k8s-rulefiles-2\") pod \"f0104c14-0836-4133-a002-da6d3e8201f0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") "
Jul 27 21:07:52.094725 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.094698    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0104c14-0836-4133-a002-da6d3e8201f0-prometheus-trusted-ca-bundle\") pod \"f0104c14-0836-4133-a002-da6d3e8201f0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") "
Jul 27 21:07:52.094861 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.094732    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f0104c14-0836-4133-a002-da6d3e8201f0-thanos-prometheus-http-client-file\") pod \"f0104c14-0836-4133-a002-da6d3e8201f0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") "
Jul 27 21:07:52.094861 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.094813    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0104c14-0836-4133-a002-da6d3e8201f0-configmap-serving-certs-ca-bundle\") pod \"f0104c14-0836-4133-a002-da6d3e8201f0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") "
Jul 27 21:07:52.094861 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.094851    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0104c14-0836-4133-a002-da6d3e8201f0-configmap-kubelet-serving-ca-bundle\") pod \"f0104c14-0836-4133-a002-da6d3e8201f0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") "
Jul 27 21:07:52.094991 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.094876    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f0104c14-0836-4133-a002-da6d3e8201f0-secret-kube-rbac-proxy\") pod \"f0104c14-0836-4133-a002-da6d3e8201f0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") "
Jul 27 21:07:52.094991 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.094915    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f0104c14-0836-4133-a002-da6d3e8201f0-web-config\") pod \"f0104c14-0836-4133-a002-da6d3e8201f0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") "
Jul 27 21:07:52.094991 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.094944    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f0104c14-0836-4133-a002-da6d3e8201f0-config-out\") pod \"f0104c14-0836-4133-a002-da6d3e8201f0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") "
Jul 27 21:07:52.094991 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.094972    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f0104c14-0836-4133-a002-da6d3e8201f0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"f0104c14-0836-4133-a002-da6d3e8201f0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") "
Jul 27 21:07:52.095147 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.095011    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f0104c14-0836-4133-a002-da6d3e8201f0-secret-prometheus-k8s-tls\") pod \"f0104c14-0836-4133-a002-da6d3e8201f0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") "
Jul 27 21:07:52.095147 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.095038    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f0104c14-0836-4133-a002-da6d3e8201f0-config\") pod \"f0104c14-0836-4133-a002-da6d3e8201f0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") "
Jul 27 21:07:52.095147 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.095073    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k5lgt\" (UniqueName: \"kubernetes.io/projected/f0104c14-0836-4133-a002-da6d3e8201f0-kube-api-access-k5lgt\") pod \"f0104c14-0836-4133-a002-da6d3e8201f0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") "
Jul 27 21:07:52.095147 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.095107    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f0104c14-0836-4133-a002-da6d3e8201f0-prometheus-k8s-db\") pod \"f0104c14-0836-4133-a002-da6d3e8201f0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") "
Jul 27 21:07:52.095310 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.095153    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f0104c14-0836-4133-a002-da6d3e8201f0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"f0104c14-0836-4133-a002-da6d3e8201f0\" (UID: \"f0104c14-0836-4133-a002-da6d3e8201f0\") "
Jul 27 21:07:52.095441 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.095414    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0104c14-0836-4133-a002-da6d3e8201f0-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "f0104c14-0836-4133-a002-da6d3e8201f0" (UID: "f0104c14-0836-4133-a002-da6d3e8201f0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:07:52.095944 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.095835    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0104c14-0836-4133-a002-da6d3e8201f0-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "f0104c14-0836-4133-a002-da6d3e8201f0" (UID: "f0104c14-0836-4133-a002-da6d3e8201f0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:07:52.096470 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.096196    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0104c14-0836-4133-a002-da6d3e8201f0-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "f0104c14-0836-4133-a002-da6d3e8201f0" (UID: "f0104c14-0836-4133-a002-da6d3e8201f0"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:07:52.096470 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.096226    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0104c14-0836-4133-a002-da6d3e8201f0-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "f0104c14-0836-4133-a002-da6d3e8201f0" (UID: "f0104c14-0836-4133-a002-da6d3e8201f0"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:07:52.096998 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.096778    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0104c14-0836-4133-a002-da6d3e8201f0-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "f0104c14-0836-4133-a002-da6d3e8201f0" (UID: "f0104c14-0836-4133-a002-da6d3e8201f0"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:07:52.097823 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.097795    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0104c14-0836-4133-a002-da6d3e8201f0-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "f0104c14-0836-4133-a002-da6d3e8201f0" (UID: "f0104c14-0836-4133-a002-da6d3e8201f0"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:07:52.098411 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.098190    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0104c14-0836-4133-a002-da6d3e8201f0-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "f0104c14-0836-4133-a002-da6d3e8201f0" (UID: "f0104c14-0836-4133-a002-da6d3e8201f0"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:07:52.099011 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.098988    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0104c14-0836-4133-a002-da6d3e8201f0-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "f0104c14-0836-4133-a002-da6d3e8201f0" (UID: "f0104c14-0836-4133-a002-da6d3e8201f0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:07:52.099314 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.099291    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0104c14-0836-4133-a002-da6d3e8201f0-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "f0104c14-0836-4133-a002-da6d3e8201f0" (UID: "f0104c14-0836-4133-a002-da6d3e8201f0"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:07:52.100726 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.100692    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0104c14-0836-4133-a002-da6d3e8201f0-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "f0104c14-0836-4133-a002-da6d3e8201f0" (UID: "f0104c14-0836-4133-a002-da6d3e8201f0"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:07:52.101060 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.101025    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0104c14-0836-4133-a002-da6d3e8201f0-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "f0104c14-0836-4133-a002-da6d3e8201f0" (UID: "f0104c14-0836-4133-a002-da6d3e8201f0"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:07:52.104459 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.104145    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0104c14-0836-4133-a002-da6d3e8201f0-config-out" (OuterVolumeSpecName: "config-out") pod "f0104c14-0836-4133-a002-da6d3e8201f0" (UID: "f0104c14-0836-4133-a002-da6d3e8201f0"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:07:52.104459 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.104170    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0104c14-0836-4133-a002-da6d3e8201f0-kube-api-access-k5lgt" (OuterVolumeSpecName: "kube-api-access-k5lgt") pod "f0104c14-0836-4133-a002-da6d3e8201f0" (UID: "f0104c14-0836-4133-a002-da6d3e8201f0"). InnerVolumeSpecName "kube-api-access-k5lgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:07:52.104459 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.104237    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0104c14-0836-4133-a002-da6d3e8201f0-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "f0104c14-0836-4133-a002-da6d3e8201f0" (UID: "f0104c14-0836-4133-a002-da6d3e8201f0"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:07:52.104459 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.104318    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0104c14-0836-4133-a002-da6d3e8201f0-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "f0104c14-0836-4133-a002-da6d3e8201f0" (UID: "f0104c14-0836-4133-a002-da6d3e8201f0"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:07:52.104713 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.104557    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0104c14-0836-4133-a002-da6d3e8201f0-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "f0104c14-0836-4133-a002-da6d3e8201f0" (UID: "f0104c14-0836-4133-a002-da6d3e8201f0"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:07:52.104947 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.104918    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0104c14-0836-4133-a002-da6d3e8201f0-config" (OuterVolumeSpecName: "config") pod "f0104c14-0836-4133-a002-da6d3e8201f0" (UID: "f0104c14-0836-4133-a002-da6d3e8201f0"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:07:52.105038 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.104940    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0104c14-0836-4133-a002-da6d3e8201f0-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "f0104c14-0836-4133-a002-da6d3e8201f0" (UID: "f0104c14-0836-4133-a002-da6d3e8201f0"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:07:52.105827 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.105808    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0104c14-0836-4133-a002-da6d3e8201f0-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "f0104c14-0836-4133-a002-da6d3e8201f0" (UID: "f0104c14-0836-4133-a002-da6d3e8201f0"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:07:52.112803 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.112751    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0104c14-0836-4133-a002-da6d3e8201f0-web-config" (OuterVolumeSpecName: "web-config") pod "f0104c14-0836-4133-a002-da6d3e8201f0" (UID: "f0104c14-0836-4133-a002-da6d3e8201f0"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:07:52.196671 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.196643    2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f0104c14-0836-4133-a002-da6d3e8201f0-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:07:52.196671 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.196667    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f0104c14-0836-4133-a002-da6d3e8201f0-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:07:52.196839 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.196677    2577 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f0104c14-0836-4133-a002-da6d3e8201f0-configmap-metrics-client-ca\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:07:52.196839 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.196687    2577 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f0104c14-0836-4133-a002-da6d3e8201f0-secret-grpc-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:07:52.196839 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.196695    2577 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f0104c14-0836-4133-a002-da6d3e8201f0-tls-assets\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:07:52.196839 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.196703    2577 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f0104c14-0836-4133-a002-da6d3e8201f0-secret-metrics-client-certs\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:07:52.196839 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.196711    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f0104c14-0836-4133-a002-da6d3e8201f0-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:07:52.196839 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.196719    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f0104c14-0836-4133-a002-da6d3e8201f0-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:07:52.196839 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.196727    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0104c14-0836-4133-a002-da6d3e8201f0-prometheus-trusted-ca-bundle\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:07:52.196839 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.196735    2577 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f0104c14-0836-4133-a002-da6d3e8201f0-thanos-prometheus-http-client-file\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:07:52.196839 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.196743    2577 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0104c14-0836-4133-a002-da6d3e8201f0-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:07:52.196839 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.196753    2577 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0104c14-0836-4133-a002-da6d3e8201f0-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:07:52.196839 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.196772    2577 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f0104c14-0836-4133-a002-da6d3e8201f0-secret-kube-rbac-proxy\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:07:52.196839 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.196780    2577 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f0104c14-0836-4133-a002-da6d3e8201f0-web-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:07:52.196839 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.196788    2577 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f0104c14-0836-4133-a002-da6d3e8201f0-config-out\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:07:52.196839 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.196795    2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f0104c14-0836-4133-a002-da6d3e8201f0-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:07:52.196839 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.196805    2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f0104c14-0836-4133-a002-da6d3e8201f0-secret-prometheus-k8s-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:07:52.196839 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.196814    2577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/f0104c14-0836-4133-a002-da6d3e8201f0-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:07:52.196839 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.196821    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5lgt\" (UniqueName: \"kubernetes.io/projected/f0104c14-0836-4133-a002-da6d3e8201f0-kube-api-access-k5lgt\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:07:52.196839 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.196830    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f0104c14-0836-4133-a002-da6d3e8201f0-prometheus-k8s-db\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:07:52.972072 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.972040    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f0104c14-0836-4133-a002-da6d3e8201f0","Type":"ContainerDied","Data":"1c6e777359f5b3bc49845086892c596361f3a33a60e72876e11bbb811e258718"}
Jul 27 21:07:52.972428 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.972053    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:52.972428 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.972131    2577 scope.go:117] "RemoveContainer" containerID="fddf5ca2dadecb211ff5f5105ea121b5de4e4dbbf0585ecb456e1109225f3515"
Jul 27 21:07:52.979979 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.979964    2577 scope.go:117] "RemoveContainer" containerID="a9ffac14b0d393831019345f8ea9b4c04b6029d42d27bf3622ac23d8bdd267d1"
Jul 27 21:07:52.986672 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.986656    2577 scope.go:117] "RemoveContainer" containerID="b557aaec80a751da3e43f6cf3c340e86630b2a74d7357c20e38a349ad148f2ad"
Jul 27 21:07:52.992993 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.992977    2577 scope.go:117] "RemoveContainer" containerID="03d9007e8240c96b4e4cf654b1a335a1ebf7a44c5e3844f32b9770d595b60e6b"
Jul 27 21:07:52.997113 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.997093    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 21:07:52.999637 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:52.999620    2577 scope.go:117] "RemoveContainer" containerID="45f8c41fa8367dfd515edb300dfe2fd9e83838743d31767e12cd89c93161fc6b"
Jul 27 21:07:53.004153 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.004133    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 21:07:53.006243 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.006228    2577 scope.go:117] "RemoveContainer" containerID="e96e82d193435c9c366be7df58fc466fa55ef9cf1804b9a25d0ae87c61d8d8f7"
Jul 27 21:07:53.016991 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.016967    2577 scope.go:117] "RemoveContainer" containerID="73dda3777ef18ae905eff003fc8da402d89c46dbc698796069910318c6e6c7fc"
Jul 27 21:07:53.032234 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.032207    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 21:07:53.032456 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.032444    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0104c14-0836-4133-a002-da6d3e8201f0" containerName="kube-rbac-proxy"
Jul 27 21:07:53.032526 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.032457    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0104c14-0836-4133-a002-da6d3e8201f0" containerName="kube-rbac-proxy"
Jul 27 21:07:53.032526 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.032467    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e963cdc-42c5-4717-8d40-a30efe01c299" containerName="config-reloader"
Jul 27 21:07:53.032526 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.032472    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e963cdc-42c5-4717-8d40-a30efe01c299" containerName="config-reloader"
Jul 27 21:07:53.032526 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.032483    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e963cdc-42c5-4717-8d40-a30efe01c299" containerName="kube-rbac-proxy-metric"
Jul 27 21:07:53.032526 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.032488    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e963cdc-42c5-4717-8d40-a30efe01c299" containerName="kube-rbac-proxy-metric"
Jul 27 21:07:53.032526 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.032494    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0104c14-0836-4133-a002-da6d3e8201f0" containerName="prometheus"
Jul 27 21:07:53.032526 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.032498    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0104c14-0836-4133-a002-da6d3e8201f0" containerName="prometheus"
Jul 27 21:07:53.032526 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.032505    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0104c14-0836-4133-a002-da6d3e8201f0" containerName="kube-rbac-proxy-web"
Jul 27 21:07:53.032526 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.032510    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0104c14-0836-4133-a002-da6d3e8201f0" containerName="kube-rbac-proxy-web"
Jul 27 21:07:53.032526 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.032517    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e963cdc-42c5-4717-8d40-a30efe01c299" containerName="init-config-reloader"
Jul 27 21:07:53.032526 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.032522    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e963cdc-42c5-4717-8d40-a30efe01c299" containerName="init-config-reloader"
Jul 27 21:07:53.032526 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.032528    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0104c14-0836-4133-a002-da6d3e8201f0" containerName="init-config-reloader"
Jul 27 21:07:53.032882 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.032533    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0104c14-0836-4133-a002-da6d3e8201f0" containerName="init-config-reloader"
Jul 27 21:07:53.032882 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.032541    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0104c14-0836-4133-a002-da6d3e8201f0" containerName="config-reloader"
Jul 27 21:07:53.032882 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.032546    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0104c14-0836-4133-a002-da6d3e8201f0" containerName="config-reloader"
Jul 27 21:07:53.032882 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.032553    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e963cdc-42c5-4717-8d40-a30efe01c299" containerName="kube-rbac-proxy"
Jul 27 21:07:53.032882 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.032557    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e963cdc-42c5-4717-8d40-a30efe01c299" containerName="kube-rbac-proxy"
Jul 27 21:07:53.032882 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.032562    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0104c14-0836-4133-a002-da6d3e8201f0" containerName="thanos-sidecar"
Jul 27 21:07:53.032882 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.032566    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0104c14-0836-4133-a002-da6d3e8201f0" containerName="thanos-sidecar"
Jul 27 21:07:53.032882 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.032571    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0104c14-0836-4133-a002-da6d3e8201f0" containerName="kube-rbac-proxy-thanos"
Jul 27 21:07:53.032882 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.032576    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0104c14-0836-4133-a002-da6d3e8201f0" containerName="kube-rbac-proxy-thanos"
Jul 27 21:07:53.032882 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.032582    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e963cdc-42c5-4717-8d40-a30efe01c299" containerName="prom-label-proxy"
Jul 27 21:07:53.032882 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.032586    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e963cdc-42c5-4717-8d40-a30efe01c299" containerName="prom-label-proxy"
Jul 27 21:07:53.032882 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.032595    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e963cdc-42c5-4717-8d40-a30efe01c299" containerName="alertmanager"
Jul 27 21:07:53.032882 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.032599    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e963cdc-42c5-4717-8d40-a30efe01c299" containerName="alertmanager"
Jul 27 21:07:53.032882 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.032605    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e963cdc-42c5-4717-8d40-a30efe01c299" containerName="kube-rbac-proxy-web"
Jul 27 21:07:53.032882 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.032610    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e963cdc-42c5-4717-8d40-a30efe01c299" containerName="kube-rbac-proxy-web"
Jul 27 21:07:53.032882 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.032616    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2aba92c-19d7-4f7b-8da0-c6d960e57111" containerName="console"
Jul 27 21:07:53.032882 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.032620    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2aba92c-19d7-4f7b-8da0-c6d960e57111" containerName="console"
Jul 27 21:07:53.032882 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.032657    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e963cdc-42c5-4717-8d40-a30efe01c299" containerName="kube-rbac-proxy-metric"
Jul 27 21:07:53.032882 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.032664    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e963cdc-42c5-4717-8d40-a30efe01c299" containerName="prom-label-proxy"
Jul 27 21:07:53.032882 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.032671    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e963cdc-42c5-4717-8d40-a30efe01c299" containerName="alertmanager"
Jul 27 21:07:53.032882 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.032677    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e963cdc-42c5-4717-8d40-a30efe01c299" containerName="kube-rbac-proxy-web"
Jul 27 21:07:53.032882 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.032682    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0104c14-0836-4133-a002-da6d3e8201f0" containerName="config-reloader"
Jul 27 21:07:53.032882 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.032688    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e963cdc-42c5-4717-8d40-a30efe01c299" containerName="config-reloader"
Jul 27 21:07:53.032882 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.032692    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0104c14-0836-4133-a002-da6d3e8201f0" containerName="kube-rbac-proxy-web"
Jul 27 21:07:53.032882 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.032697    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0104c14-0836-4133-a002-da6d3e8201f0" containerName="kube-rbac-proxy-thanos"
Jul 27 21:07:53.032882 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.032702    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e963cdc-42c5-4717-8d40-a30efe01c299" containerName="kube-rbac-proxy"
Jul 27 21:07:53.032882 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.032707    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0104c14-0836-4133-a002-da6d3e8201f0" containerName="prometheus"
Jul 27 21:07:53.032882 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.032712    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2aba92c-19d7-4f7b-8da0-c6d960e57111" containerName="console"
Jul 27 21:07:53.032882 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.032717    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0104c14-0836-4133-a002-da6d3e8201f0" containerName="thanos-sidecar"
Jul 27 21:07:53.032882 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.032742    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0104c14-0836-4133-a002-da6d3e8201f0" containerName="kube-rbac-proxy"
Jul 27 21:07:53.036255 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.036237    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.039277 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.039259    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 27 21:07:53.039353 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.039288    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 27 21:07:53.039747 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.039734    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 27 21:07:53.040464 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.040436    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 27 21:07:53.040560 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.040511    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 27 21:07:53.040560 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.040531    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-l4lcr\""
Jul 27 21:07:53.040998 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.040974    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 27 21:07:53.041094 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.040976    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 27 21:07:53.041649 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.041612    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 27 21:07:53.042135 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.042109    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 27 21:07:53.042305 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.042288    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 27 21:07:53.042357 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.042338    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 27 21:07:53.043013 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.042994    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 27 21:07:53.043509 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.043353    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 27 21:07:53.043593 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.043530    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fl5nch1nqsboi\""
Jul 27 21:07:53.044334 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.044316    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 27 21:07:53.048164 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.048147    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 27 21:07:53.063466 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.063416    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 21:07:53.103664 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.103640    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0f14c1b0-8963-4e4a-8f39-1501689b67d6-config-out\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.103810 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.103669    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0f14c1b0-8963-4e4a-8f39-1501689b67d6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.103810 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.103710    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0f14c1b0-8963-4e4a-8f39-1501689b67d6-web-config\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.103810 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.103782    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0f14c1b0-8963-4e4a-8f39-1501689b67d6-config\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.103810 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.103800    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0f14c1b0-8963-4e4a-8f39-1501689b67d6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.104036 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.103820    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0f14c1b0-8963-4e4a-8f39-1501689b67d6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.104036 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.103837    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s4c8\" (UniqueName: \"kubernetes.io/projected/0f14c1b0-8963-4e4a-8f39-1501689b67d6-kube-api-access-9s4c8\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.104036 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.103861    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0f14c1b0-8963-4e4a-8f39-1501689b67d6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.104036 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.103903    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f14c1b0-8963-4e4a-8f39-1501689b67d6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.104036 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.103940    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0f14c1b0-8963-4e4a-8f39-1501689b67d6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.104036 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.103964    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0f14c1b0-8963-4e4a-8f39-1501689b67d6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.104036 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.103979    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0f14c1b0-8963-4e4a-8f39-1501689b67d6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.104036 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.103995    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f14c1b0-8963-4e4a-8f39-1501689b67d6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.104036 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.104016    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0f14c1b0-8963-4e4a-8f39-1501689b67d6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.104439 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.104081    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0f14c1b0-8963-4e4a-8f39-1501689b67d6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.104439 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.104102    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f14c1b0-8963-4e4a-8f39-1501689b67d6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.104439 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.104119    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f14c1b0-8963-4e4a-8f39-1501689b67d6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.104439 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.104135    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0f14c1b0-8963-4e4a-8f39-1501689b67d6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.104439 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.104150    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0f14c1b0-8963-4e4a-8f39-1501689b67d6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.104439 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.104179    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0f14c1b0-8963-4e4a-8f39-1501689b67d6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.135483 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.135461    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0104c14-0836-4133-a002-da6d3e8201f0" path="/var/lib/kubelet/pods/f0104c14-0836-4133-a002-da6d3e8201f0/volumes"
Jul 27 21:07:53.204755 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.204725    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0f14c1b0-8963-4e4a-8f39-1501689b67d6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.204854 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.204778    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f14c1b0-8963-4e4a-8f39-1501689b67d6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.204854 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.204805    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0f14c1b0-8963-4e4a-8f39-1501689b67d6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.204854 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.204832    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0f14c1b0-8963-4e4a-8f39-1501689b67d6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.204990 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.204859    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0f14c1b0-8963-4e4a-8f39-1501689b67d6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.204990 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.204907    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f14c1b0-8963-4e4a-8f39-1501689b67d6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.204990 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.204945    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0f14c1b0-8963-4e4a-8f39-1501689b67d6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.204990 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.204986    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0f14c1b0-8963-4e4a-8f39-1501689b67d6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.205182 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.205014    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f14c1b0-8963-4e4a-8f39-1501689b67d6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.205182 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.205036    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f14c1b0-8963-4e4a-8f39-1501689b67d6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.205182 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.205058    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0f14c1b0-8963-4e4a-8f39-1501689b67d6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.205182 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.205078    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0f14c1b0-8963-4e4a-8f39-1501689b67d6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.205182 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.205109    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0f14c1b0-8963-4e4a-8f39-1501689b67d6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.205182 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.205144    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0f14c1b0-8963-4e4a-8f39-1501689b67d6-config-out\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.205182 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.205171    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0f14c1b0-8963-4e4a-8f39-1501689b67d6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.205516 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.205203    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0f14c1b0-8963-4e4a-8f39-1501689b67d6-web-config\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.205516 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.205237    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0f14c1b0-8963-4e4a-8f39-1501689b67d6-config\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.205516 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.205258    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0f14c1b0-8963-4e4a-8f39-1501689b67d6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.205516 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.205284    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0f14c1b0-8963-4e4a-8f39-1501689b67d6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.205516 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.205312    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s4c8\" (UniqueName: \"kubernetes.io/projected/0f14c1b0-8963-4e4a-8f39-1501689b67d6-kube-api-access-9s4c8\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.205516 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.205511    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0f14c1b0-8963-4e4a-8f39-1501689b67d6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.205838 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.205615    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f14c1b0-8963-4e4a-8f39-1501689b67d6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.206695 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.206672    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f14c1b0-8963-4e4a-8f39-1501689b67d6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.207682 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.207657    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f14c1b0-8963-4e4a-8f39-1501689b67d6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.207784 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.207699    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f14c1b0-8963-4e4a-8f39-1501689b67d6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.208169 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.208147    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0f14c1b0-8963-4e4a-8f39-1501689b67d6-config-out\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.208300 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.208150    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0f14c1b0-8963-4e4a-8f39-1501689b67d6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.208403 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.208347    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0f14c1b0-8963-4e4a-8f39-1501689b67d6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.208490 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.208397    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0f14c1b0-8963-4e4a-8f39-1501689b67d6-config\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.208598 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.208576    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0f14c1b0-8963-4e4a-8f39-1501689b67d6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.208864 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.208835    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0f14c1b0-8963-4e4a-8f39-1501689b67d6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.209805 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.209732    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0f14c1b0-8963-4e4a-8f39-1501689b67d6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.209805 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.209756    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0f14c1b0-8963-4e4a-8f39-1501689b67d6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.210121 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.210096    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0f14c1b0-8963-4e4a-8f39-1501689b67d6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.210222 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.210138    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0f14c1b0-8963-4e4a-8f39-1501689b67d6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.210443 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.210415    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0f14c1b0-8963-4e4a-8f39-1501689b67d6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.210527 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.210495    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0f14c1b0-8963-4e4a-8f39-1501689b67d6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.210718 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.210699    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0f14c1b0-8963-4e4a-8f39-1501689b67d6-web-config\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.211356 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.211339    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0f14c1b0-8963-4e4a-8f39-1501689b67d6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.214707 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.214688    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s4c8\" (UniqueName: \"kubernetes.io/projected/0f14c1b0-8963-4e4a-8f39-1501689b67d6-kube-api-access-9s4c8\") pod \"prometheus-k8s-0\" (UID: \"0f14c1b0-8963-4e4a-8f39-1501689b67d6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.345965 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.345910    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:07:53.478999 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.478966    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 21:07:53.482352 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:07:53.482330    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f14c1b0_8963_4e4a_8f39_1501689b67d6.slice/crio-414fa9b1ceb18f905090af57ddf514da4a803108092e527fb194efa3dc9acb15 WatchSource:0}: Error finding container 414fa9b1ceb18f905090af57ddf514da4a803108092e527fb194efa3dc9acb15: Status 404 returned error can't find the container with id 414fa9b1ceb18f905090af57ddf514da4a803108092e527fb194efa3dc9acb15
Jul 27 21:07:53.978075 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.978041    2577 generic.go:356] "Generic (PLEG): container finished" podID="0f14c1b0-8963-4e4a-8f39-1501689b67d6" containerID="371f5815b43d4e619b50943a19057cea01d512dfe9a74d7b022ad5706451441e" exitCode=0
Jul 27 21:07:53.978403 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.978092    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f14c1b0-8963-4e4a-8f39-1501689b67d6","Type":"ContainerDied","Data":"371f5815b43d4e619b50943a19057cea01d512dfe9a74d7b022ad5706451441e"}
Jul 27 21:07:53.978403 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:53.978149    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f14c1b0-8963-4e4a-8f39-1501689b67d6","Type":"ContainerStarted","Data":"414fa9b1ceb18f905090af57ddf514da4a803108092e527fb194efa3dc9acb15"}
Jul 27 21:07:54.987140 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:54.987105    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f14c1b0-8963-4e4a-8f39-1501689b67d6","Type":"ContainerStarted","Data":"81b96248a415cf1a74df35038db268ada2c86b6b37ca01c1e5bf95f04a84feef"}
Jul 27 21:07:54.987472 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:54.987194    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f14c1b0-8963-4e4a-8f39-1501689b67d6","Type":"ContainerStarted","Data":"7204afe091558f800faf20d4822456033fa359f08b034885595e4c6467e95f77"}
Jul 27 21:07:54.987472 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:54.987223    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f14c1b0-8963-4e4a-8f39-1501689b67d6","Type":"ContainerStarted","Data":"b7636d3eec7dddc0c6171a88036f4429901382d04f54cb22885869be68632a63"}
Jul 27 21:07:54.987472 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:54.987248    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f14c1b0-8963-4e4a-8f39-1501689b67d6","Type":"ContainerStarted","Data":"ece1e3ea4680df3eaad05528ceffa2d3676a65b0f38e4495579d9773f81903a8"}
Jul 27 21:07:54.987472 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:54.987271    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f14c1b0-8963-4e4a-8f39-1501689b67d6","Type":"ContainerStarted","Data":"77ccbfa2c07ed55fc4d2923f7d085436f7c2fc20332f5c6c0bfb4710d7adf156"}
Jul 27 21:07:54.987472 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:54.987292    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f14c1b0-8963-4e4a-8f39-1501689b67d6","Type":"ContainerStarted","Data":"d2b30d98008405ca7cad78c74e2e6925b37378a5e77a46b1f024c69c0cf61e7a"}
Jul 27 21:07:55.017534 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:55.017483    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=1.999060813 podStartE2EDuration="2.017468336s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.407523ms" isStatefulPod=false podCreationTimestamp="2026-07-27 21:07:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:07:55.016306613 +0000 UTC m=+198.383496479" watchObservedRunningTime="2026-07-27 21:07:55.017468336 +0000 UTC m=+198.384658201"
Jul 27 21:07:55.967974 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:55.967935    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:07:58.346976 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:07:58.346935    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:08:04.549242 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:08:04.549207    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:08:07.240182 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:08:07.240150    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:08:07.294858 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:08:07.294820    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:08:31.763642 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:08:31.763607    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:08:51.558924 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:08:51.558889    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:08:53.346725 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:08:53.346691    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:08:53.361632 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:08:53.361610    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:08:54.213482 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:08:54.213451    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:09:01.703332 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:09:01.703263    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:09:01.780485 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:09:01.780454    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:09:07.380571 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:09:07.380529    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:09:07.480960 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:09:07.480929    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:09:07.705327 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:09:07.705303    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:09:07.772803 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:09:07.772755    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:09:08.363797 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:09:08.363749    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:09:08.433017 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:09:08.432990    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:09:13.626194 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:09:13.626158    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:09:13.728035 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:09:13.728005    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:09:18.553307 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:09:18.553279    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:09:18.653280 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:09:18.653253    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:09:18.711451 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:09:18.711414    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:09:19.666534 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:09:19.666506    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:09:19.810638 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:09:19.810604    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:09:28.044047 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:09:28.044013    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:09:37.004528 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:09:37.004500    2577 kubelet.go:1648] "Image garbage collection succeeded"
Jul 27 21:11:00.461867 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:11:00.461825    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:11:01.801816 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:11:01.801786    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:11:23.624254 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:11:23.624215    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:11:23.720933 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:11:23.720892    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:11:29.404942 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:11:29.404907    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:11:29.665480 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:11:29.665396    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:11:29.894979 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:11:29.894945    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:11:30.125748 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:11:30.125715    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:11:30.260409 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:11:30.260380    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:11:30.369031 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:11:30.368998    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:11:30.465790 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:11:30.465742    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:11:30.548264 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:11:30.548219    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:11:30.665616 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:11:30.665553    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:11:30.851465 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:11:30.851389    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:11:31.052387 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:11:31.052354    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:11:31.150904 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:11:31.150815    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:11:33.229345 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:11:33.229311    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:11:33.329501 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:11:33.329474    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:11:33.809569 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:11:33.809529    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:11:33.872073 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:11:33.872043    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:12:18.190684 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:12:18.190647    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:12:47.992183 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:12:47.992141    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:12:48.091072 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:12:48.091038    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:12:48.159181 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:12:48.159142    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:12:48.261889 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:12:48.261799    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:12:48.470110 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:12:48.470069    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:12:48.843395 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:12:48.843322    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:12:48.901300 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:12:48.901260    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:13:02.768650 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:13:02.768611    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:13:02.872072 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:13:02.872035    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:13:02.929031 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:13:02.928997    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:13:03.504597 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:13:03.504562    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:13:03.606125 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:13:03.606091    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:13:13.578732 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:13:13.578677    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:13:13.634315 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:13:13.634278    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:13:13.699355 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:13:13.699318    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:14:05.406726 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:14:05.406687    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:14:05.482288 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:14:05.482239    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:14:05.546117 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:14:05.546082    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:14:05.803666 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:14:05.803629    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:14:06.017567 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:14:06.017532    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:14:06.072422 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:14:06.072338    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:14:21.360015 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:14:21.359983    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-vmmff"]
Jul 27 21:14:21.363290 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:14:21.363273    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-vmmff"
Jul 27 21:14:21.365835 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:14:21.365811    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 27 21:14:21.365965 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:14:21.365811    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 27 21:14:21.365965 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:14:21.365866    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-d7slw\""
Jul 27 21:14:21.366479 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:14:21.366466    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 27 21:14:21.371178 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:14:21.371158    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-vmmff"]
Jul 27 21:14:21.447403 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:14:21.447369    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtxsp\" (UniqueName: \"kubernetes.io/projected/cd489225-7e37-4dca-895f-3f44c79209c8-kube-api-access-wtxsp\") pod \"s3-init-vmmff\" (UID: \"cd489225-7e37-4dca-895f-3f44c79209c8\") " pod="kserve/s3-init-vmmff"
Jul 27 21:14:21.447566 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:14:21.447465    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd489225-7e37-4dca-895f-3f44c79209c8-model-cache\") pod \"s3-init-vmmff\" (UID: \"cd489225-7e37-4dca-895f-3f44c79209c8\") " pod="kserve/s3-init-vmmff"
Jul 27 21:14:21.548841 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:14:21.548800    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd489225-7e37-4dca-895f-3f44c79209c8-model-cache\") pod \"s3-init-vmmff\" (UID: \"cd489225-7e37-4dca-895f-3f44c79209c8\") " pod="kserve/s3-init-vmmff"
Jul 27 21:14:21.549081 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:14:21.548871    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtxsp\" (UniqueName: \"kubernetes.io/projected/cd489225-7e37-4dca-895f-3f44c79209c8-kube-api-access-wtxsp\") pod \"s3-init-vmmff\" (UID: \"cd489225-7e37-4dca-895f-3f44c79209c8\") " pod="kserve/s3-init-vmmff"
Jul 27 21:14:21.549227 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:14:21.549204    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd489225-7e37-4dca-895f-3f44c79209c8-model-cache\") pod \"s3-init-vmmff\" (UID: \"cd489225-7e37-4dca-895f-3f44c79209c8\") " pod="kserve/s3-init-vmmff"
Jul 27 21:14:21.557535 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:14:21.557502    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtxsp\" (UniqueName: \"kubernetes.io/projected/cd489225-7e37-4dca-895f-3f44c79209c8-kube-api-access-wtxsp\") pod \"s3-init-vmmff\" (UID: \"cd489225-7e37-4dca-895f-3f44c79209c8\") " pod="kserve/s3-init-vmmff"
Jul 27 21:14:21.672956 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:14:21.672922    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-vmmff"
Jul 27 21:14:21.802698 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:14:21.802657    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-vmmff"]
Jul 27 21:14:21.805226 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:14:21.805200    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd489225_7e37_4dca_895f_3f44c79209c8.slice/crio-879041df2ce916bb089827e7653c5bb4505d816fac205ac56e81bacc8e8da828 WatchSource:0}: Error finding container 879041df2ce916bb089827e7653c5bb4505d816fac205ac56e81bacc8e8da828: Status 404 returned error can't find the container with id 879041df2ce916bb089827e7653c5bb4505d816fac205ac56e81bacc8e8da828
Jul 27 21:14:21.807052 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:14:21.807035    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 21:14:22.360521 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:14:22.360489    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-vmmff" event={"ID":"cd489225-7e37-4dca-895f-3f44c79209c8","Type":"ContainerStarted","Data":"879041df2ce916bb089827e7653c5bb4505d816fac205ac56e81bacc8e8da828"}
Jul 27 21:14:26.380081 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:14:26.380039    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-vmmff" event={"ID":"cd489225-7e37-4dca-895f-3f44c79209c8","Type":"ContainerStarted","Data":"8ed52e58cb958c8da389aad697bf8b5c3e73e9988e6c738a7ff4e1088127f284"}
Jul 27 21:14:27.384428 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:14:27.384347    2577 generic.go:356] "Generic (PLEG): container finished" podID="cd489225-7e37-4dca-895f-3f44c79209c8" containerID="8ed52e58cb958c8da389aad697bf8b5c3e73e9988e6c738a7ff4e1088127f284" exitCode=0
Jul 27 21:14:27.384780 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:14:27.384432    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-vmmff" event={"ID":"cd489225-7e37-4dca-895f-3f44c79209c8","Type":"ContainerDied","Data":"8ed52e58cb958c8da389aad697bf8b5c3e73e9988e6c738a7ff4e1088127f284"}
Jul 27 21:14:32.406804 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:14:32.406749    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-vmmff" event={"ID":"cd489225-7e37-4dca-895f-3f44c79209c8","Type":"ContainerStarted","Data":"d7e132bab8aec1be239619b9d5d4153c0d4f0bafc039ba3c3689684024187689"}
Jul 27 21:14:32.422581 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:14:32.422535    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-vmmff" podStartSLOduration=1.7830440950000002 podStartE2EDuration="11.422519147s" totalImagesPullingTime="7.98685238s" totalInitContainerRuntime="1.652622672s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:14:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:14:32.421553919 +0000 UTC m=+595.788743780" watchObservedRunningTime="2026-07-27 21:14:32.422519147 +0000 UTC m=+595.789709013"
Jul 27 21:14:57.096312 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:14:57.096277    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:14:57.153263 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:14:57.153228    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:15:37.652371 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:15:37.652339    2577 generic.go:356] "Generic (PLEG): container finished" podID="cd489225-7e37-4dca-895f-3f44c79209c8" containerID="d7e132bab8aec1be239619b9d5d4153c0d4f0bafc039ba3c3689684024187689" exitCode=0
Jul 27 21:15:37.652826 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:15:37.652377    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-vmmff" event={"ID":"cd489225-7e37-4dca-895f-3f44c79209c8","Type":"ContainerDied","Data":"d7e132bab8aec1be239619b9d5d4153c0d4f0bafc039ba3c3689684024187689"}
Jul 27 21:15:38.787371 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:15:38.787345    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-vmmff"
Jul 27 21:15:38.857265 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:15:38.857235    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtxsp\" (UniqueName: \"kubernetes.io/projected/cd489225-7e37-4dca-895f-3f44c79209c8-kube-api-access-wtxsp\") pod \"cd489225-7e37-4dca-895f-3f44c79209c8\" (UID: \"cd489225-7e37-4dca-895f-3f44c79209c8\") "
Jul 27 21:15:38.857432 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:15:38.857329    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd489225-7e37-4dca-895f-3f44c79209c8-model-cache\") pod \"cd489225-7e37-4dca-895f-3f44c79209c8\" (UID: \"cd489225-7e37-4dca-895f-3f44c79209c8\") "
Jul 27 21:15:38.859501 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:15:38.859469    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd489225-7e37-4dca-895f-3f44c79209c8-kube-api-access-wtxsp" (OuterVolumeSpecName: "kube-api-access-wtxsp") pod "cd489225-7e37-4dca-895f-3f44c79209c8" (UID: "cd489225-7e37-4dca-895f-3f44c79209c8"). InnerVolumeSpecName "kube-api-access-wtxsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:15:38.879079 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:15:38.879043    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd489225-7e37-4dca-895f-3f44c79209c8-model-cache" (OuterVolumeSpecName: "model-cache") pod "cd489225-7e37-4dca-895f-3f44c79209c8" (UID: "cd489225-7e37-4dca-895f-3f44c79209c8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:15:38.958010 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:15:38.957977    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd489225-7e37-4dca-895f-3f44c79209c8-model-cache\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:15:38.958010 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:15:38.958006    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtxsp\" (UniqueName: \"kubernetes.io/projected/cd489225-7e37-4dca-895f-3f44c79209c8-kube-api-access-wtxsp\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:15:39.661436 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:15:39.661410    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-vmmff"
Jul 27 21:15:39.661600 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:15:39.661409    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-vmmff" event={"ID":"cd489225-7e37-4dca-895f-3f44c79209c8","Type":"ContainerDied","Data":"879041df2ce916bb089827e7653c5bb4505d816fac205ac56e81bacc8e8da828"}
Jul 27 21:15:39.661600 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:15:39.661520    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="879041df2ce916bb089827e7653c5bb4505d816fac205ac56e81bacc8e8da828"
Jul 27 21:15:40.392227 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:15:40.392189    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:15:47.429826 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:15:47.429791    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-tw4gf"]
Jul 27 21:15:47.430198 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:15:47.430061    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd489225-7e37-4dca-895f-3f44c79209c8" containerName="download-hf-model"
Jul 27 21:15:47.430198 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:15:47.430072    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd489225-7e37-4dca-895f-3f44c79209c8" containerName="download-hf-model"
Jul 27 21:15:47.430198 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:15:47.430090    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd489225-7e37-4dca-895f-3f44c79209c8" containerName="s3-init"
Jul 27 21:15:47.430198 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:15:47.430095    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd489225-7e37-4dca-895f-3f44c79209c8" containerName="s3-init"
Jul 27 21:15:47.430198 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:15:47.430143    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd489225-7e37-4dca-895f-3f44c79209c8" containerName="s3-init"
Jul 27 21:15:47.433193 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:15:47.433177    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-tw4gf"
Jul 27 21:15:47.435421 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:15:47.435396    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Jul 27 21:15:47.435557 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:15:47.435438    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-d7slw\""
Jul 27 21:15:47.436297 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:15:47.436276    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 27 21:15:47.436404 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:15:47.436281    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 27 21:15:47.440374 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:15:47.440342    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-tw4gf"]
Jul 27 21:15:47.517191 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:15:47.517161    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/672995b3-20db-4384-bc79-d21f79e6c10f-model-cache\") pod \"s3-tls-init-custom-tw4gf\" (UID: \"672995b3-20db-4384-bc79-d21f79e6c10f\") " pod="kserve/s3-tls-init-custom-tw4gf"
Jul 27 21:15:47.517364 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:15:47.517214    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvd5h\" (UniqueName: \"kubernetes.io/projected/672995b3-20db-4384-bc79-d21f79e6c10f-kube-api-access-wvd5h\") pod \"s3-tls-init-custom-tw4gf\" (UID: \"672995b3-20db-4384-bc79-d21f79e6c10f\") " pod="kserve/s3-tls-init-custom-tw4gf"
Jul 27 21:15:47.618290 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:15:47.618247    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/672995b3-20db-4384-bc79-d21f79e6c10f-model-cache\") pod \"s3-tls-init-custom-tw4gf\" (UID: \"672995b3-20db-4384-bc79-d21f79e6c10f\") " pod="kserve/s3-tls-init-custom-tw4gf"
Jul 27 21:15:47.618458 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:15:47.618316    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvd5h\" (UniqueName: \"kubernetes.io/projected/672995b3-20db-4384-bc79-d21f79e6c10f-kube-api-access-wvd5h\") pod \"s3-tls-init-custom-tw4gf\" (UID: \"672995b3-20db-4384-bc79-d21f79e6c10f\") " pod="kserve/s3-tls-init-custom-tw4gf"
Jul 27 21:15:47.618649 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:15:47.618630    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/672995b3-20db-4384-bc79-d21f79e6c10f-model-cache\") pod \"s3-tls-init-custom-tw4gf\" (UID: \"672995b3-20db-4384-bc79-d21f79e6c10f\") " pod="kserve/s3-tls-init-custom-tw4gf"
Jul 27 21:15:47.626410 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:15:47.626375    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvd5h\" (UniqueName: \"kubernetes.io/projected/672995b3-20db-4384-bc79-d21f79e6c10f-kube-api-access-wvd5h\") pod \"s3-tls-init-custom-tw4gf\" (UID: \"672995b3-20db-4384-bc79-d21f79e6c10f\") " pod="kserve/s3-tls-init-custom-tw4gf"
Jul 27 21:15:47.742366 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:15:47.742288    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-tw4gf"
Jul 27 21:15:47.871735 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:15:47.871536    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-tw4gf"]
Jul 27 21:15:47.876559 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:15:47.876517    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod672995b3_20db_4384_bc79_d21f79e6c10f.slice/crio-5cb26903adf98bea071fb2046f3cd44cdda9724383f810ee903e4f66d3068ea0 WatchSource:0}: Error finding container 5cb26903adf98bea071fb2046f3cd44cdda9724383f810ee903e4f66d3068ea0: Status 404 returned error can't find the container with id 5cb26903adf98bea071fb2046f3cd44cdda9724383f810ee903e4f66d3068ea0
Jul 27 21:15:48.696250 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:15:48.696176    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-tw4gf" event={"ID":"672995b3-20db-4384-bc79-d21f79e6c10f","Type":"ContainerStarted","Data":"44a179a778d8a21db6e22574eb90430365abbfe52cd4be3568da252d2f798f4c"}
Jul 27 21:15:48.696250 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:15:48.696240    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-tw4gf" event={"ID":"672995b3-20db-4384-bc79-d21f79e6c10f","Type":"ContainerStarted","Data":"5cb26903adf98bea071fb2046f3cd44cdda9724383f810ee903e4f66d3068ea0"}
Jul 27 21:15:49.700983 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:15:49.700944    2577 generic.go:356] "Generic (PLEG): container finished" podID="672995b3-20db-4384-bc79-d21f79e6c10f" containerID="44a179a778d8a21db6e22574eb90430365abbfe52cd4be3568da252d2f798f4c" exitCode=0
Jul 27 21:15:49.701354 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:15:49.701033    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-tw4gf" event={"ID":"672995b3-20db-4384-bc79-d21f79e6c10f","Type":"ContainerDied","Data":"44a179a778d8a21db6e22574eb90430365abbfe52cd4be3568da252d2f798f4c"}
Jul 27 21:15:50.706651 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:15:50.706611    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-tw4gf" event={"ID":"672995b3-20db-4384-bc79-d21f79e6c10f","Type":"ContainerStarted","Data":"dd8603f317d75d6ef31b3ae16d2f44ec77eba277aeb6726af4d9700103256a13"}
Jul 27 21:15:50.722672 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:15:50.722625    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-tw4gf" podStartSLOduration=2.178034978 podStartE2EDuration="3.72260775s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.544572772s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:15:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:15:50.721689694 +0000 UTC m=+674.088879564" watchObservedRunningTime="2026-07-27 21:15:50.72260775 +0000 UTC m=+674.089797616"
Jul 27 21:15:56.733846 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:15:56.733810    2577 generic.go:356] "Generic (PLEG): container finished" podID="672995b3-20db-4384-bc79-d21f79e6c10f" containerID="dd8603f317d75d6ef31b3ae16d2f44ec77eba277aeb6726af4d9700103256a13" exitCode=0
Jul 27 21:15:56.734219 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:15:56.733880    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-tw4gf" event={"ID":"672995b3-20db-4384-bc79-d21f79e6c10f","Type":"ContainerDied","Data":"dd8603f317d75d6ef31b3ae16d2f44ec77eba277aeb6726af4d9700103256a13"}
Jul 27 21:15:57.877773 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:15:57.877741    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-tw4gf"
Jul 27 21:15:57.904597 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:15:57.904571    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/672995b3-20db-4384-bc79-d21f79e6c10f-model-cache\") pod \"672995b3-20db-4384-bc79-d21f79e6c10f\" (UID: \"672995b3-20db-4384-bc79-d21f79e6c10f\") "
Jul 27 21:15:57.904736 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:15:57.904612    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wvd5h\" (UniqueName: \"kubernetes.io/projected/672995b3-20db-4384-bc79-d21f79e6c10f-kube-api-access-wvd5h\") pod \"672995b3-20db-4384-bc79-d21f79e6c10f\" (UID: \"672995b3-20db-4384-bc79-d21f79e6c10f\") "
Jul 27 21:15:57.906688 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:15:57.906657    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/672995b3-20db-4384-bc79-d21f79e6c10f-kube-api-access-wvd5h" (OuterVolumeSpecName: "kube-api-access-wvd5h") pod "672995b3-20db-4384-bc79-d21f79e6c10f" (UID: "672995b3-20db-4384-bc79-d21f79e6c10f"). InnerVolumeSpecName "kube-api-access-wvd5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:15:57.930345 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:15:57.930305    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/672995b3-20db-4384-bc79-d21f79e6c10f-model-cache" (OuterVolumeSpecName: "model-cache") pod "672995b3-20db-4384-bc79-d21f79e6c10f" (UID: "672995b3-20db-4384-bc79-d21f79e6c10f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:15:58.005458 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:15:58.005387    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvd5h\" (UniqueName: \"kubernetes.io/projected/672995b3-20db-4384-bc79-d21f79e6c10f-kube-api-access-wvd5h\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:15:58.005458 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:15:58.005415    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/672995b3-20db-4384-bc79-d21f79e6c10f-model-cache\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:15:58.742946 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:15:58.742908    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-tw4gf" event={"ID":"672995b3-20db-4384-bc79-d21f79e6c10f","Type":"ContainerDied","Data":"5cb26903adf98bea071fb2046f3cd44cdda9724383f810ee903e4f66d3068ea0"}
Jul 27 21:15:58.742946 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:15:58.742940    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-tw4gf"
Jul 27 21:15:58.742946 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:15:58.742953    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5cb26903adf98bea071fb2046f3cd44cdda9724383f810ee903e4f66d3068ea0"
Jul 27 21:15:59.336632 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:15:59.336594    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:15:59.590114 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:15:59.590027    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:16:01.751852 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:01.751823    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-vfwph"]
Jul 27 21:16:01.752225 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:01.752098    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="672995b3-20db-4384-bc79-d21f79e6c10f" containerName="s3-tls-init-custom"
Jul 27 21:16:01.752225 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:01.752109    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="672995b3-20db-4384-bc79-d21f79e6c10f" containerName="s3-tls-init-custom"
Jul 27 21:16:01.752225 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:01.752121    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="672995b3-20db-4384-bc79-d21f79e6c10f" containerName="download-hf-model"
Jul 27 21:16:01.752225 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:01.752126    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="672995b3-20db-4384-bc79-d21f79e6c10f" containerName="download-hf-model"
Jul 27 21:16:01.752225 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:01.752181    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="672995b3-20db-4384-bc79-d21f79e6c10f" containerName="s3-tls-init-custom"
Jul 27 21:16:01.755252 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:01.755234    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-vfwph"
Jul 27 21:16:01.757898 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:01.757873    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Jul 27 21:16:01.757898 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:01.757886    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 27 21:16:01.758636 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:01.758620    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-d7slw\""
Jul 27 21:16:01.758731 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:01.758636    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 27 21:16:01.763361 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:01.763339    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-vfwph"]
Jul 27 21:16:01.835174 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:01.835143    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvxg6\" (UniqueName: \"kubernetes.io/projected/4958abd9-5a06-4b04-a8d6-aae8d80013ae-kube-api-access-jvxg6\") pod \"s3-tls-init-serving-vfwph\" (UID: \"4958abd9-5a06-4b04-a8d6-aae8d80013ae\") " pod="kserve/s3-tls-init-serving-vfwph"
Jul 27 21:16:01.835174 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:01.835181    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4958abd9-5a06-4b04-a8d6-aae8d80013ae-model-cache\") pod \"s3-tls-init-serving-vfwph\" (UID: \"4958abd9-5a06-4b04-a8d6-aae8d80013ae\") " pod="kserve/s3-tls-init-serving-vfwph"
Jul 27 21:16:01.935930 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:01.935898    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvxg6\" (UniqueName: \"kubernetes.io/projected/4958abd9-5a06-4b04-a8d6-aae8d80013ae-kube-api-access-jvxg6\") pod \"s3-tls-init-serving-vfwph\" (UID: \"4958abd9-5a06-4b04-a8d6-aae8d80013ae\") " pod="kserve/s3-tls-init-serving-vfwph"
Jul 27 21:16:01.935930 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:01.935934    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4958abd9-5a06-4b04-a8d6-aae8d80013ae-model-cache\") pod \"s3-tls-init-serving-vfwph\" (UID: \"4958abd9-5a06-4b04-a8d6-aae8d80013ae\") " pod="kserve/s3-tls-init-serving-vfwph"
Jul 27 21:16:01.936310 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:01.936292    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4958abd9-5a06-4b04-a8d6-aae8d80013ae-model-cache\") pod \"s3-tls-init-serving-vfwph\" (UID: \"4958abd9-5a06-4b04-a8d6-aae8d80013ae\") " pod="kserve/s3-tls-init-serving-vfwph"
Jul 27 21:16:01.944330 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:01.944304    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvxg6\" (UniqueName: \"kubernetes.io/projected/4958abd9-5a06-4b04-a8d6-aae8d80013ae-kube-api-access-jvxg6\") pod \"s3-tls-init-serving-vfwph\" (UID: \"4958abd9-5a06-4b04-a8d6-aae8d80013ae\") " pod="kserve/s3-tls-init-serving-vfwph"
Jul 27 21:16:02.064668 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:02.064586    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-vfwph"
Jul 27 21:16:02.189644 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:02.189619    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-vfwph"]
Jul 27 21:16:02.191812 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:16:02.191786    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4958abd9_5a06_4b04_a8d6_aae8d80013ae.slice/crio-331591e0762ef58c02a0d60e6efb9ef73c54ef24353ec50cb03e11adc063c010 WatchSource:0}: Error finding container 331591e0762ef58c02a0d60e6efb9ef73c54ef24353ec50cb03e11adc063c010: Status 404 returned error can't find the container with id 331591e0762ef58c02a0d60e6efb9ef73c54ef24353ec50cb03e11adc063c010
Jul 27 21:16:02.760066 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:02.760027    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-vfwph" event={"ID":"4958abd9-5a06-4b04-a8d6-aae8d80013ae","Type":"ContainerStarted","Data":"e37c82e500fa4b309db93276246d3623fd1621fae72aa4792a7ff7828831f8ce"}
Jul 27 21:16:02.760066 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:02.760071    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-vfwph" event={"ID":"4958abd9-5a06-4b04-a8d6-aae8d80013ae","Type":"ContainerStarted","Data":"331591e0762ef58c02a0d60e6efb9ef73c54ef24353ec50cb03e11adc063c010"}
Jul 27 21:16:04.770889 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:04.770842    2577 generic.go:356] "Generic (PLEG): container finished" podID="4958abd9-5a06-4b04-a8d6-aae8d80013ae" containerID="e37c82e500fa4b309db93276246d3623fd1621fae72aa4792a7ff7828831f8ce" exitCode=0
Jul 27 21:16:04.771236 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:04.770916    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-vfwph" event={"ID":"4958abd9-5a06-4b04-a8d6-aae8d80013ae","Type":"ContainerDied","Data":"e37c82e500fa4b309db93276246d3623fd1621fae72aa4792a7ff7828831f8ce"}
Jul 27 21:16:05.775836 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:05.775801    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-vfwph" event={"ID":"4958abd9-5a06-4b04-a8d6-aae8d80013ae","Type":"ContainerStarted","Data":"54d6e2304dc462f707020abf8b9367e190a495f304eacc37ae99aa8b2dad7b02"}
Jul 27 21:16:05.792780 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:05.792716    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-vfwph" podStartSLOduration=3.045900783 podStartE2EDuration="4.79270264s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.746801857s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:16:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:16:05.790990596 +0000 UTC m=+689.158180473" watchObservedRunningTime="2026-07-27 21:16:05.79270264 +0000 UTC m=+689.159892505"
Jul 27 21:16:08.454558 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:08.454522    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:16:10.801005 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:10.800973    2577 generic.go:356] "Generic (PLEG): container finished" podID="4958abd9-5a06-4b04-a8d6-aae8d80013ae" containerID="54d6e2304dc462f707020abf8b9367e190a495f304eacc37ae99aa8b2dad7b02" exitCode=0
Jul 27 21:16:10.801375 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:10.801017    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-vfwph" event={"ID":"4958abd9-5a06-4b04-a8d6-aae8d80013ae","Type":"ContainerDied","Data":"54d6e2304dc462f707020abf8b9367e190a495f304eacc37ae99aa8b2dad7b02"}
Jul 27 21:16:11.956011 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:11.955984    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-vfwph"
Jul 27 21:16:12.025475 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:12.025441    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvxg6\" (UniqueName: \"kubernetes.io/projected/4958abd9-5a06-4b04-a8d6-aae8d80013ae-kube-api-access-jvxg6\") pod \"4958abd9-5a06-4b04-a8d6-aae8d80013ae\" (UID: \"4958abd9-5a06-4b04-a8d6-aae8d80013ae\") "
Jul 27 21:16:12.025635 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:12.025492    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4958abd9-5a06-4b04-a8d6-aae8d80013ae-model-cache\") pod \"4958abd9-5a06-4b04-a8d6-aae8d80013ae\" (UID: \"4958abd9-5a06-4b04-a8d6-aae8d80013ae\") "
Jul 27 21:16:12.029239 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:12.028426    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4958abd9-5a06-4b04-a8d6-aae8d80013ae-kube-api-access-jvxg6" (OuterVolumeSpecName: "kube-api-access-jvxg6") pod "4958abd9-5a06-4b04-a8d6-aae8d80013ae" (UID: "4958abd9-5a06-4b04-a8d6-aae8d80013ae"). InnerVolumeSpecName "kube-api-access-jvxg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:16:12.052655 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:12.052602    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4958abd9-5a06-4b04-a8d6-aae8d80013ae-model-cache" (OuterVolumeSpecName: "model-cache") pod "4958abd9-5a06-4b04-a8d6-aae8d80013ae" (UID: "4958abd9-5a06-4b04-a8d6-aae8d80013ae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:16:12.127024 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:12.126946    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4958abd9-5a06-4b04-a8d6-aae8d80013ae-model-cache\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:16:12.127024 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:12.126978    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvxg6\" (UniqueName: \"kubernetes.io/projected/4958abd9-5a06-4b04-a8d6-aae8d80013ae-kube-api-access-jvxg6\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:16:12.809063 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:12.809037    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-vfwph"
Jul 27 21:16:12.809233 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:12.809036    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-vfwph" event={"ID":"4958abd9-5a06-4b04-a8d6-aae8d80013ae","Type":"ContainerDied","Data":"331591e0762ef58c02a0d60e6efb9ef73c54ef24353ec50cb03e11adc063c010"}
Jul 27 21:16:12.809233 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:12.809152    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="331591e0762ef58c02a0d60e6efb9ef73c54ef24353ec50cb03e11adc063c010"
Jul 27 21:16:13.166405 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:13.166325    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:16:13.268161 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:13.268123    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:16:25.027910 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:25.027853    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:16:25.139556 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:25.139527    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr"]
Jul 27 21:16:25.139828 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:25.139816    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4958abd9-5a06-4b04-a8d6-aae8d80013ae" containerName="s3-tls-init-serving"
Jul 27 21:16:25.139873 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:25.139829    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4958abd9-5a06-4b04-a8d6-aae8d80013ae" containerName="s3-tls-init-serving"
Jul 27 21:16:25.139873 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:25.139843    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4958abd9-5a06-4b04-a8d6-aae8d80013ae" containerName="download-hf-model"
Jul 27 21:16:25.139873 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:25.139848    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4958abd9-5a06-4b04-a8d6-aae8d80013ae" containerName="download-hf-model"
Jul 27 21:16:25.139965 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:25.139903    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4958abd9-5a06-4b04-a8d6-aae8d80013ae" containerName="s3-tls-init-serving"
Jul 27 21:16:25.144183 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:25.144168    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr"
Jul 27 21:16:25.146565 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:25.146543    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 21:16:25.146702 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:25.146586    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 21:16:25.147431 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:25.147408    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 21:16:25.147526 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:25.147456    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 21:16:25.147526 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:25.147456    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Jul 27 21:16:25.147526 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:25.147479    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v7c66\""
Jul 27 21:16:25.147750 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:25.147722    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Jul 27 21:16:25.148466 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:25.148445    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:16:25.156481 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:25.156461    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr"]
Jul 27 21:16:25.238174 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:25.238143    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7075cca3-6cd4-4714-bfc8-3a0f65a2ddab-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr\" (UID: \"7075cca3-6cd4-4714-bfc8-3a0f65a2ddab\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr"
Jul 27 21:16:25.238336 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:25.238189    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7075cca3-6cd4-4714-bfc8-3a0f65a2ddab-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr\" (UID: \"7075cca3-6cd4-4714-bfc8-3a0f65a2ddab\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr"
Jul 27 21:16:25.238336 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:25.238220    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps8fp\" (UniqueName: \"kubernetes.io/projected/7075cca3-6cd4-4714-bfc8-3a0f65a2ddab-kube-api-access-ps8fp\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr\" (UID: \"7075cca3-6cd4-4714-bfc8-3a0f65a2ddab\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr"
Jul 27 21:16:25.238336 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:25.238287    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7075cca3-6cd4-4714-bfc8-3a0f65a2ddab-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr\" (UID: \"7075cca3-6cd4-4714-bfc8-3a0f65a2ddab\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr"
Jul 27 21:16:25.238436 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:25.238338    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7075cca3-6cd4-4714-bfc8-3a0f65a2ddab-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr\" (UID: \"7075cca3-6cd4-4714-bfc8-3a0f65a2ddab\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr"
Jul 27 21:16:25.338710 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:25.338633    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps8fp\" (UniqueName: \"kubernetes.io/projected/7075cca3-6cd4-4714-bfc8-3a0f65a2ddab-kube-api-access-ps8fp\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr\" (UID: \"7075cca3-6cd4-4714-bfc8-3a0f65a2ddab\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr"
Jul 27 21:16:25.338710 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:25.338681    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7075cca3-6cd4-4714-bfc8-3a0f65a2ddab-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr\" (UID: \"7075cca3-6cd4-4714-bfc8-3a0f65a2ddab\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr"
Jul 27 21:16:25.338902 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:25.338724    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7075cca3-6cd4-4714-bfc8-3a0f65a2ddab-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr\" (UID: \"7075cca3-6cd4-4714-bfc8-3a0f65a2ddab\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr"
Jul 27 21:16:25.338902 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:25.338743    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7075cca3-6cd4-4714-bfc8-3a0f65a2ddab-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr\" (UID: \"7075cca3-6cd4-4714-bfc8-3a0f65a2ddab\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr"
Jul 27 21:16:25.338902 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:25.338805    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7075cca3-6cd4-4714-bfc8-3a0f65a2ddab-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr\" (UID: \"7075cca3-6cd4-4714-bfc8-3a0f65a2ddab\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr"
Jul 27 21:16:25.339280 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:25.339256    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7075cca3-6cd4-4714-bfc8-3a0f65a2ddab-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr\" (UID: \"7075cca3-6cd4-4714-bfc8-3a0f65a2ddab\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr"
Jul 27 21:16:25.339446 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:25.339424    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7075cca3-6cd4-4714-bfc8-3a0f65a2ddab-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr\" (UID: \"7075cca3-6cd4-4714-bfc8-3a0f65a2ddab\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr"
Jul 27 21:16:25.339518 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:25.339433    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7075cca3-6cd4-4714-bfc8-3a0f65a2ddab-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr\" (UID: \"7075cca3-6cd4-4714-bfc8-3a0f65a2ddab\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr"
Jul 27 21:16:25.341313 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:25.341289    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7075cca3-6cd4-4714-bfc8-3a0f65a2ddab-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr\" (UID: \"7075cca3-6cd4-4714-bfc8-3a0f65a2ddab\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr"
Jul 27 21:16:25.347102 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:25.347077    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps8fp\" (UniqueName: \"kubernetes.io/projected/7075cca3-6cd4-4714-bfc8-3a0f65a2ddab-kube-api-access-ps8fp\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr\" (UID: \"7075cca3-6cd4-4714-bfc8-3a0f65a2ddab\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr"
Jul 27 21:16:25.453912 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:25.453877    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr"
Jul 27 21:16:25.586031 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:25.586005    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr"]
Jul 27 21:16:25.588437 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:16:25.588409    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7075cca3_6cd4_4714_bfc8_3a0f65a2ddab.slice/crio-102b33fec8e8469cecd2661c3056cc4b7063c904c85440a572e457f835f4f637 WatchSource:0}: Error finding container 102b33fec8e8469cecd2661c3056cc4b7063c904c85440a572e457f835f4f637: Status 404 returned error can't find the container with id 102b33fec8e8469cecd2661c3056cc4b7063c904c85440a572e457f835f4f637
Jul 27 21:16:25.865210 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:25.865116    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr" event={"ID":"7075cca3-6cd4-4714-bfc8-3a0f65a2ddab","Type":"ContainerStarted","Data":"64bc684a7159091cdc276f4f76e7227c743be2e4e9fe1cf374167d75935b980c"}
Jul 27 21:16:25.865210 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:25.865175    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr" event={"ID":"7075cca3-6cd4-4714-bfc8-3a0f65a2ddab","Type":"ContainerStarted","Data":"102b33fec8e8469cecd2661c3056cc4b7063c904c85440a572e457f835f4f637"}
Jul 27 21:16:29.882620 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:29.882583    2577 generic.go:356] "Generic (PLEG): container finished" podID="7075cca3-6cd4-4714-bfc8-3a0f65a2ddab" containerID="64bc684a7159091cdc276f4f76e7227c743be2e4e9fe1cf374167d75935b980c" exitCode=0
Jul 27 21:16:29.883017 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:29.882651    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr" event={"ID":"7075cca3-6cd4-4714-bfc8-3a0f65a2ddab","Type":"ContainerDied","Data":"64bc684a7159091cdc276f4f76e7227c743be2e4e9fe1cf374167d75935b980c"}
Jul 27 21:16:43.957668 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:43.957628    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr" event={"ID":"7075cca3-6cd4-4714-bfc8-3a0f65a2ddab","Type":"ContainerStarted","Data":"8c49fdff693fb14c56bb65fa5e25131d574ea1b3586ca9f3af730710fa95f8a3"}
Jul 27 21:16:46.979486 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:46.979450    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr" event={"ID":"7075cca3-6cd4-4714-bfc8-3a0f65a2ddab","Type":"ContainerStarted","Data":"c098841e30115324d4cecd4592c237d96d81cf190ad8fb6cb7848d1e95ceb29c"}
Jul 27 21:16:48.991315 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:48.991280    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr" event={"ID":"7075cca3-6cd4-4714-bfc8-3a0f65a2ddab","Type":"ContainerStarted","Data":"9dffa680247c2b0725c84f54332a5fef429ef61428f828815aaf73f5fb337297"}
Jul 27 21:16:48.991716 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:48.991468    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr"
Jul 27 21:16:48.991716 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:48.991581    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr"
Jul 27 21:16:48.992841 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:48.992804    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr" podUID="7075cca3-6cd4-4714-bfc8-3a0f65a2ddab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 27 21:16:49.012918 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:49.012849    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr" podStartSLOduration=1.752155461 podStartE2EDuration="24.012832855s" totalImagesPullingTime="18.871977379s" totalInitContainerRuntime="3.388700015s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:16:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:16:49.01099669 +0000 UTC m=+732.378186557" watchObservedRunningTime="2026-07-27 21:16:49.012832855 +0000 UTC m=+732.380022720"
Jul 27 21:16:49.995515 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:49.995469    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr"
Jul 27 21:16:49.995943 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:49.995639    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr" podUID="7075cca3-6cd4-4714-bfc8-3a0f65a2ddab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 27 21:16:49.996731 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:49.996705    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr" podUID="7075cca3-6cd4-4714-bfc8-3a0f65a2ddab" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:16:50.999743 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:50.999702    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr" podUID="7075cca3-6cd4-4714-bfc8-3a0f65a2ddab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 27 21:16:51.000220 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:51.000164    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr" podUID="7075cca3-6cd4-4714-bfc8-3a0f65a2ddab" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:16:51.003610 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:51.003587    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr"
Jul 27 21:16:51.963673 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:51.963631    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:16:52.007822 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:52.007700    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr" podUID="7075cca3-6cd4-4714-bfc8-3a0f65a2ddab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 27 21:16:52.008322 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:16:52.008289    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr" podUID="7075cca3-6cd4-4714-bfc8-3a0f65a2ddab" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:17:02.005912 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:17:02.005865    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr" podUID="7075cca3-6cd4-4714-bfc8-3a0f65a2ddab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 27 21:17:02.006569 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:17:02.006545    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr" podUID="7075cca3-6cd4-4714-bfc8-3a0f65a2ddab" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:17:12.006616 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:17:12.006564    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr" podUID="7075cca3-6cd4-4714-bfc8-3a0f65a2ddab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 27 21:17:12.007120 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:17:12.007068    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr" podUID="7075cca3-6cd4-4714-bfc8-3a0f65a2ddab" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:17:22.006312 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:17:22.006264    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr" podUID="7075cca3-6cd4-4714-bfc8-3a0f65a2ddab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 27 21:17:22.006792 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:17:22.006679    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr" podUID="7075cca3-6cd4-4714-bfc8-3a0f65a2ddab" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:17:32.006368 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:17:32.006324    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr" podUID="7075cca3-6cd4-4714-bfc8-3a0f65a2ddab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 27 21:17:32.006894 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:17:32.006853    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr" podUID="7075cca3-6cd4-4714-bfc8-3a0f65a2ddab" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:17:42.006752 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:17:42.006697    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr" podUID="7075cca3-6cd4-4714-bfc8-3a0f65a2ddab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 27 21:17:42.007197 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:17:42.007173    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr" podUID="7075cca3-6cd4-4714-bfc8-3a0f65a2ddab" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:17:52.006688 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:17:52.006635    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr" podUID="7075cca3-6cd4-4714-bfc8-3a0f65a2ddab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 27 21:17:52.007203 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:17:52.007120    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr" podUID="7075cca3-6cd4-4714-bfc8-3a0f65a2ddab" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:18:02.006709 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:02.006621    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr"
Jul 27 21:18:02.007161 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:02.007150    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr"
Jul 27 21:18:04.006093 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:04.006059    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:18:10.187438 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:10.187399    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:18:10.235590 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:10.235559    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr"]
Jul 27 21:18:10.235831 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:10.235808    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr" podUID="7075cca3-6cd4-4714-bfc8-3a0f65a2ddab" containerName="kserve-container" containerID="cri-o://8c49fdff693fb14c56bb65fa5e25131d574ea1b3586ca9f3af730710fa95f8a3" gracePeriod=30
Jul 27 21:18:10.235957 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:10.235901    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr" podUID="7075cca3-6cd4-4714-bfc8-3a0f65a2ddab" containerName="agent" containerID="cri-o://9dffa680247c2b0725c84f54332a5fef429ef61428f828815aaf73f5fb337297" gracePeriod=30
Jul 27 21:18:10.236090 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:10.235901    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr" podUID="7075cca3-6cd4-4714-bfc8-3a0f65a2ddab" containerName="kube-rbac-proxy" containerID="cri-o://c098841e30115324d4cecd4592c237d96d81cf190ad8fb6cb7848d1e95ceb29c" gracePeriod=30
Jul 27 21:18:10.254243 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:10.254221    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg"]
Jul 27 21:18:10.256794 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:10.256778    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg"
Jul 27 21:18:10.259707 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:10.259680    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Jul 27 21:18:10.260152 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:10.260093    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Jul 27 21:18:10.261203 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:10.261115    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:18:10.273545 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:10.273526    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg"]
Jul 27 21:18:10.368809 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:10.368779    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:18:10.434110 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:10.434076    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z7jc\" (UniqueName: \"kubernetes.io/projected/85e94499-a400-45f4-b021-3bda235a5580-kube-api-access-8z7jc\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg\" (UID: \"85e94499-a400-45f4-b021-3bda235a5580\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg"
Jul 27 21:18:10.434253 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:10.434135    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85e94499-a400-45f4-b021-3bda235a5580-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg\" (UID: \"85e94499-a400-45f4-b021-3bda235a5580\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg"
Jul 27 21:18:10.434253 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:10.434175    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85e94499-a400-45f4-b021-3bda235a5580-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg\" (UID: \"85e94499-a400-45f4-b021-3bda235a5580\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg"
Jul 27 21:18:10.434379 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:10.434264    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85e94499-a400-45f4-b021-3bda235a5580-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg\" (UID: \"85e94499-a400-45f4-b021-3bda235a5580\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg"
Jul 27 21:18:10.434379 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:10.434291    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85e94499-a400-45f4-b021-3bda235a5580-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg\" (UID: \"85e94499-a400-45f4-b021-3bda235a5580\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg"
Jul 27 21:18:10.535805 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:10.535684    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85e94499-a400-45f4-b021-3bda235a5580-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg\" (UID: \"85e94499-a400-45f4-b021-3bda235a5580\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg"
Jul 27 21:18:10.535805 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:10.535720    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85e94499-a400-45f4-b021-3bda235a5580-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg\" (UID: \"85e94499-a400-45f4-b021-3bda235a5580\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg"
Jul 27 21:18:10.535805 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:10.535786    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z7jc\" (UniqueName: \"kubernetes.io/projected/85e94499-a400-45f4-b021-3bda235a5580-kube-api-access-8z7jc\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg\" (UID: \"85e94499-a400-45f4-b021-3bda235a5580\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg"
Jul 27 21:18:10.536070 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:10.535827    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85e94499-a400-45f4-b021-3bda235a5580-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg\" (UID: \"85e94499-a400-45f4-b021-3bda235a5580\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg"
Jul 27 21:18:10.536070 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:10.535870    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85e94499-a400-45f4-b021-3bda235a5580-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg\" (UID: \"85e94499-a400-45f4-b021-3bda235a5580\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg"
Jul 27 21:18:10.536316 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:10.536282    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85e94499-a400-45f4-b021-3bda235a5580-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg\" (UID: \"85e94499-a400-45f4-b021-3bda235a5580\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg"
Jul 27 21:18:10.536584 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:10.536565    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85e94499-a400-45f4-b021-3bda235a5580-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg\" (UID: \"85e94499-a400-45f4-b021-3bda235a5580\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg"
Jul 27 21:18:10.536663 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:10.536583    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85e94499-a400-45f4-b021-3bda235a5580-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg\" (UID: \"85e94499-a400-45f4-b021-3bda235a5580\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg"
Jul 27 21:18:10.538318 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:10.538296    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85e94499-a400-45f4-b021-3bda235a5580-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg\" (UID: \"85e94499-a400-45f4-b021-3bda235a5580\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg"
Jul 27 21:18:10.544296 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:10.544276    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z7jc\" (UniqueName: \"kubernetes.io/projected/85e94499-a400-45f4-b021-3bda235a5580-kube-api-access-8z7jc\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg\" (UID: \"85e94499-a400-45f4-b021-3bda235a5580\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg"
Jul 27 21:18:10.567249 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:10.567228    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg"
Jul 27 21:18:10.699153 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:10.699127    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg"]
Jul 27 21:18:10.701909 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:18:10.701881    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85e94499_a400_45f4_b021_3bda235a5580.slice/crio-34b71dc22290905756ea8d47ad9d9762af6637325a7711a08b53aa53b30edde4 WatchSource:0}: Error finding container 34b71dc22290905756ea8d47ad9d9762af6637325a7711a08b53aa53b30edde4: Status 404 returned error can't find the container with id 34b71dc22290905756ea8d47ad9d9762af6637325a7711a08b53aa53b30edde4
Jul 27 21:18:10.999978 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:10.999938    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr" podUID="7075cca3-6cd4-4714-bfc8-3a0f65a2ddab" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Jul 27 21:18:11.321084 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:11.320999    2577 generic.go:356] "Generic (PLEG): container finished" podID="7075cca3-6cd4-4714-bfc8-3a0f65a2ddab" containerID="c098841e30115324d4cecd4592c237d96d81cf190ad8fb6cb7848d1e95ceb29c" exitCode=2
Jul 27 21:18:11.321476 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:11.321074    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr" event={"ID":"7075cca3-6cd4-4714-bfc8-3a0f65a2ddab","Type":"ContainerDied","Data":"c098841e30115324d4cecd4592c237d96d81cf190ad8fb6cb7848d1e95ceb29c"}
Jul 27 21:18:11.322496 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:11.322478    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg" event={"ID":"85e94499-a400-45f4-b021-3bda235a5580","Type":"ContainerStarted","Data":"e2737d686650d91bb249705968a6c90c0aa8a94f122dc1dd10d6247a40107f51"}
Jul 27 21:18:11.322573 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:11.322529    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg" event={"ID":"85e94499-a400-45f4-b021-3bda235a5580","Type":"ContainerStarted","Data":"34b71dc22290905756ea8d47ad9d9762af6637325a7711a08b53aa53b30edde4"}
Jul 27 21:18:12.006782 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:12.006717    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr" podUID="7075cca3-6cd4-4714-bfc8-3a0f65a2ddab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 27 21:18:12.007701 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:12.007672    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr" podUID="7075cca3-6cd4-4714-bfc8-3a0f65a2ddab" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:18:14.336264 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:14.336229    2577 generic.go:356] "Generic (PLEG): container finished" podID="85e94499-a400-45f4-b021-3bda235a5580" containerID="e2737d686650d91bb249705968a6c90c0aa8a94f122dc1dd10d6247a40107f51" exitCode=0
Jul 27 21:18:14.336642 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:14.336269    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg" event={"ID":"85e94499-a400-45f4-b021-3bda235a5580","Type":"ContainerDied","Data":"e2737d686650d91bb249705968a6c90c0aa8a94f122dc1dd10d6247a40107f51"}
Jul 27 21:18:15.343257 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:15.343221    2577 generic.go:356] "Generic (PLEG): container finished" podID="7075cca3-6cd4-4714-bfc8-3a0f65a2ddab" containerID="8c49fdff693fb14c56bb65fa5e25131d574ea1b3586ca9f3af730710fa95f8a3" exitCode=0
Jul 27 21:18:15.343675 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:15.343289    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr" event={"ID":"7075cca3-6cd4-4714-bfc8-3a0f65a2ddab","Type":"ContainerDied","Data":"8c49fdff693fb14c56bb65fa5e25131d574ea1b3586ca9f3af730710fa95f8a3"}
Jul 27 21:18:15.345643 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:15.345611    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg" event={"ID":"85e94499-a400-45f4-b021-3bda235a5580","Type":"ContainerStarted","Data":"969e974d8109e2fae4facb72de9ec7312b0172442519bc6490db2f9ec3925b6c"}
Jul 27 21:18:15.345756 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:15.345674    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg" event={"ID":"85e94499-a400-45f4-b021-3bda235a5580","Type":"ContainerStarted","Data":"5b437bca27827ddc9dedd56c5c1b12e7da9c34373eefe29f7e28f6720c122048"}
Jul 27 21:18:15.345756 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:15.345700    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg" event={"ID":"85e94499-a400-45f4-b021-3bda235a5580","Type":"ContainerStarted","Data":"ce2dd91139751d307848e8e26489ce94b7b638588820121e758465582cc41665"}
Jul 27 21:18:15.346041 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:15.346016    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg"
Jul 27 21:18:15.346150 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:15.346084    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg"
Jul 27 21:18:15.346150 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:15.346121    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg"
Jul 27 21:18:15.347650 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:15.347616    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg" podUID="85e94499-a400-45f4-b021-3bda235a5580" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:5000: connect: connection refused"
Jul 27 21:18:15.348308 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:15.348286    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg" podUID="85e94499-a400-45f4-b021-3bda235a5580" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:18:15.368244 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:15.368202    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg" podStartSLOduration=1.875672766 podStartE2EDuration="5.368190007s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.492517241s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:18:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:18:15.366374609 +0000 UTC m=+818.733564471" watchObservedRunningTime="2026-07-27 21:18:15.368190007 +0000 UTC m=+818.735379871"
Jul 27 21:18:16.000631 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:16.000583    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr" podUID="7075cca3-6cd4-4714-bfc8-3a0f65a2ddab" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Jul 27 21:18:16.350191 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:16.350088    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg" podUID="85e94499-a400-45f4-b021-3bda235a5580" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:5000: connect: connection refused"
Jul 27 21:18:16.350605 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:16.350582    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg" podUID="85e94499-a400-45f4-b021-3bda235a5580" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:18:21.000209 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:21.000169    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr" podUID="7075cca3-6cd4-4714-bfc8-3a0f65a2ddab" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Jul 27 21:18:21.000595 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:21.000289    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr"
Jul 27 21:18:21.354196 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:21.354114    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg"
Jul 27 21:18:21.354667 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:21.354622    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg" podUID="85e94499-a400-45f4-b021-3bda235a5580" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:5000: connect: connection refused"
Jul 27 21:18:21.355137 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:21.355114    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg" podUID="85e94499-a400-45f4-b021-3bda235a5580" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:18:22.006121 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:22.006078    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr" podUID="7075cca3-6cd4-4714-bfc8-3a0f65a2ddab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 27 21:18:22.007776 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:22.007737    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr" podUID="7075cca3-6cd4-4714-bfc8-3a0f65a2ddab" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:18:26.000839 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:26.000793    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr" podUID="7075cca3-6cd4-4714-bfc8-3a0f65a2ddab" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Jul 27 21:18:31.000486 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:31.000445    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr" podUID="7075cca3-6cd4-4714-bfc8-3a0f65a2ddab" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Jul 27 21:18:31.354611 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:31.354505    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg" podUID="85e94499-a400-45f4-b021-3bda235a5580" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:5000: connect: connection refused"
Jul 27 21:18:31.354959 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:31.354935    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg" podUID="85e94499-a400-45f4-b021-3bda235a5580" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:18:32.006704 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:32.006650    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr" podUID="7075cca3-6cd4-4714-bfc8-3a0f65a2ddab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 27 21:18:32.007107 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:32.006865    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr"
Jul 27 21:18:32.007693 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:32.007659    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr" podUID="7075cca3-6cd4-4714-bfc8-3a0f65a2ddab" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:18:32.007811 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:32.007802    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr"
Jul 27 21:18:36.000785 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:36.000711    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr" podUID="7075cca3-6cd4-4714-bfc8-3a0f65a2ddab" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Jul 27 21:18:40.376710 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:40.376684    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr"
Jul 27 21:18:40.451595 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:40.451558    2577 generic.go:356] "Generic (PLEG): container finished" podID="7075cca3-6cd4-4714-bfc8-3a0f65a2ddab" containerID="9dffa680247c2b0725c84f54332a5fef429ef61428f828815aaf73f5fb337297" exitCode=0
Jul 27 21:18:40.451782 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:40.451625    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr"
Jul 27 21:18:40.451782 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:40.451630    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr" event={"ID":"7075cca3-6cd4-4714-bfc8-3a0f65a2ddab","Type":"ContainerDied","Data":"9dffa680247c2b0725c84f54332a5fef429ef61428f828815aaf73f5fb337297"}
Jul 27 21:18:40.451782 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:40.451696    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr" event={"ID":"7075cca3-6cd4-4714-bfc8-3a0f65a2ddab","Type":"ContainerDied","Data":"102b33fec8e8469cecd2661c3056cc4b7063c904c85440a572e457f835f4f637"}
Jul 27 21:18:40.451782 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:40.451728    2577 scope.go:117] "RemoveContainer" containerID="9dffa680247c2b0725c84f54332a5fef429ef61428f828815aaf73f5fb337297"
Jul 27 21:18:40.459641 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:40.459623    2577 scope.go:117] "RemoveContainer" containerID="c098841e30115324d4cecd4592c237d96d81cf190ad8fb6cb7848d1e95ceb29c"
Jul 27 21:18:40.466645 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:40.466629    2577 scope.go:117] "RemoveContainer" containerID="8c49fdff693fb14c56bb65fa5e25131d574ea1b3586ca9f3af730710fa95f8a3"
Jul 27 21:18:40.473600 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:40.473584    2577 scope.go:117] "RemoveContainer" containerID="64bc684a7159091cdc276f4f76e7227c743be2e4e9fe1cf374167d75935b980c"
Jul 27 21:18:40.476714 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:40.476695    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7075cca3-6cd4-4714-bfc8-3a0f65a2ddab-kube-rbac-proxy-sar-config\") pod \"7075cca3-6cd4-4714-bfc8-3a0f65a2ddab\" (UID: \"7075cca3-6cd4-4714-bfc8-3a0f65a2ddab\") "
Jul 27 21:18:40.476820 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:40.476731    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7075cca3-6cd4-4714-bfc8-3a0f65a2ddab-cabundle-cert\") pod \"7075cca3-6cd4-4714-bfc8-3a0f65a2ddab\" (UID: \"7075cca3-6cd4-4714-bfc8-3a0f65a2ddab\") "
Jul 27 21:18:40.476820 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:40.476798    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ps8fp\" (UniqueName: \"kubernetes.io/projected/7075cca3-6cd4-4714-bfc8-3a0f65a2ddab-kube-api-access-ps8fp\") pod \"7075cca3-6cd4-4714-bfc8-3a0f65a2ddab\" (UID: \"7075cca3-6cd4-4714-bfc8-3a0f65a2ddab\") "
Jul 27 21:18:40.476933 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:40.476875    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7075cca3-6cd4-4714-bfc8-3a0f65a2ddab-proxy-tls\") pod \"7075cca3-6cd4-4714-bfc8-3a0f65a2ddab\" (UID: \"7075cca3-6cd4-4714-bfc8-3a0f65a2ddab\") "
Jul 27 21:18:40.476988 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:40.476932    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7075cca3-6cd4-4714-bfc8-3a0f65a2ddab-kserve-provision-location\") pod \"7075cca3-6cd4-4714-bfc8-3a0f65a2ddab\" (UID: \"7075cca3-6cd4-4714-bfc8-3a0f65a2ddab\") "
Jul 27 21:18:40.477110 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:40.477047    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7075cca3-6cd4-4714-bfc8-3a0f65a2ddab-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7075cca3-6cd4-4714-bfc8-3a0f65a2ddab" (UID: "7075cca3-6cd4-4714-bfc8-3a0f65a2ddab"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:18:40.477110 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:40.477080    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7075cca3-6cd4-4714-bfc8-3a0f65a2ddab-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7075cca3-6cd4-4714-bfc8-3a0f65a2ddab" (UID: "7075cca3-6cd4-4714-bfc8-3a0f65a2ddab"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:18:40.477387 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:40.477183    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7075cca3-6cd4-4714-bfc8-3a0f65a2ddab-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:18:40.477387 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:40.477302    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7075cca3-6cd4-4714-bfc8-3a0f65a2ddab-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:18:40.477387 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:40.477329    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7075cca3-6cd4-4714-bfc8-3a0f65a2ddab-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7075cca3-6cd4-4714-bfc8-3a0f65a2ddab" (UID: "7075cca3-6cd4-4714-bfc8-3a0f65a2ddab"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:18:40.479115 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:40.479092    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7075cca3-6cd4-4714-bfc8-3a0f65a2ddab-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7075cca3-6cd4-4714-bfc8-3a0f65a2ddab" (UID: "7075cca3-6cd4-4714-bfc8-3a0f65a2ddab"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:18:40.479227 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:40.479213    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7075cca3-6cd4-4714-bfc8-3a0f65a2ddab-kube-api-access-ps8fp" (OuterVolumeSpecName: "kube-api-access-ps8fp") pod "7075cca3-6cd4-4714-bfc8-3a0f65a2ddab" (UID: "7075cca3-6cd4-4714-bfc8-3a0f65a2ddab"). InnerVolumeSpecName "kube-api-access-ps8fp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:18:40.482505 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:40.482485    2577 scope.go:117] "RemoveContainer" containerID="9dffa680247c2b0725c84f54332a5fef429ef61428f828815aaf73f5fb337297"
Jul 27 21:18:40.482794 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:18:40.482774    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dffa680247c2b0725c84f54332a5fef429ef61428f828815aaf73f5fb337297\": container with ID starting with 9dffa680247c2b0725c84f54332a5fef429ef61428f828815aaf73f5fb337297 not found: ID does not exist" containerID="9dffa680247c2b0725c84f54332a5fef429ef61428f828815aaf73f5fb337297"
Jul 27 21:18:40.482878 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:40.482806    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dffa680247c2b0725c84f54332a5fef429ef61428f828815aaf73f5fb337297"} err="failed to get container status \"9dffa680247c2b0725c84f54332a5fef429ef61428f828815aaf73f5fb337297\": rpc error: code = NotFound desc = could not find container \"9dffa680247c2b0725c84f54332a5fef429ef61428f828815aaf73f5fb337297\": container with ID starting with 9dffa680247c2b0725c84f54332a5fef429ef61428f828815aaf73f5fb337297 not found: ID does not exist"
Jul 27 21:18:40.482878 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:40.482832    2577 scope.go:117] "RemoveContainer" containerID="c098841e30115324d4cecd4592c237d96d81cf190ad8fb6cb7848d1e95ceb29c"
Jul 27 21:18:40.483071 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:18:40.483050    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c098841e30115324d4cecd4592c237d96d81cf190ad8fb6cb7848d1e95ceb29c\": container with ID starting with c098841e30115324d4cecd4592c237d96d81cf190ad8fb6cb7848d1e95ceb29c not found: ID does not exist" containerID="c098841e30115324d4cecd4592c237d96d81cf190ad8fb6cb7848d1e95ceb29c"
Jul 27 21:18:40.483116 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:40.483079    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c098841e30115324d4cecd4592c237d96d81cf190ad8fb6cb7848d1e95ceb29c"} err="failed to get container status \"c098841e30115324d4cecd4592c237d96d81cf190ad8fb6cb7848d1e95ceb29c\": rpc error: code = NotFound desc = could not find container \"c098841e30115324d4cecd4592c237d96d81cf190ad8fb6cb7848d1e95ceb29c\": container with ID starting with c098841e30115324d4cecd4592c237d96d81cf190ad8fb6cb7848d1e95ceb29c not found: ID does not exist"
Jul 27 21:18:40.483116 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:40.483096    2577 scope.go:117] "RemoveContainer" containerID="8c49fdff693fb14c56bb65fa5e25131d574ea1b3586ca9f3af730710fa95f8a3"
Jul 27 21:18:40.483329 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:18:40.483313    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c49fdff693fb14c56bb65fa5e25131d574ea1b3586ca9f3af730710fa95f8a3\": container with ID starting with 8c49fdff693fb14c56bb65fa5e25131d574ea1b3586ca9f3af730710fa95f8a3 not found: ID does not exist" containerID="8c49fdff693fb14c56bb65fa5e25131d574ea1b3586ca9f3af730710fa95f8a3"
Jul 27 21:18:40.483394 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:40.483334    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c49fdff693fb14c56bb65fa5e25131d574ea1b3586ca9f3af730710fa95f8a3"} err="failed to get container status \"8c49fdff693fb14c56bb65fa5e25131d574ea1b3586ca9f3af730710fa95f8a3\": rpc error: code = NotFound desc = could not find container \"8c49fdff693fb14c56bb65fa5e25131d574ea1b3586ca9f3af730710fa95f8a3\": container with ID starting with 8c49fdff693fb14c56bb65fa5e25131d574ea1b3586ca9f3af730710fa95f8a3 not found: ID does not exist"
Jul 27 21:18:40.483394 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:40.483353    2577 scope.go:117] "RemoveContainer" containerID="64bc684a7159091cdc276f4f76e7227c743be2e4e9fe1cf374167d75935b980c"
Jul 27 21:18:40.483603 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:18:40.483582    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64bc684a7159091cdc276f4f76e7227c743be2e4e9fe1cf374167d75935b980c\": container with ID starting with 64bc684a7159091cdc276f4f76e7227c743be2e4e9fe1cf374167d75935b980c not found: ID does not exist" containerID="64bc684a7159091cdc276f4f76e7227c743be2e4e9fe1cf374167d75935b980c"
Jul 27 21:18:40.483694 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:40.483605    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64bc684a7159091cdc276f4f76e7227c743be2e4e9fe1cf374167d75935b980c"} err="failed to get container status \"64bc684a7159091cdc276f4f76e7227c743be2e4e9fe1cf374167d75935b980c\": rpc error: code = NotFound desc = could not find container \"64bc684a7159091cdc276f4f76e7227c743be2e4e9fe1cf374167d75935b980c\": container with ID starting with 64bc684a7159091cdc276f4f76e7227c743be2e4e9fe1cf374167d75935b980c not found: ID does not exist"
Jul 27 21:18:40.578459 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:40.578378    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7075cca3-6cd4-4714-bfc8-3a0f65a2ddab-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:18:40.578459 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:40.578406    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7075cca3-6cd4-4714-bfc8-3a0f65a2ddab-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:18:40.578459 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:40.578415    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps8fp\" (UniqueName: \"kubernetes.io/projected/7075cca3-6cd4-4714-bfc8-3a0f65a2ddab-kube-api-access-ps8fp\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:18:40.773889 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:40.773862    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr"]
Jul 27 21:18:40.778187 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:40.778161    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-drtqr"]
Jul 27 21:18:41.135634 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:41.135601    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7075cca3-6cd4-4714-bfc8-3a0f65a2ddab" path="/var/lib/kubelet/pods/7075cca3-6cd4-4714-bfc8-3a0f65a2ddab/volumes"
Jul 27 21:18:41.354842 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:41.354802    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg" podUID="85e94499-a400-45f4-b021-3bda235a5580" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:5000: connect: connection refused"
Jul 27 21:18:41.355277 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:41.355255    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg" podUID="85e94499-a400-45f4-b021-3bda235a5580" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:18:51.354902 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:51.354856    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg" podUID="85e94499-a400-45f4-b021-3bda235a5580" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:5000: connect: connection refused"
Jul 27 21:18:51.355320 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:18:51.355296    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg" podUID="85e94499-a400-45f4-b021-3bda235a5580" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:19:01.355000 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:01.354950    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg" podUID="85e94499-a400-45f4-b021-3bda235a5580" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:5000: connect: connection refused"
Jul 27 21:19:01.355550 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:01.355334    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg" podUID="85e94499-a400-45f4-b021-3bda235a5580" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:19:11.354928 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:11.354886    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg" podUID="85e94499-a400-45f4-b021-3bda235a5580" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:5000: connect: connection refused"
Jul 27 21:19:11.355427 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:11.355402    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg" podUID="85e94499-a400-45f4-b021-3bda235a5580" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:19:20.695618 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:20.695585    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:19:21.355432 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:21.355392    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg" podUID="85e94499-a400-45f4-b021-3bda235a5580" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:5000: connect: connection refused"
Jul 27 21:19:21.355888 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:21.355861    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg" podUID="85e94499-a400-45f4-b021-3bda235a5580" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:19:31.355474 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:31.355396    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg"
Jul 27 21:19:31.355987 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:31.355576    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg"
Jul 27 21:19:45.368876 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:45.368842    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:19:45.410924 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:45.410890    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-85khn"]
Jul 27 21:19:45.411278 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:45.411259    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7075cca3-6cd4-4714-bfc8-3a0f65a2ddab" containerName="storage-initializer"
Jul 27 21:19:45.411359 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:45.411279    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7075cca3-6cd4-4714-bfc8-3a0f65a2ddab" containerName="storage-initializer"
Jul 27 21:19:45.411359 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:45.411297    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7075cca3-6cd4-4714-bfc8-3a0f65a2ddab" containerName="kube-rbac-proxy"
Jul 27 21:19:45.411359 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:45.411305    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7075cca3-6cd4-4714-bfc8-3a0f65a2ddab" containerName="kube-rbac-proxy"
Jul 27 21:19:45.411359 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:45.411324    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7075cca3-6cd4-4714-bfc8-3a0f65a2ddab" containerName="kserve-container"
Jul 27 21:19:45.411359 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:45.411333    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7075cca3-6cd4-4714-bfc8-3a0f65a2ddab" containerName="kserve-container"
Jul 27 21:19:45.411359 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:45.411351    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7075cca3-6cd4-4714-bfc8-3a0f65a2ddab" containerName="agent"
Jul 27 21:19:45.411359 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:45.411360    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7075cca3-6cd4-4714-bfc8-3a0f65a2ddab" containerName="agent"
Jul 27 21:19:45.411853 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:45.411432    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7075cca3-6cd4-4714-bfc8-3a0f65a2ddab" containerName="agent"
Jul 27 21:19:45.411853 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:45.411443    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7075cca3-6cd4-4714-bfc8-3a0f65a2ddab" containerName="kserve-container"
Jul 27 21:19:45.411853 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:45.411457    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7075cca3-6cd4-4714-bfc8-3a0f65a2ddab" containerName="kube-rbac-proxy"
Jul 27 21:19:45.414722 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:45.414702    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-85khn"
Jul 27 21:19:45.417936 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:45.417907    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Jul 27 21:19:45.418051 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:45.417917    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Jul 27 21:19:45.419423 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:45.419402    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg"]
Jul 27 21:19:45.419690 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:45.419665    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg" podUID="85e94499-a400-45f4-b021-3bda235a5580" containerName="kserve-container" containerID="cri-o://ce2dd91139751d307848e8e26489ce94b7b638588820121e758465582cc41665" gracePeriod=30
Jul 27 21:19:45.419787 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:45.419691    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg" podUID="85e94499-a400-45f4-b021-3bda235a5580" containerName="agent" containerID="cri-o://969e974d8109e2fae4facb72de9ec7312b0172442519bc6490db2f9ec3925b6c" gracePeriod=30
Jul 27 21:19:45.419787 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:45.419700    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg" podUID="85e94499-a400-45f4-b021-3bda235a5580" containerName="kube-rbac-proxy" containerID="cri-o://5b437bca27827ddc9dedd56c5c1b12e7da9c34373eefe29f7e28f6720c122048" gracePeriod=30
Jul 27 21:19:45.425548 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:45.425510    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-85khn"]
Jul 27 21:19:45.442336 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:45.442310    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:19:45.500833 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:45.500799    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2lvp\" (UniqueName: \"kubernetes.io/projected/b6343eda-b4e9-4058-a547-318b2ebf6d90-kube-api-access-k2lvp\") pod \"message-dumper-predictor-6575bcfdfd-85khn\" (UID: \"b6343eda-b4e9-4058-a547-318b2ebf6d90\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-85khn"
Jul 27 21:19:45.500984 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:45.500856    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6343eda-b4e9-4058-a547-318b2ebf6d90-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-85khn\" (UID: \"b6343eda-b4e9-4058-a547-318b2ebf6d90\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-85khn"
Jul 27 21:19:45.500984 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:45.500937    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6343eda-b4e9-4058-a547-318b2ebf6d90-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-85khn\" (UID: \"b6343eda-b4e9-4058-a547-318b2ebf6d90\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-85khn"
Jul 27 21:19:45.524243 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:45.524216    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:19:45.602358 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:45.602322    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2lvp\" (UniqueName: \"kubernetes.io/projected/b6343eda-b4e9-4058-a547-318b2ebf6d90-kube-api-access-k2lvp\") pod \"message-dumper-predictor-6575bcfdfd-85khn\" (UID: \"b6343eda-b4e9-4058-a547-318b2ebf6d90\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-85khn"
Jul 27 21:19:45.602529 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:45.602382    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6343eda-b4e9-4058-a547-318b2ebf6d90-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-85khn\" (UID: \"b6343eda-b4e9-4058-a547-318b2ebf6d90\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-85khn"
Jul 27 21:19:45.602529 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:45.602451    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6343eda-b4e9-4058-a547-318b2ebf6d90-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-85khn\" (UID: \"b6343eda-b4e9-4058-a547-318b2ebf6d90\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-85khn"
Jul 27 21:19:45.603008 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:45.602984    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6343eda-b4e9-4058-a547-318b2ebf6d90-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-85khn\" (UID: \"b6343eda-b4e9-4058-a547-318b2ebf6d90\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-85khn"
Jul 27 21:19:45.604756 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:45.604736    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6343eda-b4e9-4058-a547-318b2ebf6d90-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-85khn\" (UID: \"b6343eda-b4e9-4058-a547-318b2ebf6d90\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-85khn"
Jul 27 21:19:45.610337 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:45.610318    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2lvp\" (UniqueName: \"kubernetes.io/projected/b6343eda-b4e9-4058-a547-318b2ebf6d90-kube-api-access-k2lvp\") pod \"message-dumper-predictor-6575bcfdfd-85khn\" (UID: \"b6343eda-b4e9-4058-a547-318b2ebf6d90\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-85khn"
Jul 27 21:19:45.721441 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:45.721403    2577 generic.go:356] "Generic (PLEG): container finished" podID="85e94499-a400-45f4-b021-3bda235a5580" containerID="5b437bca27827ddc9dedd56c5c1b12e7da9c34373eefe29f7e28f6720c122048" exitCode=2
Jul 27 21:19:45.721632 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:45.721470    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg" event={"ID":"85e94499-a400-45f4-b021-3bda235a5580","Type":"ContainerDied","Data":"5b437bca27827ddc9dedd56c5c1b12e7da9c34373eefe29f7e28f6720c122048"}
Jul 27 21:19:45.728593 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:45.728569    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-85khn"
Jul 27 21:19:45.854440 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:45.854386    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-85khn"]
Jul 27 21:19:45.856655 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:19:45.856614    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6343eda_b4e9_4058_a547_318b2ebf6d90.slice/crio-2918ad8892f4f7442fa1f07a1b1c5c99435d30c5d2a0e77d6f272aaa6952d967 WatchSource:0}: Error finding container 2918ad8892f4f7442fa1f07a1b1c5c99435d30c5d2a0e77d6f272aaa6952d967: Status 404 returned error can't find the container with id 2918ad8892f4f7442fa1f07a1b1c5c99435d30c5d2a0e77d6f272aaa6952d967
Jul 27 21:19:45.858552 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:45.858538    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 21:19:46.350723 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:46.350680    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg" podUID="85e94499-a400-45f4-b021-3bda235a5580" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Jul 27 21:19:46.726421 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:46.726389    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-85khn" event={"ID":"b6343eda-b4e9-4058-a547-318b2ebf6d90","Type":"ContainerStarted","Data":"2918ad8892f4f7442fa1f07a1b1c5c99435d30c5d2a0e77d6f272aaa6952d967"}
Jul 27 21:19:47.732437 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:47.732398    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-85khn" event={"ID":"b6343eda-b4e9-4058-a547-318b2ebf6d90","Type":"ContainerStarted","Data":"435c630d67191ebd532540d9a44705c35fa27d2c545a5175eda3f77cdfa21def"}
Jul 27 21:19:47.732838 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:47.732458    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-85khn" event={"ID":"b6343eda-b4e9-4058-a547-318b2ebf6d90","Type":"ContainerStarted","Data":"06d89aa1a53684a54e43e05e4f4a46ed2adfab2748f4b258d11c4df3ea790339"}
Jul 27 21:19:47.732838 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:47.732485    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-85khn"
Jul 27 21:19:47.749911 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:47.749864    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-85khn" podStartSLOduration=1.461932786 podStartE2EDuration="2.749848771s" totalImagesPullingTime="1.287915985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:19:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:19:47.747846117 +0000 UTC m=+911.115035985" watchObservedRunningTime="2026-07-27 21:19:47.749848771 +0000 UTC m=+911.117038730"
Jul 27 21:19:48.736909 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:48.736877    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-85khn"
Jul 27 21:19:48.738751 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:48.738727    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-85khn"
Jul 27 21:19:50.747092 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:50.747058    2577 generic.go:356] "Generic (PLEG): container finished" podID="85e94499-a400-45f4-b021-3bda235a5580" containerID="ce2dd91139751d307848e8e26489ce94b7b638588820121e758465582cc41665" exitCode=0
Jul 27 21:19:50.747436 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:50.747124    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg" event={"ID":"85e94499-a400-45f4-b021-3bda235a5580","Type":"ContainerDied","Data":"ce2dd91139751d307848e8e26489ce94b7b638588820121e758465582cc41665"}
Jul 27 21:19:51.350659 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:51.350607    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg" podUID="85e94499-a400-45f4-b021-3bda235a5580" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Jul 27 21:19:51.354920 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:51.354883    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg" podUID="85e94499-a400-45f4-b021-3bda235a5580" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:5000: connect: connection refused"
Jul 27 21:19:51.355293 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:51.355270    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg" podUID="85e94499-a400-45f4-b021-3bda235a5580" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:19:55.751524 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:55.751498    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-85khn"
Jul 27 21:19:56.350745 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:56.350701    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg" podUID="85e94499-a400-45f4-b021-3bda235a5580" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Jul 27 21:19:56.350958 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:19:56.350868    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg"
Jul 27 21:20:01.350367 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:01.350322    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg" podUID="85e94499-a400-45f4-b021-3bda235a5580" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Jul 27 21:20:01.354648 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:01.354613    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg" podUID="85e94499-a400-45f4-b021-3bda235a5580" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:5000: connect: connection refused"
Jul 27 21:20:01.355078 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:01.355050    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg" podUID="85e94499-a400-45f4-b021-3bda235a5580" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:20:05.390947 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:05.390900    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:05.419685 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:05.419654    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9"]
Jul 27 21:20:05.422143 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:05.422122    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9"
Jul 27 21:20:05.424559 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:05.424537    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Jul 27 21:20:05.424700 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:05.424577    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Jul 27 21:20:05.434690 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:05.434664    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9"]
Jul 27 21:20:05.532085 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:05.532048    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:05.570656 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:05.570626    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f31527c-46ab-4f25-a0b8-4722b846b68f-cabundle-cert\") pod \"isvc-logger-predictor-6b6746bb7-njkq9\" (UID: \"3f31527c-46ab-4f25-a0b8-4722b846b68f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9"
Jul 27 21:20:05.570827 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:05.570670    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f31527c-46ab-4f25-a0b8-4722b846b68f-proxy-tls\") pod \"isvc-logger-predictor-6b6746bb7-njkq9\" (UID: \"3f31527c-46ab-4f25-a0b8-4722b846b68f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9"
Jul 27 21:20:05.570827 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:05.570726    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7tlx\" (UniqueName: \"kubernetes.io/projected/3f31527c-46ab-4f25-a0b8-4722b846b68f-kube-api-access-p7tlx\") pod \"isvc-logger-predictor-6b6746bb7-njkq9\" (UID: \"3f31527c-46ab-4f25-a0b8-4722b846b68f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9"
Jul 27 21:20:05.570827 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:05.570745    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f31527c-46ab-4f25-a0b8-4722b846b68f-kserve-provision-location\") pod \"isvc-logger-predictor-6b6746bb7-njkq9\" (UID: \"3f31527c-46ab-4f25-a0b8-4722b846b68f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9"
Jul 27 21:20:05.570827 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:05.570789    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f31527c-46ab-4f25-a0b8-4722b846b68f-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6b6746bb7-njkq9\" (UID: \"3f31527c-46ab-4f25-a0b8-4722b846b68f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9"
Jul 27 21:20:05.671395 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:05.671369    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f31527c-46ab-4f25-a0b8-4722b846b68f-cabundle-cert\") pod \"isvc-logger-predictor-6b6746bb7-njkq9\" (UID: \"3f31527c-46ab-4f25-a0b8-4722b846b68f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9"
Jul 27 21:20:05.671571 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:05.671428    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f31527c-46ab-4f25-a0b8-4722b846b68f-proxy-tls\") pod \"isvc-logger-predictor-6b6746bb7-njkq9\" (UID: \"3f31527c-46ab-4f25-a0b8-4722b846b68f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9"
Jul 27 21:20:05.671571 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:05.671482    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7tlx\" (UniqueName: \"kubernetes.io/projected/3f31527c-46ab-4f25-a0b8-4722b846b68f-kube-api-access-p7tlx\") pod \"isvc-logger-predictor-6b6746bb7-njkq9\" (UID: \"3f31527c-46ab-4f25-a0b8-4722b846b68f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9"
Jul 27 21:20:05.671683 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:05.671611    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f31527c-46ab-4f25-a0b8-4722b846b68f-kserve-provision-location\") pod \"isvc-logger-predictor-6b6746bb7-njkq9\" (UID: \"3f31527c-46ab-4f25-a0b8-4722b846b68f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9"
Jul 27 21:20:05.671683 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:05.671665    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f31527c-46ab-4f25-a0b8-4722b846b68f-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6b6746bb7-njkq9\" (UID: \"3f31527c-46ab-4f25-a0b8-4722b846b68f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9"
Jul 27 21:20:05.672020 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:05.672001    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f31527c-46ab-4f25-a0b8-4722b846b68f-kserve-provision-location\") pod \"isvc-logger-predictor-6b6746bb7-njkq9\" (UID: \"3f31527c-46ab-4f25-a0b8-4722b846b68f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9"
Jul 27 21:20:05.672178 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:05.672156    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f31527c-46ab-4f25-a0b8-4722b846b68f-cabundle-cert\") pod \"isvc-logger-predictor-6b6746bb7-njkq9\" (UID: \"3f31527c-46ab-4f25-a0b8-4722b846b68f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9"
Jul 27 21:20:05.672280 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:05.672263    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f31527c-46ab-4f25-a0b8-4722b846b68f-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6b6746bb7-njkq9\" (UID: \"3f31527c-46ab-4f25-a0b8-4722b846b68f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9"
Jul 27 21:20:05.673982 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:05.673965    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f31527c-46ab-4f25-a0b8-4722b846b68f-proxy-tls\") pod \"isvc-logger-predictor-6b6746bb7-njkq9\" (UID: \"3f31527c-46ab-4f25-a0b8-4722b846b68f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9"
Jul 27 21:20:05.679755 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:05.679730    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7tlx\" (UniqueName: \"kubernetes.io/projected/3f31527c-46ab-4f25-a0b8-4722b846b68f-kube-api-access-p7tlx\") pod \"isvc-logger-predictor-6b6746bb7-njkq9\" (UID: \"3f31527c-46ab-4f25-a0b8-4722b846b68f\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9"
Jul 27 21:20:05.733360 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:05.733334    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9"
Jul 27 21:20:05.860066 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:05.860024    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9"]
Jul 27 21:20:05.861499 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:20:05.861474    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f31527c_46ab_4f25_a0b8_4722b846b68f.slice/crio-f29f738d29a29c966251fd399926b00cfb97e4c783f268ad3437eec21b971099 WatchSource:0}: Error finding container f29f738d29a29c966251fd399926b00cfb97e4c783f268ad3437eec21b971099: Status 404 returned error can't find the container with id f29f738d29a29c966251fd399926b00cfb97e4c783f268ad3437eec21b971099
Jul 27 21:20:06.351033 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:06.350986    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg" podUID="85e94499-a400-45f4-b021-3bda235a5580" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Jul 27 21:20:06.812306 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:06.812271    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9" event={"ID":"3f31527c-46ab-4f25-a0b8-4722b846b68f","Type":"ContainerStarted","Data":"1f59ee999c494f9c27604d6eb4e1686e27dd0118d7331eb067714d134d7fc715"}
Jul 27 21:20:06.812810 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:06.812344    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9" event={"ID":"3f31527c-46ab-4f25-a0b8-4722b846b68f","Type":"ContainerStarted","Data":"f29f738d29a29c966251fd399926b00cfb97e4c783f268ad3437eec21b971099"}
Jul 27 21:20:10.830592 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:10.830557    2577 generic.go:356] "Generic (PLEG): container finished" podID="3f31527c-46ab-4f25-a0b8-4722b846b68f" containerID="1f59ee999c494f9c27604d6eb4e1686e27dd0118d7331eb067714d134d7fc715" exitCode=0
Jul 27 21:20:10.831009 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:10.830628    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9" event={"ID":"3f31527c-46ab-4f25-a0b8-4722b846b68f","Type":"ContainerDied","Data":"1f59ee999c494f9c27604d6eb4e1686e27dd0118d7331eb067714d134d7fc715"}
Jul 27 21:20:11.351134 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:11.351092    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg" podUID="85e94499-a400-45f4-b021-3bda235a5580" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Jul 27 21:20:11.355437 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:11.355406    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg" podUID="85e94499-a400-45f4-b021-3bda235a5580" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:5000: connect: connection refused"
Jul 27 21:20:11.355588 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:11.355533    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg"
Jul 27 21:20:11.355820 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:11.355797    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg" podUID="85e94499-a400-45f4-b021-3bda235a5580" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:20:11.355912 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:11.355889    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg"
Jul 27 21:20:11.836710 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:11.836678    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9" event={"ID":"3f31527c-46ab-4f25-a0b8-4722b846b68f","Type":"ContainerStarted","Data":"3aa38ddf283fa88350d9e235f09c773a84e38ac01a2eb05b31b1c4c245be66d2"}
Jul 27 21:20:11.837108 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:11.836741    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9" event={"ID":"3f31527c-46ab-4f25-a0b8-4722b846b68f","Type":"ContainerStarted","Data":"0311f40d53afb2115388a398ac58232ab7b6bdbd82208f3a153a9e49230b2c56"}
Jul 27 21:20:11.837108 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:11.836781    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9" event={"ID":"3f31527c-46ab-4f25-a0b8-4722b846b68f","Type":"ContainerStarted","Data":"1e8589ff158ebb00734f861c8608b3dfbb0abc5aef9c3a153b7d44fdae73308d"}
Jul 27 21:20:11.837108 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:11.837005    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9"
Jul 27 21:20:11.858583 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:11.858533    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9" podStartSLOduration=2.954099282 podStartE2EDuration="6.858517908s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.904418626s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:20:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:20:11.857161988 +0000 UTC m=+935.224351855" watchObservedRunningTime="2026-07-27 21:20:11.858517908 +0000 UTC m=+935.225707774"
Jul 27 21:20:12.841000 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:12.840971    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9"
Jul 27 21:20:12.841364 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:12.841032    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9"
Jul 27 21:20:12.842330 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:12.842292    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9" podUID="3f31527c-46ab-4f25-a0b8-4722b846b68f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 27 21:20:12.842972 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:12.842950    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9" podUID="3f31527c-46ab-4f25-a0b8-4722b846b68f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:20:13.845010 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:13.844965    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9" podUID="3f31527c-46ab-4f25-a0b8-4722b846b68f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 27 21:20:13.845420 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:13.845330    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9" podUID="3f31527c-46ab-4f25-a0b8-4722b846b68f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:20:15.857668 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:15.857614    2577 generic.go:356] "Generic (PLEG): container finished" podID="85e94499-a400-45f4-b021-3bda235a5580" containerID="969e974d8109e2fae4facb72de9ec7312b0172442519bc6490db2f9ec3925b6c" exitCode=0
Jul 27 21:20:15.858074 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:15.857682    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg" event={"ID":"85e94499-a400-45f4-b021-3bda235a5580","Type":"ContainerDied","Data":"969e974d8109e2fae4facb72de9ec7312b0172442519bc6490db2f9ec3925b6c"}
Jul 27 21:20:16.065158 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:16.065133    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg"
Jul 27 21:20:16.166475 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:16.166395    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85e94499-a400-45f4-b021-3bda235a5580-kube-rbac-proxy-sar-config\") pod \"85e94499-a400-45f4-b021-3bda235a5580\" (UID: \"85e94499-a400-45f4-b021-3bda235a5580\") "
Jul 27 21:20:16.166475 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:16.166433    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85e94499-a400-45f4-b021-3bda235a5580-proxy-tls\") pod \"85e94499-a400-45f4-b021-3bda235a5580\" (UID: \"85e94499-a400-45f4-b021-3bda235a5580\") "
Jul 27 21:20:16.166475 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:16.166473    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8z7jc\" (UniqueName: \"kubernetes.io/projected/85e94499-a400-45f4-b021-3bda235a5580-kube-api-access-8z7jc\") pod \"85e94499-a400-45f4-b021-3bda235a5580\" (UID: \"85e94499-a400-45f4-b021-3bda235a5580\") "
Jul 27 21:20:16.166741 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:16.166501    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85e94499-a400-45f4-b021-3bda235a5580-cabundle-cert\") pod \"85e94499-a400-45f4-b021-3bda235a5580\" (UID: \"85e94499-a400-45f4-b021-3bda235a5580\") "
Jul 27 21:20:16.166741 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:16.166601    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85e94499-a400-45f4-b021-3bda235a5580-kserve-provision-location\") pod \"85e94499-a400-45f4-b021-3bda235a5580\" (UID: \"85e94499-a400-45f4-b021-3bda235a5580\") "
Jul 27 21:20:16.166881 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:16.166836    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85e94499-a400-45f4-b021-3bda235a5580-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "85e94499-a400-45f4-b021-3bda235a5580" (UID: "85e94499-a400-45f4-b021-3bda235a5580"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:20:16.166994 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:16.166965    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85e94499-a400-45f4-b021-3bda235a5580-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "85e94499-a400-45f4-b021-3bda235a5580" (UID: "85e94499-a400-45f4-b021-3bda235a5580"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:20:16.167053 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:16.167006    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85e94499-a400-45f4-b021-3bda235a5580-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "85e94499-a400-45f4-b021-3bda235a5580" (UID: "85e94499-a400-45f4-b021-3bda235a5580"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:20:16.168650 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:16.168629    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85e94499-a400-45f4-b021-3bda235a5580-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "85e94499-a400-45f4-b021-3bda235a5580" (UID: "85e94499-a400-45f4-b021-3bda235a5580"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:20:16.168705 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:16.168681    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85e94499-a400-45f4-b021-3bda235a5580-kube-api-access-8z7jc" (OuterVolumeSpecName: "kube-api-access-8z7jc") pod "85e94499-a400-45f4-b021-3bda235a5580" (UID: "85e94499-a400-45f4-b021-3bda235a5580"). InnerVolumeSpecName "kube-api-access-8z7jc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:20:16.267346 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:16.267291    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z7jc\" (UniqueName: \"kubernetes.io/projected/85e94499-a400-45f4-b021-3bda235a5580-kube-api-access-8z7jc\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:20:16.267346 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:16.267323    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85e94499-a400-45f4-b021-3bda235a5580-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:20:16.267346 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:16.267338    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85e94499-a400-45f4-b021-3bda235a5580-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:20:16.267346 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:16.267352    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85e94499-a400-45f4-b021-3bda235a5580-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:20:16.267618 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:16.267364    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85e94499-a400-45f4-b021-3bda235a5580-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:20:16.863350 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:16.863321    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg"
Jul 27 21:20:16.863775 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:16.863319    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg" event={"ID":"85e94499-a400-45f4-b021-3bda235a5580","Type":"ContainerDied","Data":"34b71dc22290905756ea8d47ad9d9762af6637325a7711a08b53aa53b30edde4"}
Jul 27 21:20:16.863775 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:16.863480    2577 scope.go:117] "RemoveContainer" containerID="969e974d8109e2fae4facb72de9ec7312b0172442519bc6490db2f9ec3925b6c"
Jul 27 21:20:16.872686 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:16.872664    2577 scope.go:117] "RemoveContainer" containerID="5b437bca27827ddc9dedd56c5c1b12e7da9c34373eefe29f7e28f6720c122048"
Jul 27 21:20:16.879701 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:16.879686    2577 scope.go:117] "RemoveContainer" containerID="ce2dd91139751d307848e8e26489ce94b7b638588820121e758465582cc41665"
Jul 27 21:20:16.884653 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:16.884629    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg"]
Jul 27 21:20:16.886973 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:16.886954    2577 scope.go:117] "RemoveContainer" containerID="e2737d686650d91bb249705968a6c90c0aa8a94f122dc1dd10d6247a40107f51"
Jul 27 21:20:16.890666 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:16.890648    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-p9shg"]
Jul 27 21:20:17.136179 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:17.136103    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85e94499-a400-45f4-b021-3bda235a5580" path="/var/lib/kubelet/pods/85e94499-a400-45f4-b021-3bda235a5580/volumes"
Jul 27 21:20:18.849279 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:18.849251    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9"
Jul 27 21:20:18.849888 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:18.849858    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9" podUID="3f31527c-46ab-4f25-a0b8-4722b846b68f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 27 21:20:18.850140 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:18.850114    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9" podUID="3f31527c-46ab-4f25-a0b8-4722b846b68f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:20:28.850730 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:28.850687    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9" podUID="3f31527c-46ab-4f25-a0b8-4722b846b68f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 27 21:20:28.851186 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:28.851164    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9" podUID="3f31527c-46ab-4f25-a0b8-4722b846b68f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:20:38.850279 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:38.850234    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9" podUID="3f31527c-46ab-4f25-a0b8-4722b846b68f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 27 21:20:38.850751 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:38.850731    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9" podUID="3f31527c-46ab-4f25-a0b8-4722b846b68f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:20:48.849877 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:48.849824    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9" podUID="3f31527c-46ab-4f25-a0b8-4722b846b68f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 27 21:20:48.850352 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:48.850210    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9" podUID="3f31527c-46ab-4f25-a0b8-4722b846b68f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:20:58.849895 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:58.849800    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9" podUID="3f31527c-46ab-4f25-a0b8-4722b846b68f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 27 21:20:58.850407 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:20:58.850226    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9" podUID="3f31527c-46ab-4f25-a0b8-4722b846b68f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:21:08.850675 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:08.850631    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9" podUID="3f31527c-46ab-4f25-a0b8-4722b846b68f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 27 21:21:08.851124 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:08.851077    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9" podUID="3f31527c-46ab-4f25-a0b8-4722b846b68f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:21:18.850620 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:18.850575    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9" podUID="3f31527c-46ab-4f25-a0b8-4722b846b68f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 27 21:21:18.851109 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:18.851085    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9" podUID="3f31527c-46ab-4f25-a0b8-4722b846b68f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:21:28.130992 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:28.130956    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9"
Jul 27 21:21:28.131360 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:28.131137    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9"
Jul 27 21:21:40.439980 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:40.439950    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-85khn_b6343eda-b4e9-4058-a547-318b2ebf6d90/kserve-container/0.log"
Jul 27 21:21:40.661474 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:40.661443    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9"]
Jul 27 21:21:40.661756 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:40.661727    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9" podUID="3f31527c-46ab-4f25-a0b8-4722b846b68f" containerName="kserve-container" containerID="cri-o://1e8589ff158ebb00734f861c8608b3dfbb0abc5aef9c3a153b7d44fdae73308d" gracePeriod=30
Jul 27 21:21:40.661888 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:40.661779    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9" podUID="3f31527c-46ab-4f25-a0b8-4722b846b68f" containerName="kube-rbac-proxy" containerID="cri-o://0311f40d53afb2115388a398ac58232ab7b6bdbd82208f3a153a9e49230b2c56" gracePeriod=30
Jul 27 21:21:40.661888 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:40.661796    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9" podUID="3f31527c-46ab-4f25-a0b8-4722b846b68f" containerName="agent" containerID="cri-o://3aa38ddf283fa88350d9e235f09c773a84e38ac01a2eb05b31b1c4c245be66d2" gracePeriod=30
Jul 27 21:21:40.690305 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:40.690246    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:21:40.726185 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:40.726161    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cbsds"]
Jul 27 21:21:40.726484 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:40.726473    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85e94499-a400-45f4-b021-3bda235a5580" containerName="agent"
Jul 27 21:21:40.726528 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:40.726487    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="85e94499-a400-45f4-b021-3bda235a5580" containerName="agent"
Jul 27 21:21:40.726528 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:40.726496    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85e94499-a400-45f4-b021-3bda235a5580" containerName="storage-initializer"
Jul 27 21:21:40.726528 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:40.726502    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="85e94499-a400-45f4-b021-3bda235a5580" containerName="storage-initializer"
Jul 27 21:21:40.726528 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:40.726515    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85e94499-a400-45f4-b021-3bda235a5580" containerName="kube-rbac-proxy"
Jul 27 21:21:40.726528 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:40.726521    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="85e94499-a400-45f4-b021-3bda235a5580" containerName="kube-rbac-proxy"
Jul 27 21:21:40.726684 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:40.726536    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85e94499-a400-45f4-b021-3bda235a5580" containerName="kserve-container"
Jul 27 21:21:40.726684 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:40.726542    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="85e94499-a400-45f4-b021-3bda235a5580" containerName="kserve-container"
Jul 27 21:21:40.726684 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:40.726592    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="85e94499-a400-45f4-b021-3bda235a5580" containerName="kserve-container"
Jul 27 21:21:40.726684 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:40.726598    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="85e94499-a400-45f4-b021-3bda235a5580" containerName="kube-rbac-proxy"
Jul 27 21:21:40.726684 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:40.726604    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="85e94499-a400-45f4-b021-3bda235a5580" containerName="agent"
Jul 27 21:21:40.729657 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:40.729637    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cbsds"
Jul 27 21:21:40.731925 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:40.731903    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Jul 27 21:21:40.732033 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:40.731955    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Jul 27 21:21:40.738616 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:40.738591    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzmbx\" (UniqueName: \"kubernetes.io/projected/7e80f21b-5c24-4dd5-9bc7-a29706f4e282-kube-api-access-pzmbx\") pod \"isvc-autogluon-v1-predictor-574b5b7495-cbsds\" (UID: \"7e80f21b-5c24-4dd5-9bc7-a29706f4e282\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cbsds"
Jul 27 21:21:40.738722 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:40.738643    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e80f21b-5c24-4dd5-9bc7-a29706f4e282-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-cbsds\" (UID: \"7e80f21b-5c24-4dd5-9bc7-a29706f4e282\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cbsds"
Jul 27 21:21:40.738858 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:40.738729    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e80f21b-5c24-4dd5-9bc7-a29706f4e282-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-cbsds\" (UID: \"7e80f21b-5c24-4dd5-9bc7-a29706f4e282\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cbsds"
Jul 27 21:21:40.738858 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:40.738793    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e80f21b-5c24-4dd5-9bc7-a29706f4e282-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-cbsds\" (UID: \"7e80f21b-5c24-4dd5-9bc7-a29706f4e282\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cbsds"
Jul 27 21:21:40.738858 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:40.738838    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e80f21b-5c24-4dd5-9bc7-a29706f4e282-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-cbsds\" (UID: \"7e80f21b-5c24-4dd5-9bc7-a29706f4e282\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cbsds"
Jul 27 21:21:40.740595 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:40.740577    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cbsds"]
Jul 27 21:21:40.796631 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:40.796606    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-85khn"]
Jul 27 21:21:40.796867 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:40.796838    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-85khn" podUID="b6343eda-b4e9-4058-a547-318b2ebf6d90" containerName="kserve-container" containerID="cri-o://06d89aa1a53684a54e43e05e4f4a46ed2adfab2748f4b258d11c4df3ea790339" gracePeriod=30
Jul 27 21:21:40.796998 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:40.796878    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-85khn" podUID="b6343eda-b4e9-4058-a547-318b2ebf6d90" containerName="kube-rbac-proxy" containerID="cri-o://435c630d67191ebd532540d9a44705c35fa27d2c545a5175eda3f77cdfa21def" gracePeriod=30
Jul 27 21:21:40.839369 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:40.839350    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzmbx\" (UniqueName: \"kubernetes.io/projected/7e80f21b-5c24-4dd5-9bc7-a29706f4e282-kube-api-access-pzmbx\") pod \"isvc-autogluon-v1-predictor-574b5b7495-cbsds\" (UID: \"7e80f21b-5c24-4dd5-9bc7-a29706f4e282\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cbsds"
Jul 27 21:21:40.839461 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:40.839380    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e80f21b-5c24-4dd5-9bc7-a29706f4e282-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-cbsds\" (UID: \"7e80f21b-5c24-4dd5-9bc7-a29706f4e282\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cbsds"
Jul 27 21:21:40.839461 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:40.839407    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e80f21b-5c24-4dd5-9bc7-a29706f4e282-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-cbsds\" (UID: \"7e80f21b-5c24-4dd5-9bc7-a29706f4e282\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cbsds"
Jul 27 21:21:40.839596 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:40.839573    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e80f21b-5c24-4dd5-9bc7-a29706f4e282-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-cbsds\" (UID: \"7e80f21b-5c24-4dd5-9bc7-a29706f4e282\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cbsds"
Jul 27 21:21:40.839644 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:40.839631    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e80f21b-5c24-4dd5-9bc7-a29706f4e282-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-cbsds\" (UID: \"7e80f21b-5c24-4dd5-9bc7-a29706f4e282\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cbsds"
Jul 27 21:21:40.839796 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:21:40.839745    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v1-predictor-serving-cert: secret "isvc-autogluon-v1-predictor-serving-cert" not found
Jul 27 21:21:40.839796 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:40.839776    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e80f21b-5c24-4dd5-9bc7-a29706f4e282-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-cbsds\" (UID: \"7e80f21b-5c24-4dd5-9bc7-a29706f4e282\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cbsds"
Jul 27 21:21:40.839892 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:21:40.839812    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e80f21b-5c24-4dd5-9bc7-a29706f4e282-proxy-tls podName:7e80f21b-5c24-4dd5-9bc7-a29706f4e282 nodeName:}" failed. No retries permitted until 2026-07-27 21:21:41.33979517 +0000 UTC m=+1024.706985026 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7e80f21b-5c24-4dd5-9bc7-a29706f4e282-proxy-tls") pod "isvc-autogluon-v1-predictor-574b5b7495-cbsds" (UID: "7e80f21b-5c24-4dd5-9bc7-a29706f4e282") : secret "isvc-autogluon-v1-predictor-serving-cert" not found
Jul 27 21:21:40.840062 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:40.840043    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e80f21b-5c24-4dd5-9bc7-a29706f4e282-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-cbsds\" (UID: \"7e80f21b-5c24-4dd5-9bc7-a29706f4e282\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cbsds"
Jul 27 21:21:40.840195 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:40.840180    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e80f21b-5c24-4dd5-9bc7-a29706f4e282-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-cbsds\" (UID: \"7e80f21b-5c24-4dd5-9bc7-a29706f4e282\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cbsds"
Jul 27 21:21:40.847524 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:40.847503    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzmbx\" (UniqueName: \"kubernetes.io/projected/7e80f21b-5c24-4dd5-9bc7-a29706f4e282-kube-api-access-pzmbx\") pod \"isvc-autogluon-v1-predictor-574b5b7495-cbsds\" (UID: \"7e80f21b-5c24-4dd5-9bc7-a29706f4e282\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cbsds"
Jul 27 21:21:41.045261 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:41.045227    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-85khn"
Jul 27 21:21:41.142262 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:41.142238    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2lvp\" (UniqueName: \"kubernetes.io/projected/b6343eda-b4e9-4058-a547-318b2ebf6d90-kube-api-access-k2lvp\") pod \"b6343eda-b4e9-4058-a547-318b2ebf6d90\" (UID: \"b6343eda-b4e9-4058-a547-318b2ebf6d90\") "
Jul 27 21:21:41.142443 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:41.142269    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6343eda-b4e9-4058-a547-318b2ebf6d90-kube-rbac-proxy-sar-config\") pod \"b6343eda-b4e9-4058-a547-318b2ebf6d90\" (UID: \"b6343eda-b4e9-4058-a547-318b2ebf6d90\") "
Jul 27 21:21:41.142443 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:41.142350    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6343eda-b4e9-4058-a547-318b2ebf6d90-proxy-tls\") pod \"b6343eda-b4e9-4058-a547-318b2ebf6d90\" (UID: \"b6343eda-b4e9-4058-a547-318b2ebf6d90\") "
Jul 27 21:21:41.142683 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:41.142651    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6343eda-b4e9-4058-a547-318b2ebf6d90-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b6343eda-b4e9-4058-a547-318b2ebf6d90" (UID: "b6343eda-b4e9-4058-a547-318b2ebf6d90"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:21:41.144307 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:41.144284    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6343eda-b4e9-4058-a547-318b2ebf6d90-kube-api-access-k2lvp" (OuterVolumeSpecName: "kube-api-access-k2lvp") pod "b6343eda-b4e9-4058-a547-318b2ebf6d90" (UID: "b6343eda-b4e9-4058-a547-318b2ebf6d90"). InnerVolumeSpecName "kube-api-access-k2lvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:21:41.144368 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:41.144286    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6343eda-b4e9-4058-a547-318b2ebf6d90-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b6343eda-b4e9-4058-a547-318b2ebf6d90" (UID: "b6343eda-b4e9-4058-a547-318b2ebf6d90"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:21:41.232322 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:41.232246    2577 generic.go:356] "Generic (PLEG): container finished" podID="3f31527c-46ab-4f25-a0b8-4722b846b68f" containerID="0311f40d53afb2115388a398ac58232ab7b6bdbd82208f3a153a9e49230b2c56" exitCode=2
Jul 27 21:21:41.232452 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:41.232319    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9" event={"ID":"3f31527c-46ab-4f25-a0b8-4722b846b68f","Type":"ContainerDied","Data":"0311f40d53afb2115388a398ac58232ab7b6bdbd82208f3a153a9e49230b2c56"}
Jul 27 21:21:41.233842 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:41.233823    2577 generic.go:356] "Generic (PLEG): container finished" podID="b6343eda-b4e9-4058-a547-318b2ebf6d90" containerID="435c630d67191ebd532540d9a44705c35fa27d2c545a5175eda3f77cdfa21def" exitCode=2
Jul 27 21:21:41.233842 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:41.233841    2577 generic.go:356] "Generic (PLEG): container finished" podID="b6343eda-b4e9-4058-a547-318b2ebf6d90" containerID="06d89aa1a53684a54e43e05e4f4a46ed2adfab2748f4b258d11c4df3ea790339" exitCode=2
Jul 27 21:21:41.233981 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:41.233878    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-85khn" event={"ID":"b6343eda-b4e9-4058-a547-318b2ebf6d90","Type":"ContainerDied","Data":"435c630d67191ebd532540d9a44705c35fa27d2c545a5175eda3f77cdfa21def"}
Jul 27 21:21:41.233981 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:41.233883    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-85khn"
Jul 27 21:21:41.233981 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:41.233921    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-85khn" event={"ID":"b6343eda-b4e9-4058-a547-318b2ebf6d90","Type":"ContainerDied","Data":"06d89aa1a53684a54e43e05e4f4a46ed2adfab2748f4b258d11c4df3ea790339"}
Jul 27 21:21:41.233981 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:41.233949    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-85khn" event={"ID":"b6343eda-b4e9-4058-a547-318b2ebf6d90","Type":"ContainerDied","Data":"2918ad8892f4f7442fa1f07a1b1c5c99435d30c5d2a0e77d6f272aaa6952d967"}
Jul 27 21:21:41.233981 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:41.233973    2577 scope.go:117] "RemoveContainer" containerID="435c630d67191ebd532540d9a44705c35fa27d2c545a5175eda3f77cdfa21def"
Jul 27 21:21:41.243110 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:41.243091    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6343eda-b4e9-4058-a547-318b2ebf6d90-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:21:41.243215 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:41.243114    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2lvp\" (UniqueName: \"kubernetes.io/projected/b6343eda-b4e9-4058-a547-318b2ebf6d90-kube-api-access-k2lvp\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:21:41.243215 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:41.243127    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6343eda-b4e9-4058-a547-318b2ebf6d90-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:21:41.244470 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:41.244448    2577 scope.go:117] "RemoveContainer" containerID="06d89aa1a53684a54e43e05e4f4a46ed2adfab2748f4b258d11c4df3ea790339"
Jul 27 21:21:41.251443 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:41.251425    2577 scope.go:117] "RemoveContainer" containerID="435c630d67191ebd532540d9a44705c35fa27d2c545a5175eda3f77cdfa21def"
Jul 27 21:21:41.251691 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:21:41.251672    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"435c630d67191ebd532540d9a44705c35fa27d2c545a5175eda3f77cdfa21def\": container with ID starting with 435c630d67191ebd532540d9a44705c35fa27d2c545a5175eda3f77cdfa21def not found: ID does not exist" containerID="435c630d67191ebd532540d9a44705c35fa27d2c545a5175eda3f77cdfa21def"
Jul 27 21:21:41.251735 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:41.251699    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"435c630d67191ebd532540d9a44705c35fa27d2c545a5175eda3f77cdfa21def"} err="failed to get container status \"435c630d67191ebd532540d9a44705c35fa27d2c545a5175eda3f77cdfa21def\": rpc error: code = NotFound desc = could not find container \"435c630d67191ebd532540d9a44705c35fa27d2c545a5175eda3f77cdfa21def\": container with ID starting with 435c630d67191ebd532540d9a44705c35fa27d2c545a5175eda3f77cdfa21def not found: ID does not exist"
Jul 27 21:21:41.251735 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:41.251716    2577 scope.go:117] "RemoveContainer" containerID="06d89aa1a53684a54e43e05e4f4a46ed2adfab2748f4b258d11c4df3ea790339"
Jul 27 21:21:41.251989 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:21:41.251970    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06d89aa1a53684a54e43e05e4f4a46ed2adfab2748f4b258d11c4df3ea790339\": container with ID starting with 06d89aa1a53684a54e43e05e4f4a46ed2adfab2748f4b258d11c4df3ea790339 not found: ID does not exist" containerID="06d89aa1a53684a54e43e05e4f4a46ed2adfab2748f4b258d11c4df3ea790339"
Jul 27 21:21:41.252049 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:41.251993    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06d89aa1a53684a54e43e05e4f4a46ed2adfab2748f4b258d11c4df3ea790339"} err="failed to get container status \"06d89aa1a53684a54e43e05e4f4a46ed2adfab2748f4b258d11c4df3ea790339\": rpc error: code = NotFound desc = could not find container \"06d89aa1a53684a54e43e05e4f4a46ed2adfab2748f4b258d11c4df3ea790339\": container with ID starting with 06d89aa1a53684a54e43e05e4f4a46ed2adfab2748f4b258d11c4df3ea790339 not found: ID does not exist"
Jul 27 21:21:41.252049 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:41.252008    2577 scope.go:117] "RemoveContainer" containerID="435c630d67191ebd532540d9a44705c35fa27d2c545a5175eda3f77cdfa21def"
Jul 27 21:21:41.252236 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:41.252218    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"435c630d67191ebd532540d9a44705c35fa27d2c545a5175eda3f77cdfa21def"} err="failed to get container status \"435c630d67191ebd532540d9a44705c35fa27d2c545a5175eda3f77cdfa21def\": rpc error: code = NotFound desc = could not find container \"435c630d67191ebd532540d9a44705c35fa27d2c545a5175eda3f77cdfa21def\": container with ID starting with 435c630d67191ebd532540d9a44705c35fa27d2c545a5175eda3f77cdfa21def not found: ID does not exist"
Jul 27 21:21:41.252278 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:41.252237    2577 scope.go:117] "RemoveContainer" containerID="06d89aa1a53684a54e43e05e4f4a46ed2adfab2748f4b258d11c4df3ea790339"
Jul 27 21:21:41.252468 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:41.252451    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06d89aa1a53684a54e43e05e4f4a46ed2adfab2748f4b258d11c4df3ea790339"} err="failed to get container status \"06d89aa1a53684a54e43e05e4f4a46ed2adfab2748f4b258d11c4df3ea790339\": rpc error: code = NotFound desc = could not find container \"06d89aa1a53684a54e43e05e4f4a46ed2adfab2748f4b258d11c4df3ea790339\": container with ID starting with 06d89aa1a53684a54e43e05e4f4a46ed2adfab2748f4b258d11c4df3ea790339 not found: ID does not exist"
Jul 27 21:21:41.255542 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:41.255522    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-85khn"]
Jul 27 21:21:41.259695 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:41.259670    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-85khn"]
Jul 27 21:21:41.344214 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:41.344194    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e80f21b-5c24-4dd5-9bc7-a29706f4e282-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-cbsds\" (UID: \"7e80f21b-5c24-4dd5-9bc7-a29706f4e282\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cbsds"
Jul 27 21:21:41.346431 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:41.346408    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e80f21b-5c24-4dd5-9bc7-a29706f4e282-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-cbsds\" (UID: \"7e80f21b-5c24-4dd5-9bc7-a29706f4e282\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cbsds"
Jul 27 21:21:41.639572 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:41.639489    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cbsds"
Jul 27 21:21:41.774160 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:41.774137    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cbsds"]
Jul 27 21:21:41.776510 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:21:41.776483    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e80f21b_5c24_4dd5_9bc7_a29706f4e282.slice/crio-1a7f294469b526c5d9428c94aec6c947614ca0ab6442c93c2e46acbbe4eee6c5 WatchSource:0}: Error finding container 1a7f294469b526c5d9428c94aec6c947614ca0ab6442c93c2e46acbbe4eee6c5: Status 404 returned error can't find the container with id 1a7f294469b526c5d9428c94aec6c947614ca0ab6442c93c2e46acbbe4eee6c5
Jul 27 21:21:41.803645 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:41.803623    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:21:42.240775 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:42.240741    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cbsds" event={"ID":"7e80f21b-5c24-4dd5-9bc7-a29706f4e282","Type":"ContainerStarted","Data":"e13c7ab69f3b62a278edeef5459bd81ccf20071bebf13176a46106510e5c0333"}
Jul 27 21:21:42.240977 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:42.240804    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cbsds" event={"ID":"7e80f21b-5c24-4dd5-9bc7-a29706f4e282","Type":"ContainerStarted","Data":"1a7f294469b526c5d9428c94aec6c947614ca0ab6442c93c2e46acbbe4eee6c5"}
Jul 27 21:21:43.135240 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:43.135209    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6343eda-b4e9-4058-a547-318b2ebf6d90" path="/var/lib/kubelet/pods/b6343eda-b4e9-4058-a547-318b2ebf6d90/volumes"
Jul 27 21:21:43.846108 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:43.846066    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9" podUID="3f31527c-46ab-4f25-a0b8-4722b846b68f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jul 27 21:21:45.254917 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:45.254888    2577 generic.go:356] "Generic (PLEG): container finished" podID="3f31527c-46ab-4f25-a0b8-4722b846b68f" containerID="1e8589ff158ebb00734f861c8608b3dfbb0abc5aef9c3a153b7d44fdae73308d" exitCode=0
Jul 27 21:21:45.255271 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:45.254922    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9" event={"ID":"3f31527c-46ab-4f25-a0b8-4722b846b68f","Type":"ContainerDied","Data":"1e8589ff158ebb00734f861c8608b3dfbb0abc5aef9c3a153b7d44fdae73308d"}
Jul 27 21:21:48.130595 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:48.130552    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9" podUID="3f31527c-46ab-4f25-a0b8-4722b846b68f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 27 21:21:48.131062 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:48.130989    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9" podUID="3f31527c-46ab-4f25-a0b8-4722b846b68f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:21:48.846022 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:48.845979    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9" podUID="3f31527c-46ab-4f25-a0b8-4722b846b68f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jul 27 21:21:53.846039 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:53.845994    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9" podUID="3f31527c-46ab-4f25-a0b8-4722b846b68f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jul 27 21:21:53.846537 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:53.846114    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9"
Jul 27 21:21:58.130665 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:58.130616    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9" podUID="3f31527c-46ab-4f25-a0b8-4722b846b68f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 27 21:21:58.131103 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:58.130971    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9" podUID="3f31527c-46ab-4f25-a0b8-4722b846b68f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:21:58.845984 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:21:58.845942    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9" podUID="3f31527c-46ab-4f25-a0b8-4722b846b68f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jul 27 21:22:03.845300 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:22:03.845257    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9" podUID="3f31527c-46ab-4f25-a0b8-4722b846b68f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jul 27 21:22:08.131174 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:22:08.131120    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9" podUID="3f31527c-46ab-4f25-a0b8-4722b846b68f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 27 21:22:08.131618 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:22:08.131279    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9"
Jul 27 21:22:08.131618 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:22:08.131417    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9" podUID="3f31527c-46ab-4f25-a0b8-4722b846b68f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:22:08.131618 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:22:08.131556    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9"
Jul 27 21:22:08.845561 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:22:08.845522    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9" podUID="3f31527c-46ab-4f25-a0b8-4722b846b68f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jul 27 21:22:10.808195 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:22:10.808170    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9"
Jul 27 21:22:10.891529 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:22:10.891495    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f31527c-46ab-4f25-a0b8-4722b846b68f-kserve-provision-location\") pod \"3f31527c-46ab-4f25-a0b8-4722b846b68f\" (UID: \"3f31527c-46ab-4f25-a0b8-4722b846b68f\") "
Jul 27 21:22:10.891711 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:22:10.891552    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f31527c-46ab-4f25-a0b8-4722b846b68f-cabundle-cert\") pod \"3f31527c-46ab-4f25-a0b8-4722b846b68f\" (UID: \"3f31527c-46ab-4f25-a0b8-4722b846b68f\") "
Jul 27 21:22:10.891711 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:22:10.891574    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p7tlx\" (UniqueName: \"kubernetes.io/projected/3f31527c-46ab-4f25-a0b8-4722b846b68f-kube-api-access-p7tlx\") pod \"3f31527c-46ab-4f25-a0b8-4722b846b68f\" (UID: \"3f31527c-46ab-4f25-a0b8-4722b846b68f\") "
Jul 27 21:22:10.891711 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:22:10.891593    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f31527c-46ab-4f25-a0b8-4722b846b68f-proxy-tls\") pod \"3f31527c-46ab-4f25-a0b8-4722b846b68f\" (UID: \"3f31527c-46ab-4f25-a0b8-4722b846b68f\") "
Jul 27 21:22:10.891711 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:22:10.891641    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f31527c-46ab-4f25-a0b8-4722b846b68f-kube-rbac-proxy-sar-config\") pod \"3f31527c-46ab-4f25-a0b8-4722b846b68f\" (UID: \"3f31527c-46ab-4f25-a0b8-4722b846b68f\") "
Jul 27 21:22:10.891968 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:22:10.891860    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f31527c-46ab-4f25-a0b8-4722b846b68f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3f31527c-46ab-4f25-a0b8-4722b846b68f" (UID: "3f31527c-46ab-4f25-a0b8-4722b846b68f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:22:10.892024 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:22:10.891971    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f31527c-46ab-4f25-a0b8-4722b846b68f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3f31527c-46ab-4f25-a0b8-4722b846b68f" (UID: "3f31527c-46ab-4f25-a0b8-4722b846b68f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:22:10.892084 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:22:10.892064    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f31527c-46ab-4f25-a0b8-4722b846b68f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3f31527c-46ab-4f25-a0b8-4722b846b68f" (UID: "3f31527c-46ab-4f25-a0b8-4722b846b68f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:22:10.893775 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:22:10.893731    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f31527c-46ab-4f25-a0b8-4722b846b68f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3f31527c-46ab-4f25-a0b8-4722b846b68f" (UID: "3f31527c-46ab-4f25-a0b8-4722b846b68f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:22:10.893775 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:22:10.893750    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f31527c-46ab-4f25-a0b8-4722b846b68f-kube-api-access-p7tlx" (OuterVolumeSpecName: "kube-api-access-p7tlx") pod "3f31527c-46ab-4f25-a0b8-4722b846b68f" (UID: "3f31527c-46ab-4f25-a0b8-4722b846b68f"). InnerVolumeSpecName "kube-api-access-p7tlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:22:10.992166 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:22:10.992133    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f31527c-46ab-4f25-a0b8-4722b846b68f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:22:10.992166 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:22:10.992160    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f31527c-46ab-4f25-a0b8-4722b846b68f-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:22:10.992166 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:22:10.992170    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f31527c-46ab-4f25-a0b8-4722b846b68f-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:22:10.992408 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:22:10.992178    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7tlx\" (UniqueName: \"kubernetes.io/projected/3f31527c-46ab-4f25-a0b8-4722b846b68f-kube-api-access-p7tlx\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:22:10.992408 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:22:10.992189    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f31527c-46ab-4f25-a0b8-4722b846b68f-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:22:11.359354 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:22:11.359270    2577 generic.go:356] "Generic (PLEG): container finished" podID="7e80f21b-5c24-4dd5-9bc7-a29706f4e282" containerID="e13c7ab69f3b62a278edeef5459bd81ccf20071bebf13176a46106510e5c0333" exitCode=0
Jul 27 21:22:11.359354 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:22:11.359344    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cbsds" event={"ID":"7e80f21b-5c24-4dd5-9bc7-a29706f4e282","Type":"ContainerDied","Data":"e13c7ab69f3b62a278edeef5459bd81ccf20071bebf13176a46106510e5c0333"}
Jul 27 21:22:11.361787 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:22:11.361740    2577 generic.go:356] "Generic (PLEG): container finished" podID="3f31527c-46ab-4f25-a0b8-4722b846b68f" containerID="3aa38ddf283fa88350d9e235f09c773a84e38ac01a2eb05b31b1c4c245be66d2" exitCode=0
Jul 27 21:22:11.362050 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:22:11.361795    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9" event={"ID":"3f31527c-46ab-4f25-a0b8-4722b846b68f","Type":"ContainerDied","Data":"3aa38ddf283fa88350d9e235f09c773a84e38ac01a2eb05b31b1c4c245be66d2"}
Jul 27 21:22:11.362050 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:22:11.361829    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9"
Jul 27 21:22:11.362050 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:22:11.361847    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9" event={"ID":"3f31527c-46ab-4f25-a0b8-4722b846b68f","Type":"ContainerDied","Data":"f29f738d29a29c966251fd399926b00cfb97e4c783f268ad3437eec21b971099"}
Jul 27 21:22:11.362050 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:22:11.361884    2577 scope.go:117] "RemoveContainer" containerID="3aa38ddf283fa88350d9e235f09c773a84e38ac01a2eb05b31b1c4c245be66d2"
Jul 27 21:22:11.370278 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:22:11.370262    2577 scope.go:117] "RemoveContainer" containerID="0311f40d53afb2115388a398ac58232ab7b6bdbd82208f3a153a9e49230b2c56"
Jul 27 21:22:11.377523 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:22:11.377508    2577 scope.go:117] "RemoveContainer" containerID="1e8589ff158ebb00734f861c8608b3dfbb0abc5aef9c3a153b7d44fdae73308d"
Jul 27 21:22:11.384826 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:22:11.384812    2577 scope.go:117] "RemoveContainer" containerID="1f59ee999c494f9c27604d6eb4e1686e27dd0118d7331eb067714d134d7fc715"
Jul 27 21:22:11.391119 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:22:11.391098    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9"]
Jul 27 21:22:11.392174 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:22:11.392158    2577 scope.go:117] "RemoveContainer" containerID="3aa38ddf283fa88350d9e235f09c773a84e38ac01a2eb05b31b1c4c245be66d2"
Jul 27 21:22:11.392403 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:22:11.392385    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3aa38ddf283fa88350d9e235f09c773a84e38ac01a2eb05b31b1c4c245be66d2\": container with ID starting with 3aa38ddf283fa88350d9e235f09c773a84e38ac01a2eb05b31b1c4c245be66d2 not found: ID does not exist" containerID="3aa38ddf283fa88350d9e235f09c773a84e38ac01a2eb05b31b1c4c245be66d2"
Jul 27 21:22:11.392446 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:22:11.392412    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aa38ddf283fa88350d9e235f09c773a84e38ac01a2eb05b31b1c4c245be66d2"} err="failed to get container status \"3aa38ddf283fa88350d9e235f09c773a84e38ac01a2eb05b31b1c4c245be66d2\": rpc error: code = NotFound desc = could not find container \"3aa38ddf283fa88350d9e235f09c773a84e38ac01a2eb05b31b1c4c245be66d2\": container with ID starting with 3aa38ddf283fa88350d9e235f09c773a84e38ac01a2eb05b31b1c4c245be66d2 not found: ID does not exist"
Jul 27 21:22:11.392446 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:22:11.392432    2577 scope.go:117] "RemoveContainer" containerID="0311f40d53afb2115388a398ac58232ab7b6bdbd82208f3a153a9e49230b2c56"
Jul 27 21:22:11.392699 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:22:11.392681    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0311f40d53afb2115388a398ac58232ab7b6bdbd82208f3a153a9e49230b2c56\": container with ID starting with 0311f40d53afb2115388a398ac58232ab7b6bdbd82208f3a153a9e49230b2c56 not found: ID does not exist" containerID="0311f40d53afb2115388a398ac58232ab7b6bdbd82208f3a153a9e49230b2c56"
Jul 27 21:22:11.392755 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:22:11.392705    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0311f40d53afb2115388a398ac58232ab7b6bdbd82208f3a153a9e49230b2c56"} err="failed to get container status \"0311f40d53afb2115388a398ac58232ab7b6bdbd82208f3a153a9e49230b2c56\": rpc error: code = NotFound desc = could not find container \"0311f40d53afb2115388a398ac58232ab7b6bdbd82208f3a153a9e49230b2c56\": container with ID starting with 0311f40d53afb2115388a398ac58232ab7b6bdbd82208f3a153a9e49230b2c56 not found: ID does not exist"
Jul 27 21:22:11.392755 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:22:11.392721    2577 scope.go:117] "RemoveContainer" containerID="1e8589ff158ebb00734f861c8608b3dfbb0abc5aef9c3a153b7d44fdae73308d"
Jul 27 21:22:11.393033 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:22:11.392992    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e8589ff158ebb00734f861c8608b3dfbb0abc5aef9c3a153b7d44fdae73308d\": container with ID starting with 1e8589ff158ebb00734f861c8608b3dfbb0abc5aef9c3a153b7d44fdae73308d not found: ID does not exist" containerID="1e8589ff158ebb00734f861c8608b3dfbb0abc5aef9c3a153b7d44fdae73308d"
Jul 27 21:22:11.393033 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:22:11.393020    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e8589ff158ebb00734f861c8608b3dfbb0abc5aef9c3a153b7d44fdae73308d"} err="failed to get container status \"1e8589ff158ebb00734f861c8608b3dfbb0abc5aef9c3a153b7d44fdae73308d\": rpc error: code = NotFound desc = could not find container \"1e8589ff158ebb00734f861c8608b3dfbb0abc5aef9c3a153b7d44fdae73308d\": container with ID starting with 1e8589ff158ebb00734f861c8608b3dfbb0abc5aef9c3a153b7d44fdae73308d not found: ID does not exist"
Jul 27 21:22:11.393228 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:22:11.393041    2577 scope.go:117] "RemoveContainer" containerID="1f59ee999c494f9c27604d6eb4e1686e27dd0118d7331eb067714d134d7fc715"
Jul 27 21:22:11.393406 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:22:11.393384    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f59ee999c494f9c27604d6eb4e1686e27dd0118d7331eb067714d134d7fc715\": container with ID starting with 1f59ee999c494f9c27604d6eb4e1686e27dd0118d7331eb067714d134d7fc715 not found: ID does not exist" containerID="1f59ee999c494f9c27604d6eb4e1686e27dd0118d7331eb067714d134d7fc715"
Jul 27 21:22:11.393500 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:22:11.393413    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f59ee999c494f9c27604d6eb4e1686e27dd0118d7331eb067714d134d7fc715"} err="failed to get container status \"1f59ee999c494f9c27604d6eb4e1686e27dd0118d7331eb067714d134d7fc715\": rpc error: code = NotFound desc = could not find container \"1f59ee999c494f9c27604d6eb4e1686e27dd0118d7331eb067714d134d7fc715\": container with ID starting with 1f59ee999c494f9c27604d6eb4e1686e27dd0118d7331eb067714d134d7fc715 not found: ID does not exist"
Jul 27 21:22:11.395036 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:22:11.395015    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-njkq9"]
Jul 27 21:22:13.135606 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:22:13.135575    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f31527c-46ab-4f25-a0b8-4722b846b68f" path="/var/lib/kubelet/pods/3f31527c-46ab-4f25-a0b8-4722b846b68f/volumes"
Jul 27 21:23:38.729378 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:23:38.729230    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:24:20.318549 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:24:20.318449    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:24:38.997179 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:24:38.997144    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cbsds" event={"ID":"7e80f21b-5c24-4dd5-9bc7-a29706f4e282","Type":"ContainerStarted","Data":"c7150a53322730e50def37aa4704b772087f0c58b84073d1f70fd86943e305bd"}
Jul 27 21:24:38.997564 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:24:38.997241    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cbsds" event={"ID":"7e80f21b-5c24-4dd5-9bc7-a29706f4e282","Type":"ContainerStarted","Data":"22fd518e4a787a40ba91febb924e8b5e3ffe00e4560a717207cfde624d1f9259"}
Jul 27 21:24:38.997564 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:24:38.997314    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cbsds"
Jul 27 21:24:39.024049 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:24:39.024007    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cbsds" podStartSLOduration=2.661873141 podStartE2EDuration="2m59.023994526s" totalImagesPullingTime="2m27.161697167s" totalInitContainerRuntime="29.200424218s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:21:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:24:39.022179896 +0000 UTC m=+1202.389369763" watchObservedRunningTime="2026-07-27 21:24:39.023994526 +0000 UTC m=+1202.391184390"
Jul 27 21:24:40.001735 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:24:40.001701    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cbsds"
Jul 27 21:24:40.002873 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:24:40.002828    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cbsds" podUID="7e80f21b-5c24-4dd5-9bc7-a29706f4e282" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 27 21:24:41.005892 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:24:41.005851    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cbsds" podUID="7e80f21b-5c24-4dd5-9bc7-a29706f4e282" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 27 21:24:46.010460 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:24:46.010427    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cbsds"
Jul 27 21:24:46.011115 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:24:46.011088    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cbsds" podUID="7e80f21b-5c24-4dd5-9bc7-a29706f4e282" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 27 21:24:56.012416 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:24:56.012384    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cbsds"
Jul 27 21:25:06.696844 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:06.696813    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cbsds"]
Jul 27 21:25:06.697300 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:06.697064    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cbsds" podUID="7e80f21b-5c24-4dd5-9bc7-a29706f4e282" containerName="kserve-container" containerID="cri-o://22fd518e4a787a40ba91febb924e8b5e3ffe00e4560a717207cfde624d1f9259" gracePeriod=30
Jul 27 21:25:06.697300 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:06.697113    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cbsds" podUID="7e80f21b-5c24-4dd5-9bc7-a29706f4e282" containerName="kube-rbac-proxy" containerID="cri-o://c7150a53322730e50def37aa4704b772087f0c58b84073d1f70fd86943e305bd" gracePeriod=30
Jul 27 21:25:06.720385 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:06.720345    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:25:07.109007 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:07.108919    2577 generic.go:356] "Generic (PLEG): container finished" podID="7e80f21b-5c24-4dd5-9bc7-a29706f4e282" containerID="c7150a53322730e50def37aa4704b772087f0c58b84073d1f70fd86943e305bd" exitCode=2
Jul 27 21:25:07.109007 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:07.108963    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cbsds" event={"ID":"7e80f21b-5c24-4dd5-9bc7-a29706f4e282","Type":"ContainerDied","Data":"c7150a53322730e50def37aa4704b772087f0c58b84073d1f70fd86943e305bd"}
Jul 27 21:25:07.970567 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:07.970539    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cbsds"
Jul 27 21:25:08.086843 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:08.086753    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e80f21b-5c24-4dd5-9bc7-a29706f4e282-proxy-tls\") pod \"7e80f21b-5c24-4dd5-9bc7-a29706f4e282\" (UID: \"7e80f21b-5c24-4dd5-9bc7-a29706f4e282\") "
Jul 27 21:25:08.086843 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:08.086809    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e80f21b-5c24-4dd5-9bc7-a29706f4e282-kube-rbac-proxy-sar-config\") pod \"7e80f21b-5c24-4dd5-9bc7-a29706f4e282\" (UID: \"7e80f21b-5c24-4dd5-9bc7-a29706f4e282\") "
Jul 27 21:25:08.086843 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:08.086839    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e80f21b-5c24-4dd5-9bc7-a29706f4e282-kserve-provision-location\") pod \"7e80f21b-5c24-4dd5-9bc7-a29706f4e282\" (UID: \"7e80f21b-5c24-4dd5-9bc7-a29706f4e282\") "
Jul 27 21:25:08.087146 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:08.086885    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e80f21b-5c24-4dd5-9bc7-a29706f4e282-cabundle-cert\") pod \"7e80f21b-5c24-4dd5-9bc7-a29706f4e282\" (UID: \"7e80f21b-5c24-4dd5-9bc7-a29706f4e282\") "
Jul 27 21:25:08.087146 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:08.087021    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pzmbx\" (UniqueName: \"kubernetes.io/projected/7e80f21b-5c24-4dd5-9bc7-a29706f4e282-kube-api-access-pzmbx\") pod \"7e80f21b-5c24-4dd5-9bc7-a29706f4e282\" (UID: \"7e80f21b-5c24-4dd5-9bc7-a29706f4e282\") "
Jul 27 21:25:08.087245 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:08.087205    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e80f21b-5c24-4dd5-9bc7-a29706f4e282-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7e80f21b-5c24-4dd5-9bc7-a29706f4e282" (UID: "7e80f21b-5c24-4dd5-9bc7-a29706f4e282"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:25:08.087245 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:08.087217    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e80f21b-5c24-4dd5-9bc7-a29706f4e282-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7e80f21b-5c24-4dd5-9bc7-a29706f4e282" (UID: "7e80f21b-5c24-4dd5-9bc7-a29706f4e282"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:25:08.087345 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:08.087296    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7e80f21b-5c24-4dd5-9bc7-a29706f4e282-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:25:08.087345 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:08.087312    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e80f21b-5c24-4dd5-9bc7-a29706f4e282-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:25:08.088878 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:08.088858    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e80f21b-5c24-4dd5-9bc7-a29706f4e282-kube-api-access-pzmbx" (OuterVolumeSpecName: "kube-api-access-pzmbx") pod "7e80f21b-5c24-4dd5-9bc7-a29706f4e282" (UID: "7e80f21b-5c24-4dd5-9bc7-a29706f4e282"). InnerVolumeSpecName "kube-api-access-pzmbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:25:08.089060 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:08.089045    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e80f21b-5c24-4dd5-9bc7-a29706f4e282-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7e80f21b-5c24-4dd5-9bc7-a29706f4e282" (UID: "7e80f21b-5c24-4dd5-9bc7-a29706f4e282"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:25:08.092311 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:08.092285    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e80f21b-5c24-4dd5-9bc7-a29706f4e282-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7e80f21b-5c24-4dd5-9bc7-a29706f4e282" (UID: "7e80f21b-5c24-4dd5-9bc7-a29706f4e282"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:25:08.114704 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:08.114676    2577 generic.go:356] "Generic (PLEG): container finished" podID="7e80f21b-5c24-4dd5-9bc7-a29706f4e282" containerID="22fd518e4a787a40ba91febb924e8b5e3ffe00e4560a717207cfde624d1f9259" exitCode=0
Jul 27 21:25:08.114815 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:08.114723    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cbsds"
Jul 27 21:25:08.114815 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:08.114771    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cbsds" event={"ID":"7e80f21b-5c24-4dd5-9bc7-a29706f4e282","Type":"ContainerDied","Data":"22fd518e4a787a40ba91febb924e8b5e3ffe00e4560a717207cfde624d1f9259"}
Jul 27 21:25:08.114898 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:08.114827    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cbsds" event={"ID":"7e80f21b-5c24-4dd5-9bc7-a29706f4e282","Type":"ContainerDied","Data":"1a7f294469b526c5d9428c94aec6c947614ca0ab6442c93c2e46acbbe4eee6c5"}
Jul 27 21:25:08.114898 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:08.114853    2577 scope.go:117] "RemoveContainer" containerID="c7150a53322730e50def37aa4704b772087f0c58b84073d1f70fd86943e305bd"
Jul 27 21:25:08.122503 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:08.122484    2577 scope.go:117] "RemoveContainer" containerID="22fd518e4a787a40ba91febb924e8b5e3ffe00e4560a717207cfde624d1f9259"
Jul 27 21:25:08.129403 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:08.129386    2577 scope.go:117] "RemoveContainer" containerID="e13c7ab69f3b62a278edeef5459bd81ccf20071bebf13176a46106510e5c0333"
Jul 27 21:25:08.135386 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:08.135366    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cbsds"]
Jul 27 21:25:08.136240 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:08.136226    2577 scope.go:117] "RemoveContainer" containerID="c7150a53322730e50def37aa4704b772087f0c58b84073d1f70fd86943e305bd"
Jul 27 21:25:08.136503 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:25:08.136486    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7150a53322730e50def37aa4704b772087f0c58b84073d1f70fd86943e305bd\": container with ID starting with c7150a53322730e50def37aa4704b772087f0c58b84073d1f70fd86943e305bd not found: ID does not exist" containerID="c7150a53322730e50def37aa4704b772087f0c58b84073d1f70fd86943e305bd"
Jul 27 21:25:08.136548 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:08.136510    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7150a53322730e50def37aa4704b772087f0c58b84073d1f70fd86943e305bd"} err="failed to get container status \"c7150a53322730e50def37aa4704b772087f0c58b84073d1f70fd86943e305bd\": rpc error: code = NotFound desc = could not find container \"c7150a53322730e50def37aa4704b772087f0c58b84073d1f70fd86943e305bd\": container with ID starting with c7150a53322730e50def37aa4704b772087f0c58b84073d1f70fd86943e305bd not found: ID does not exist"
Jul 27 21:25:08.136548 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:08.136528    2577 scope.go:117] "RemoveContainer" containerID="22fd518e4a787a40ba91febb924e8b5e3ffe00e4560a717207cfde624d1f9259"
Jul 27 21:25:08.136756 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:25:08.136742    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22fd518e4a787a40ba91febb924e8b5e3ffe00e4560a717207cfde624d1f9259\": container with ID starting with 22fd518e4a787a40ba91febb924e8b5e3ffe00e4560a717207cfde624d1f9259 not found: ID does not exist" containerID="22fd518e4a787a40ba91febb924e8b5e3ffe00e4560a717207cfde624d1f9259"
Jul 27 21:25:08.136813 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:08.136774    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22fd518e4a787a40ba91febb924e8b5e3ffe00e4560a717207cfde624d1f9259"} err="failed to get container status \"22fd518e4a787a40ba91febb924e8b5e3ffe00e4560a717207cfde624d1f9259\": rpc error: code = NotFound desc = could not find container \"22fd518e4a787a40ba91febb924e8b5e3ffe00e4560a717207cfde624d1f9259\": container with ID starting with 22fd518e4a787a40ba91febb924e8b5e3ffe00e4560a717207cfde624d1f9259 not found: ID does not exist"
Jul 27 21:25:08.136813 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:08.136789    2577 scope.go:117] "RemoveContainer" containerID="e13c7ab69f3b62a278edeef5459bd81ccf20071bebf13176a46106510e5c0333"
Jul 27 21:25:08.137004 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:25:08.136988    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e13c7ab69f3b62a278edeef5459bd81ccf20071bebf13176a46106510e5c0333\": container with ID starting with e13c7ab69f3b62a278edeef5459bd81ccf20071bebf13176a46106510e5c0333 not found: ID does not exist" containerID="e13c7ab69f3b62a278edeef5459bd81ccf20071bebf13176a46106510e5c0333"
Jul 27 21:25:08.137043 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:08.137008    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e13c7ab69f3b62a278edeef5459bd81ccf20071bebf13176a46106510e5c0333"} err="failed to get container status \"e13c7ab69f3b62a278edeef5459bd81ccf20071bebf13176a46106510e5c0333\": rpc error: code = NotFound desc = could not find container \"e13c7ab69f3b62a278edeef5459bd81ccf20071bebf13176a46106510e5c0333\": container with ID starting with e13c7ab69f3b62a278edeef5459bd81ccf20071bebf13176a46106510e5c0333 not found: ID does not exist"
Jul 27 21:25:08.139493 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:08.139474    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-cbsds"]
Jul 27 21:25:08.187939 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:08.187919    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e80f21b-5c24-4dd5-9bc7-a29706f4e282-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:25:08.187939 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:08.187938    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e80f21b-5c24-4dd5-9bc7-a29706f4e282-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:25:08.188094 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:08.187947    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzmbx\" (UniqueName: \"kubernetes.io/projected/7e80f21b-5c24-4dd5-9bc7-a29706f4e282-kube-api-access-pzmbx\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:25:09.139345 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:09.139315    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e80f21b-5c24-4dd5-9bc7-a29706f4e282" path="/var/lib/kubelet/pods/7e80f21b-5c24-4dd5-9bc7-a29706f4e282/volumes"
Jul 27 21:25:11.804812 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:11.804779    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:25:11.854009 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:11.853977    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zbnrj"]
Jul 27 21:25:11.854288 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:11.854275    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6343eda-b4e9-4058-a547-318b2ebf6d90" containerName="kserve-container"
Jul 27 21:25:11.854342 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:11.854290    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6343eda-b4e9-4058-a547-318b2ebf6d90" containerName="kserve-container"
Jul 27 21:25:11.854342 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:11.854300    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f31527c-46ab-4f25-a0b8-4722b846b68f" containerName="kube-rbac-proxy"
Jul 27 21:25:11.854342 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:11.854306    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f31527c-46ab-4f25-a0b8-4722b846b68f" containerName="kube-rbac-proxy"
Jul 27 21:25:11.854342 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:11.854315    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e80f21b-5c24-4dd5-9bc7-a29706f4e282" containerName="storage-initializer"
Jul 27 21:25:11.854342 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:11.854321    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e80f21b-5c24-4dd5-9bc7-a29706f4e282" containerName="storage-initializer"
Jul 27 21:25:11.854342 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:11.854329    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f31527c-46ab-4f25-a0b8-4722b846b68f" containerName="storage-initializer"
Jul 27 21:25:11.854342 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:11.854334    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f31527c-46ab-4f25-a0b8-4722b846b68f" containerName="storage-initializer"
Jul 27 21:25:11.854342 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:11.854339    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e80f21b-5c24-4dd5-9bc7-a29706f4e282" containerName="kube-rbac-proxy"
Jul 27 21:25:11.854342 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:11.854346    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e80f21b-5c24-4dd5-9bc7-a29706f4e282" containerName="kube-rbac-proxy"
Jul 27 21:25:11.854663 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:11.854353    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6343eda-b4e9-4058-a547-318b2ebf6d90" containerName="kube-rbac-proxy"
Jul 27 21:25:11.854663 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:11.854358    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6343eda-b4e9-4058-a547-318b2ebf6d90" containerName="kube-rbac-proxy"
Jul 27 21:25:11.854663 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:11.854365    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f31527c-46ab-4f25-a0b8-4722b846b68f" containerName="kserve-container"
Jul 27 21:25:11.854663 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:11.854370    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f31527c-46ab-4f25-a0b8-4722b846b68f" containerName="kserve-container"
Jul 27 21:25:11.854663 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:11.854378    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f31527c-46ab-4f25-a0b8-4722b846b68f" containerName="agent"
Jul 27 21:25:11.854663 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:11.854383    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f31527c-46ab-4f25-a0b8-4722b846b68f" containerName="agent"
Jul 27 21:25:11.854663 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:11.854391    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e80f21b-5c24-4dd5-9bc7-a29706f4e282" containerName="kserve-container"
Jul 27 21:25:11.854663 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:11.854395    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e80f21b-5c24-4dd5-9bc7-a29706f4e282" containerName="kserve-container"
Jul 27 21:25:11.854663 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:11.854454    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f31527c-46ab-4f25-a0b8-4722b846b68f" containerName="kserve-container"
Jul 27 21:25:11.854663 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:11.854462    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6343eda-b4e9-4058-a547-318b2ebf6d90" containerName="kube-rbac-proxy"
Jul 27 21:25:11.854663 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:11.854470    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f31527c-46ab-4f25-a0b8-4722b846b68f" containerName="kube-rbac-proxy"
Jul 27 21:25:11.854663 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:11.854476    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f31527c-46ab-4f25-a0b8-4722b846b68f" containerName="agent"
Jul 27 21:25:11.854663 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:11.854481    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e80f21b-5c24-4dd5-9bc7-a29706f4e282" containerName="kube-rbac-proxy"
Jul 27 21:25:11.854663 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:11.854487    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e80f21b-5c24-4dd5-9bc7-a29706f4e282" containerName="kserve-container"
Jul 27 21:25:11.854663 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:11.854492    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6343eda-b4e9-4058-a547-318b2ebf6d90" containerName="kserve-container"
Jul 27 21:25:11.877315 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:11.877286    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zbnrj"]
Jul 27 21:25:11.877460 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:11.877392    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zbnrj"
Jul 27 21:25:11.879743 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:11.879724    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Jul 27 21:25:11.879875 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:11.879842    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 21:25:11.879953 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:11.879939    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v7c66\""
Jul 27 21:25:11.880040 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:11.880023    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 21:25:11.880546 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:11.880528    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Jul 27 21:25:11.880618 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:11.880562    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 21:25:11.880618 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:11.880585    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 21:25:11.898380 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:11.898355    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:25:12.014311 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:12.014289    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a66baeb-9cdf-496f-bfbd-e92ff74f79d6-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-zbnrj\" (UID: \"7a66baeb-9cdf-496f-bfbd-e92ff74f79d6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zbnrj"
Jul 27 21:25:12.014402 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:12.014316    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blj58\" (UniqueName: \"kubernetes.io/projected/7a66baeb-9cdf-496f-bfbd-e92ff74f79d6-kube-api-access-blj58\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-zbnrj\" (UID: \"7a66baeb-9cdf-496f-bfbd-e92ff74f79d6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zbnrj"
Jul 27 21:25:12.014402 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:12.014343    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a66baeb-9cdf-496f-bfbd-e92ff74f79d6-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-zbnrj\" (UID: \"7a66baeb-9cdf-496f-bfbd-e92ff74f79d6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zbnrj"
Jul 27 21:25:12.014471 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:12.014401    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a66baeb-9cdf-496f-bfbd-e92ff74f79d6-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-zbnrj\" (UID: \"7a66baeb-9cdf-496f-bfbd-e92ff74f79d6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zbnrj"
Jul 27 21:25:12.014471 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:12.014429    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a66baeb-9cdf-496f-bfbd-e92ff74f79d6-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-zbnrj\" (UID: \"7a66baeb-9cdf-496f-bfbd-e92ff74f79d6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zbnrj"
Jul 27 21:25:12.115752 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:12.115699    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a66baeb-9cdf-496f-bfbd-e92ff74f79d6-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-zbnrj\" (UID: \"7a66baeb-9cdf-496f-bfbd-e92ff74f79d6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zbnrj"
Jul 27 21:25:12.115752 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:12.115734    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a66baeb-9cdf-496f-bfbd-e92ff74f79d6-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-zbnrj\" (UID: \"7a66baeb-9cdf-496f-bfbd-e92ff74f79d6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zbnrj"
Jul 27 21:25:12.115886 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:12.115781    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a66baeb-9cdf-496f-bfbd-e92ff74f79d6-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-zbnrj\" (UID: \"7a66baeb-9cdf-496f-bfbd-e92ff74f79d6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zbnrj"
Jul 27 21:25:12.115886 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:12.115810    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blj58\" (UniqueName: \"kubernetes.io/projected/7a66baeb-9cdf-496f-bfbd-e92ff74f79d6-kube-api-access-blj58\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-zbnrj\" (UID: \"7a66baeb-9cdf-496f-bfbd-e92ff74f79d6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zbnrj"
Jul 27 21:25:12.115886 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:12.115849    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a66baeb-9cdf-496f-bfbd-e92ff74f79d6-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-zbnrj\" (UID: \"7a66baeb-9cdf-496f-bfbd-e92ff74f79d6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zbnrj"
Jul 27 21:25:12.116252 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:12.116233    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a66baeb-9cdf-496f-bfbd-e92ff74f79d6-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-zbnrj\" (UID: \"7a66baeb-9cdf-496f-bfbd-e92ff74f79d6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zbnrj"
Jul 27 21:25:12.116460 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:12.116445    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a66baeb-9cdf-496f-bfbd-e92ff74f79d6-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-zbnrj\" (UID: \"7a66baeb-9cdf-496f-bfbd-e92ff74f79d6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zbnrj"
Jul 27 21:25:12.116494 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:12.116454    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a66baeb-9cdf-496f-bfbd-e92ff74f79d6-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-zbnrj\" (UID: \"7a66baeb-9cdf-496f-bfbd-e92ff74f79d6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zbnrj"
Jul 27 21:25:12.118084 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:12.118064    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a66baeb-9cdf-496f-bfbd-e92ff74f79d6-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-zbnrj\" (UID: \"7a66baeb-9cdf-496f-bfbd-e92ff74f79d6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zbnrj"
Jul 27 21:25:12.124775 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:12.124744    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blj58\" (UniqueName: \"kubernetes.io/projected/7a66baeb-9cdf-496f-bfbd-e92ff74f79d6-kube-api-access-blj58\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-zbnrj\" (UID: \"7a66baeb-9cdf-496f-bfbd-e92ff74f79d6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zbnrj"
Jul 27 21:25:12.187264 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:12.187246    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zbnrj"
Jul 27 21:25:12.328865 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:12.328793    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zbnrj"]
Jul 27 21:25:12.331039 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:25:12.331011    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a66baeb_9cdf_496f_bfbd_e92ff74f79d6.slice/crio-95ed8a0194d42117c7d6fe4ba8f5b6b6326305f239e2e6e1481dab0657c04ced WatchSource:0}: Error finding container 95ed8a0194d42117c7d6fe4ba8f5b6b6326305f239e2e6e1481dab0657c04ced: Status 404 returned error can't find the container with id 95ed8a0194d42117c7d6fe4ba8f5b6b6326305f239e2e6e1481dab0657c04ced
Jul 27 21:25:13.141962 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:13.141920    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zbnrj" event={"ID":"7a66baeb-9cdf-496f-bfbd-e92ff74f79d6","Type":"ContainerStarted","Data":"98c676d4a781b01f9106154223590b7ee2208ef7f350f4af3aab7283a61b18db"}
Jul 27 21:25:13.142295 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:13.142009    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zbnrj" event={"ID":"7a66baeb-9cdf-496f-bfbd-e92ff74f79d6","Type":"ContainerStarted","Data":"95ed8a0194d42117c7d6fe4ba8f5b6b6326305f239e2e6e1481dab0657c04ced"}
Jul 27 21:25:17.157490 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:17.157462    2577 generic.go:356] "Generic (PLEG): container finished" podID="7a66baeb-9cdf-496f-bfbd-e92ff74f79d6" containerID="98c676d4a781b01f9106154223590b7ee2208ef7f350f4af3aab7283a61b18db" exitCode=0
Jul 27 21:25:17.157900 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:17.157509    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zbnrj" event={"ID":"7a66baeb-9cdf-496f-bfbd-e92ff74f79d6","Type":"ContainerDied","Data":"98c676d4a781b01f9106154223590b7ee2208ef7f350f4af3aab7283a61b18db"}
Jul 27 21:25:18.163692 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:18.163658    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zbnrj" event={"ID":"7a66baeb-9cdf-496f-bfbd-e92ff74f79d6","Type":"ContainerStarted","Data":"b77b1f483ff88c26281e23763d161acd23d915b7818856f004700d0292772b8b"}
Jul 27 21:25:18.164095 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:18.163714    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zbnrj" event={"ID":"7a66baeb-9cdf-496f-bfbd-e92ff74f79d6","Type":"ContainerStarted","Data":"01e94376c694268f4a0607a035c1711e3aef32f17236407d61fa3f525544f168"}
Jul 27 21:25:18.164095 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:18.164061    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zbnrj"
Jul 27 21:25:18.164175 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:18.164112    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zbnrj"
Jul 27 21:25:18.184287 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:18.184243    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zbnrj" podStartSLOduration=2.580312152 podStartE2EDuration="7.184228024s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.603915872s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:25:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:25:18.182851642 +0000 UTC m=+1241.550041513" watchObservedRunningTime="2026-07-27 21:25:18.184228024 +0000 UTC m=+1241.551417888"
Jul 27 21:25:24.172934 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:25:24.172899    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zbnrj"
Jul 27 21:26:49.135392 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:26:49.135365    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zbnrj"
Jul 27 21:26:58.135671 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:26:58.135635    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zbnrj"]
Jul 27 21:26:58.136157 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:26:58.135869    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zbnrj" podUID="7a66baeb-9cdf-496f-bfbd-e92ff74f79d6" containerName="kserve-container" containerID="cri-o://01e94376c694268f4a0607a035c1711e3aef32f17236407d61fa3f525544f168" gracePeriod=30
Jul 27 21:26:58.136157 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:26:58.135911    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zbnrj" podUID="7a66baeb-9cdf-496f-bfbd-e92ff74f79d6" containerName="kube-rbac-proxy" containerID="cri-o://b77b1f483ff88c26281e23763d161acd23d915b7818856f004700d0292772b8b" gracePeriod=30
Jul 27 21:26:58.196033 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:26:58.195998    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:58.537873 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:26:58.537842    2577 generic.go:356] "Generic (PLEG): container finished" podID="7a66baeb-9cdf-496f-bfbd-e92ff74f79d6" containerID="b77b1f483ff88c26281e23763d161acd23d915b7818856f004700d0292772b8b" exitCode=2
Jul 27 21:26:58.538058 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:26:58.537887    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zbnrj" event={"ID":"7a66baeb-9cdf-496f-bfbd-e92ff74f79d6","Type":"ContainerDied","Data":"b77b1f483ff88c26281e23763d161acd23d915b7818856f004700d0292772b8b"}
Jul 27 21:26:59.130840 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:26:59.130801    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zbnrj" podUID="7a66baeb-9cdf-496f-bfbd-e92ff74f79d6" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.30:8080/v2/models/isvc-autogluon-v2/ready\": dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 27 21:26:59.168285 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:26:59.168248    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zbnrj" podUID="7a66baeb-9cdf-496f-bfbd-e92ff74f79d6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 27 21:26:59.474343 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:26:59.474319    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zbnrj"
Jul 27 21:26:59.509054 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:26:59.509031    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-blj58\" (UniqueName: \"kubernetes.io/projected/7a66baeb-9cdf-496f-bfbd-e92ff74f79d6-kube-api-access-blj58\") pod \"7a66baeb-9cdf-496f-bfbd-e92ff74f79d6\" (UID: \"7a66baeb-9cdf-496f-bfbd-e92ff74f79d6\") "
Jul 27 21:26:59.509167 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:26:59.509067    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a66baeb-9cdf-496f-bfbd-e92ff74f79d6-proxy-tls\") pod \"7a66baeb-9cdf-496f-bfbd-e92ff74f79d6\" (UID: \"7a66baeb-9cdf-496f-bfbd-e92ff74f79d6\") "
Jul 27 21:26:59.509167 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:26:59.509147    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a66baeb-9cdf-496f-bfbd-e92ff74f79d6-kserve-provision-location\") pod \"7a66baeb-9cdf-496f-bfbd-e92ff74f79d6\" (UID: \"7a66baeb-9cdf-496f-bfbd-e92ff74f79d6\") "
Jul 27 21:26:59.509283 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:26:59.509173    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a66baeb-9cdf-496f-bfbd-e92ff74f79d6-cabundle-cert\") pod \"7a66baeb-9cdf-496f-bfbd-e92ff74f79d6\" (UID: \"7a66baeb-9cdf-496f-bfbd-e92ff74f79d6\") "
Jul 27 21:26:59.509283 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:26:59.509204    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a66baeb-9cdf-496f-bfbd-e92ff74f79d6-kube-rbac-proxy-sar-config\") pod \"7a66baeb-9cdf-496f-bfbd-e92ff74f79d6\" (UID: \"7a66baeb-9cdf-496f-bfbd-e92ff74f79d6\") "
Jul 27 21:26:59.509626 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:26:59.509589    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a66baeb-9cdf-496f-bfbd-e92ff74f79d6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7a66baeb-9cdf-496f-bfbd-e92ff74f79d6" (UID: "7a66baeb-9cdf-496f-bfbd-e92ff74f79d6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:26:59.509734 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:26:59.509636    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a66baeb-9cdf-496f-bfbd-e92ff74f79d6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7a66baeb-9cdf-496f-bfbd-e92ff74f79d6" (UID: "7a66baeb-9cdf-496f-bfbd-e92ff74f79d6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:26:59.511069 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:26:59.511036    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a66baeb-9cdf-496f-bfbd-e92ff74f79d6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7a66baeb-9cdf-496f-bfbd-e92ff74f79d6" (UID: "7a66baeb-9cdf-496f-bfbd-e92ff74f79d6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:26:59.511559 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:26:59.511543    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a66baeb-9cdf-496f-bfbd-e92ff74f79d6-kube-api-access-blj58" (OuterVolumeSpecName: "kube-api-access-blj58") pod "7a66baeb-9cdf-496f-bfbd-e92ff74f79d6" (UID: "7a66baeb-9cdf-496f-bfbd-e92ff74f79d6"). InnerVolumeSpecName "kube-api-access-blj58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:26:59.514147 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:26:59.514122    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a66baeb-9cdf-496f-bfbd-e92ff74f79d6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7a66baeb-9cdf-496f-bfbd-e92ff74f79d6" (UID: "7a66baeb-9cdf-496f-bfbd-e92ff74f79d6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:26:59.543893 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:26:59.543868    2577 generic.go:356] "Generic (PLEG): container finished" podID="7a66baeb-9cdf-496f-bfbd-e92ff74f79d6" containerID="01e94376c694268f4a0607a035c1711e3aef32f17236407d61fa3f525544f168" exitCode=0
Jul 27 21:26:59.544004 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:26:59.543935    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zbnrj"
Jul 27 21:26:59.544004 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:26:59.543962    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zbnrj" event={"ID":"7a66baeb-9cdf-496f-bfbd-e92ff74f79d6","Type":"ContainerDied","Data":"01e94376c694268f4a0607a035c1711e3aef32f17236407d61fa3f525544f168"}
Jul 27 21:26:59.544114 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:26:59.544033    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zbnrj" event={"ID":"7a66baeb-9cdf-496f-bfbd-e92ff74f79d6","Type":"ContainerDied","Data":"95ed8a0194d42117c7d6fe4ba8f5b6b6326305f239e2e6e1481dab0657c04ced"}
Jul 27 21:26:59.544114 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:26:59.544074    2577 scope.go:117] "RemoveContainer" containerID="b77b1f483ff88c26281e23763d161acd23d915b7818856f004700d0292772b8b"
Jul 27 21:26:59.551856 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:26:59.551831    2577 scope.go:117] "RemoveContainer" containerID="01e94376c694268f4a0607a035c1711e3aef32f17236407d61fa3f525544f168"
Jul 27 21:26:59.558946 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:26:59.558930    2577 scope.go:117] "RemoveContainer" containerID="98c676d4a781b01f9106154223590b7ee2208ef7f350f4af3aab7283a61b18db"
Jul 27 21:26:59.565576 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:26:59.565557    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zbnrj"]
Jul 27 21:26:59.565977 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:26:59.565959    2577 scope.go:117] "RemoveContainer" containerID="b77b1f483ff88c26281e23763d161acd23d915b7818856f004700d0292772b8b"
Jul 27 21:26:59.566230 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:26:59.566210    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b77b1f483ff88c26281e23763d161acd23d915b7818856f004700d0292772b8b\": container with ID starting with b77b1f483ff88c26281e23763d161acd23d915b7818856f004700d0292772b8b not found: ID does not exist" containerID="b77b1f483ff88c26281e23763d161acd23d915b7818856f004700d0292772b8b"
Jul 27 21:26:59.566282 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:26:59.566240    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b77b1f483ff88c26281e23763d161acd23d915b7818856f004700d0292772b8b"} err="failed to get container status \"b77b1f483ff88c26281e23763d161acd23d915b7818856f004700d0292772b8b\": rpc error: code = NotFound desc = could not find container \"b77b1f483ff88c26281e23763d161acd23d915b7818856f004700d0292772b8b\": container with ID starting with b77b1f483ff88c26281e23763d161acd23d915b7818856f004700d0292772b8b not found: ID does not exist"
Jul 27 21:26:59.566282 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:26:59.566257    2577 scope.go:117] "RemoveContainer" containerID="01e94376c694268f4a0607a035c1711e3aef32f17236407d61fa3f525544f168"
Jul 27 21:26:59.566523 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:26:59.566508    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01e94376c694268f4a0607a035c1711e3aef32f17236407d61fa3f525544f168\": container with ID starting with 01e94376c694268f4a0607a035c1711e3aef32f17236407d61fa3f525544f168 not found: ID does not exist" containerID="01e94376c694268f4a0607a035c1711e3aef32f17236407d61fa3f525544f168"
Jul 27 21:26:59.566573 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:26:59.566530    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01e94376c694268f4a0607a035c1711e3aef32f17236407d61fa3f525544f168"} err="failed to get container status \"01e94376c694268f4a0607a035c1711e3aef32f17236407d61fa3f525544f168\": rpc error: code = NotFound desc = could not find container \"01e94376c694268f4a0607a035c1711e3aef32f17236407d61fa3f525544f168\": container with ID starting with 01e94376c694268f4a0607a035c1711e3aef32f17236407d61fa3f525544f168 not found: ID does not exist"
Jul 27 21:26:59.566573 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:26:59.566543    2577 scope.go:117] "RemoveContainer" containerID="98c676d4a781b01f9106154223590b7ee2208ef7f350f4af3aab7283a61b18db"
Jul 27 21:26:59.566792 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:26:59.566752    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98c676d4a781b01f9106154223590b7ee2208ef7f350f4af3aab7283a61b18db\": container with ID starting with 98c676d4a781b01f9106154223590b7ee2208ef7f350f4af3aab7283a61b18db not found: ID does not exist" containerID="98c676d4a781b01f9106154223590b7ee2208ef7f350f4af3aab7283a61b18db"
Jul 27 21:26:59.566898 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:26:59.566801    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98c676d4a781b01f9106154223590b7ee2208ef7f350f4af3aab7283a61b18db"} err="failed to get container status \"98c676d4a781b01f9106154223590b7ee2208ef7f350f4af3aab7283a61b18db\": rpc error: code = NotFound desc = could not find container \"98c676d4a781b01f9106154223590b7ee2208ef7f350f4af3aab7283a61b18db\": container with ID starting with 98c676d4a781b01f9106154223590b7ee2208ef7f350f4af3aab7283a61b18db not found: ID does not exist"
Jul 27 21:26:59.568808 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:26:59.568788    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-zbnrj"]
Jul 27 21:26:59.609852 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:26:59.609834    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blj58\" (UniqueName: \"kubernetes.io/projected/7a66baeb-9cdf-496f-bfbd-e92ff74f79d6-kube-api-access-blj58\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:26:59.609940 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:26:59.609855    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a66baeb-9cdf-496f-bfbd-e92ff74f79d6-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:26:59.609940 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:26:59.609871    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a66baeb-9cdf-496f-bfbd-e92ff74f79d6-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:26:59.609940 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:26:59.609884    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a66baeb-9cdf-496f-bfbd-e92ff74f79d6-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:26:59.609940 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:26:59.609897    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a66baeb-9cdf-496f-bfbd-e92ff74f79d6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:27:01.135125 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:27:01.135090    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a66baeb-9cdf-496f-bfbd-e92ff74f79d6" path="/var/lib/kubelet/pods/7a66baeb-9cdf-496f-bfbd-e92ff74f79d6/volumes"
Jul 27 21:27:03.246962 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:27:03.246928    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:27:03.302656 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:27:03.302624    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-szvpd"]
Jul 27 21:27:03.302935 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:27:03.302922    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a66baeb-9cdf-496f-bfbd-e92ff74f79d6" containerName="storage-initializer"
Jul 27 21:27:03.302983 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:27:03.302936    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a66baeb-9cdf-496f-bfbd-e92ff74f79d6" containerName="storage-initializer"
Jul 27 21:27:03.302983 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:27:03.302949    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a66baeb-9cdf-496f-bfbd-e92ff74f79d6" containerName="kserve-container"
Jul 27 21:27:03.302983 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:27:03.302954    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a66baeb-9cdf-496f-bfbd-e92ff74f79d6" containerName="kserve-container"
Jul 27 21:27:03.302983 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:27:03.302963    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a66baeb-9cdf-496f-bfbd-e92ff74f79d6" containerName="kube-rbac-proxy"
Jul 27 21:27:03.302983 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:27:03.302968    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a66baeb-9cdf-496f-bfbd-e92ff74f79d6" containerName="kube-rbac-proxy"
Jul 27 21:27:03.303129 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:27:03.303013    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a66baeb-9cdf-496f-bfbd-e92ff74f79d6" containerName="kube-rbac-proxy"
Jul 27 21:27:03.303129 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:27:03.303020    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a66baeb-9cdf-496f-bfbd-e92ff74f79d6" containerName="kserve-container"
Jul 27 21:27:03.305966 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:27:03.305951    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-szvpd"
Jul 27 21:27:03.308415 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:27:03.308393    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 21:27:03.308678 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:27:03.308654    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Jul 27 21:27:03.308798 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:27:03.308729    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Jul 27 21:27:03.308899 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:27:03.308797    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 21:27:03.308972 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:27:03.308933    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 21:27:03.308972 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:27:03.308933    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 21:27:03.309085 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:27:03.308995    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v7c66\""
Jul 27 21:27:03.318000 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:27:03.317981    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-szvpd"]
Jul 27 21:27:03.339644 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:27:03.339624    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/760f8de6-914d-42f7-a40f-a66ff4f95700-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-szvpd\" (UID: \"760f8de6-914d-42f7-a40f-a66ff4f95700\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-szvpd"
Jul 27 21:27:03.339733 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:27:03.339654    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/760f8de6-914d-42f7-a40f-a66ff4f95700-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-szvpd\" (UID: \"760f8de6-914d-42f7-a40f-a66ff4f95700\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-szvpd"
Jul 27 21:27:03.339733 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:27:03.339693    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/760f8de6-914d-42f7-a40f-a66ff4f95700-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-szvpd\" (UID: \"760f8de6-914d-42f7-a40f-a66ff4f95700\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-szvpd"
Jul 27 21:27:03.339833 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:27:03.339783    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/760f8de6-914d-42f7-a40f-a66ff4f95700-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-szvpd\" (UID: \"760f8de6-914d-42f7-a40f-a66ff4f95700\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-szvpd"
Jul 27 21:27:03.339870 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:27:03.339833    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f84jx\" (UniqueName: \"kubernetes.io/projected/760f8de6-914d-42f7-a40f-a66ff4f95700-kube-api-access-f84jx\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-szvpd\" (UID: \"760f8de6-914d-42f7-a40f-a66ff4f95700\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-szvpd"
Jul 27 21:27:03.441121 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:27:03.441099    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f84jx\" (UniqueName: \"kubernetes.io/projected/760f8de6-914d-42f7-a40f-a66ff4f95700-kube-api-access-f84jx\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-szvpd\" (UID: \"760f8de6-914d-42f7-a40f-a66ff4f95700\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-szvpd"
Jul 27 21:27:03.441241 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:27:03.441134    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/760f8de6-914d-42f7-a40f-a66ff4f95700-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-szvpd\" (UID: \"760f8de6-914d-42f7-a40f-a66ff4f95700\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-szvpd"
Jul 27 21:27:03.441241 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:27:03.441154    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/760f8de6-914d-42f7-a40f-a66ff4f95700-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-szvpd\" (UID: \"760f8de6-914d-42f7-a40f-a66ff4f95700\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-szvpd"
Jul 27 21:27:03.441366 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:27:03.441342    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/760f8de6-914d-42f7-a40f-a66ff4f95700-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-szvpd\" (UID: \"760f8de6-914d-42f7-a40f-a66ff4f95700\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-szvpd"
Jul 27 21:27:03.441425 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:27:03.441388    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/760f8de6-914d-42f7-a40f-a66ff4f95700-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-szvpd\" (UID: \"760f8de6-914d-42f7-a40f-a66ff4f95700\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-szvpd"
Jul 27 21:27:03.441522 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:27:03.441508    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-serving-cert: secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found
Jul 27 21:27:03.441573 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:27:03.441565    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/760f8de6-914d-42f7-a40f-a66ff4f95700-proxy-tls podName:760f8de6-914d-42f7-a40f-a66ff4f95700 nodeName:}" failed. No retries permitted until 2026-07-27 21:27:03.941548119 +0000 UTC m=+1347.308737974 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/760f8de6-914d-42f7-a40f-a66ff4f95700-proxy-tls") pod "isvc-autogluon-v2-variants-predictor-5c889d8844-szvpd" (UID: "760f8de6-914d-42f7-a40f-a66ff4f95700") : secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found
Jul 27 21:27:03.441777 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:27:03.441742    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/760f8de6-914d-42f7-a40f-a66ff4f95700-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-szvpd\" (UID: \"760f8de6-914d-42f7-a40f-a66ff4f95700\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-szvpd"
Jul 27 21:27:03.441881 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:27:03.441864    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/760f8de6-914d-42f7-a40f-a66ff4f95700-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-szvpd\" (UID: \"760f8de6-914d-42f7-a40f-a66ff4f95700\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-szvpd"
Jul 27 21:27:03.441970 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:27:03.441950    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/760f8de6-914d-42f7-a40f-a66ff4f95700-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-szvpd\" (UID: \"760f8de6-914d-42f7-a40f-a66ff4f95700\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-szvpd"
Jul 27 21:27:03.449983 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:27:03.449961    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f84jx\" (UniqueName: \"kubernetes.io/projected/760f8de6-914d-42f7-a40f-a66ff4f95700-kube-api-access-f84jx\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-szvpd\" (UID: \"760f8de6-914d-42f7-a40f-a66ff4f95700\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-szvpd"
Jul 27 21:27:03.713339 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:27:03.713310    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:27:03.946350 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:27:03.946316    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/760f8de6-914d-42f7-a40f-a66ff4f95700-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-szvpd\" (UID: \"760f8de6-914d-42f7-a40f-a66ff4f95700\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-szvpd"
Jul 27 21:27:03.948779 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:27:03.948734    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/760f8de6-914d-42f7-a40f-a66ff4f95700-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-szvpd\" (UID: \"760f8de6-914d-42f7-a40f-a66ff4f95700\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-szvpd"
Jul 27 21:27:04.216305 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:27:04.216270    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-szvpd"
Jul 27 21:27:04.340078 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:27:04.340050    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-szvpd"]
Jul 27 21:27:04.343011 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:27:04.342988    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod760f8de6_914d_42f7_a40f_a66ff4f95700.slice/crio-f19bdd026a61851424f56a1590bae389681349bd90a20944b236f268d9f5967d WatchSource:0}: Error finding container f19bdd026a61851424f56a1590bae389681349bd90a20944b236f268d9f5967d: Status 404 returned error can't find the container with id f19bdd026a61851424f56a1590bae389681349bd90a20944b236f268d9f5967d
Jul 27 21:27:04.572062 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:27:04.571976    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-szvpd" event={"ID":"760f8de6-914d-42f7-a40f-a66ff4f95700","Type":"ContainerStarted","Data":"eb9e8c382f6997d55d72eee51db1b9a6707d83e8f48734e8815cd25471137d31"}
Jul 27 21:27:04.572062 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:27:04.572050    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-szvpd" event={"ID":"760f8de6-914d-42f7-a40f-a66ff4f95700","Type":"ContainerStarted","Data":"f19bdd026a61851424f56a1590bae389681349bd90a20944b236f268d9f5967d"}
Jul 27 21:27:33.680562 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:27:33.680534    2577 generic.go:356] "Generic (PLEG): container finished" podID="760f8de6-914d-42f7-a40f-a66ff4f95700" containerID="eb9e8c382f6997d55d72eee51db1b9a6707d83e8f48734e8815cd25471137d31" exitCode=0
Jul 27 21:27:33.681020 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:27:33.680574    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-szvpd" event={"ID":"760f8de6-914d-42f7-a40f-a66ff4f95700","Type":"ContainerDied","Data":"eb9e8c382f6997d55d72eee51db1b9a6707d83e8f48734e8815cd25471137d31"}
Jul 27 21:27:34.686721 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:27:34.686682    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-szvpd" event={"ID":"760f8de6-914d-42f7-a40f-a66ff4f95700","Type":"ContainerStarted","Data":"81a5c91bdabd48065e5a21bff58c4fdeb2617a892768f79ea309cd3ae7bb9174"}
Jul 27 21:27:34.687117 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:27:34.686795    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-szvpd" event={"ID":"760f8de6-914d-42f7-a40f-a66ff4f95700","Type":"ContainerStarted","Data":"a1b6b16fc9c6a3ec2260e37410d59c9ac20c949c7801a71a006013cb9ac92c0f"}
Jul 27 21:27:34.687117 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:27:34.686984    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-szvpd"
Jul 27 21:27:34.705569 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:27:34.705521    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-szvpd" podStartSLOduration=2.758113393 podStartE2EDuration="31.705506073s" totalImagesPullingTime="0s" totalInitContainerRuntime="28.94739268s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:27:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:27:34.704333276 +0000 UTC m=+1378.071523143" watchObservedRunningTime="2026-07-27 21:27:34.705506073 +0000 UTC m=+1378.072695937"
Jul 27 21:27:35.691531 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:27:35.691495    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-szvpd"
Jul 27 21:27:41.702698 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:27:41.702671    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-szvpd"
Jul 27 21:29:11.135702 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:11.135674    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-szvpd"
Jul 27 21:29:19.784475 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:19.784437    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-szvpd"]
Jul 27 21:29:19.785078 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:19.784788    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-szvpd" podUID="760f8de6-914d-42f7-a40f-a66ff4f95700" containerName="kserve-container" containerID="cri-o://a1b6b16fc9c6a3ec2260e37410d59c9ac20c949c7801a71a006013cb9ac92c0f" gracePeriod=30
Jul 27 21:29:19.785078 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:19.784823    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-szvpd" podUID="760f8de6-914d-42f7-a40f-a66ff4f95700" containerName="kube-rbac-proxy" containerID="cri-o://81a5c91bdabd48065e5a21bff58c4fdeb2617a892768f79ea309cd3ae7bb9174" gracePeriod=30
Jul 27 21:29:19.835432 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:19.835403    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:20.092973 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:20.092879    2577 generic.go:356] "Generic (PLEG): container finished" podID="760f8de6-914d-42f7-a40f-a66ff4f95700" containerID="81a5c91bdabd48065e5a21bff58c4fdeb2617a892768f79ea309cd3ae7bb9174" exitCode=2
Jul 27 21:29:20.092973 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:20.092934    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-szvpd" event={"ID":"760f8de6-914d-42f7-a40f-a66ff4f95700","Type":"ContainerDied","Data":"81a5c91bdabd48065e5a21bff58c4fdeb2617a892768f79ea309cd3ae7bb9174"}
Jul 27 21:29:21.098956 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:21.098923    2577 generic.go:356] "Generic (PLEG): container finished" podID="760f8de6-914d-42f7-a40f-a66ff4f95700" containerID="a1b6b16fc9c6a3ec2260e37410d59c9ac20c949c7801a71a006013cb9ac92c0f" exitCode=0
Jul 27 21:29:21.099289 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:21.098987    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-szvpd" event={"ID":"760f8de6-914d-42f7-a40f-a66ff4f95700","Type":"ContainerDied","Data":"a1b6b16fc9c6a3ec2260e37410d59c9ac20c949c7801a71a006013cb9ac92c0f"}
Jul 27 21:29:21.125213 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:21.125192    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-szvpd"
Jul 27 21:29:21.194909 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:21.194833    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/760f8de6-914d-42f7-a40f-a66ff4f95700-proxy-tls\") pod \"760f8de6-914d-42f7-a40f-a66ff4f95700\" (UID: \"760f8de6-914d-42f7-a40f-a66ff4f95700\") "
Jul 27 21:29:21.194909 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:21.194876    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/760f8de6-914d-42f7-a40f-a66ff4f95700-cabundle-cert\") pod \"760f8de6-914d-42f7-a40f-a66ff4f95700\" (UID: \"760f8de6-914d-42f7-a40f-a66ff4f95700\") "
Jul 27 21:29:21.195056 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:21.194921    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/760f8de6-914d-42f7-a40f-a66ff4f95700-kube-rbac-proxy-sar-config\") pod \"760f8de6-914d-42f7-a40f-a66ff4f95700\" (UID: \"760f8de6-914d-42f7-a40f-a66ff4f95700\") "
Jul 27 21:29:21.195056 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:21.194946    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f84jx\" (UniqueName: \"kubernetes.io/projected/760f8de6-914d-42f7-a40f-a66ff4f95700-kube-api-access-f84jx\") pod \"760f8de6-914d-42f7-a40f-a66ff4f95700\" (UID: \"760f8de6-914d-42f7-a40f-a66ff4f95700\") "
Jul 27 21:29:21.195056 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:21.194989    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/760f8de6-914d-42f7-a40f-a66ff4f95700-kserve-provision-location\") pod \"760f8de6-914d-42f7-a40f-a66ff4f95700\" (UID: \"760f8de6-914d-42f7-a40f-a66ff4f95700\") "
Jul 27 21:29:21.195559 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:21.195240    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/760f8de6-914d-42f7-a40f-a66ff4f95700-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "760f8de6-914d-42f7-a40f-a66ff4f95700" (UID: "760f8de6-914d-42f7-a40f-a66ff4f95700"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:29:21.195559 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:21.195252    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/760f8de6-914d-42f7-a40f-a66ff4f95700-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "760f8de6-914d-42f7-a40f-a66ff4f95700" (UID: "760f8de6-914d-42f7-a40f-a66ff4f95700"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:29:21.197405 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:21.197381    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/760f8de6-914d-42f7-a40f-a66ff4f95700-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "760f8de6-914d-42f7-a40f-a66ff4f95700" (UID: "760f8de6-914d-42f7-a40f-a66ff4f95700"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:29:21.197506 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:21.197388    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/760f8de6-914d-42f7-a40f-a66ff4f95700-kube-api-access-f84jx" (OuterVolumeSpecName: "kube-api-access-f84jx") pod "760f8de6-914d-42f7-a40f-a66ff4f95700" (UID: "760f8de6-914d-42f7-a40f-a66ff4f95700"). InnerVolumeSpecName "kube-api-access-f84jx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:29:21.199979 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:21.199955    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/760f8de6-914d-42f7-a40f-a66ff4f95700-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "760f8de6-914d-42f7-a40f-a66ff4f95700" (UID: "760f8de6-914d-42f7-a40f-a66ff4f95700"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:29:21.296417 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:21.296394    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/760f8de6-914d-42f7-a40f-a66ff4f95700-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:29:21.296417 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:21.296413    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/760f8de6-914d-42f7-a40f-a66ff4f95700-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:29:21.296550 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:21.296424    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/760f8de6-914d-42f7-a40f-a66ff4f95700-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:29:21.296550 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:21.296433    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f84jx\" (UniqueName: \"kubernetes.io/projected/760f8de6-914d-42f7-a40f-a66ff4f95700-kube-api-access-f84jx\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:29:21.296550 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:21.296441    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/760f8de6-914d-42f7-a40f-a66ff4f95700-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:29:22.105206 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:22.105176    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-szvpd" event={"ID":"760f8de6-914d-42f7-a40f-a66ff4f95700","Type":"ContainerDied","Data":"f19bdd026a61851424f56a1590bae389681349bd90a20944b236f268d9f5967d"}
Jul 27 21:29:22.105206 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:22.105209    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-szvpd"
Jul 27 21:29:22.105657 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:22.105240    2577 scope.go:117] "RemoveContainer" containerID="81a5c91bdabd48065e5a21bff58c4fdeb2617a892768f79ea309cd3ae7bb9174"
Jul 27 21:29:22.115283 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:22.115205    2577 scope.go:117] "RemoveContainer" containerID="a1b6b16fc9c6a3ec2260e37410d59c9ac20c949c7801a71a006013cb9ac92c0f"
Jul 27 21:29:22.123232 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:22.123216    2577 scope.go:117] "RemoveContainer" containerID="eb9e8c382f6997d55d72eee51db1b9a6707d83e8f48734e8815cd25471137d31"
Jul 27 21:29:22.127470 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:22.127450    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-szvpd"]
Jul 27 21:29:22.132798 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:22.132777    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-szvpd"]
Jul 27 21:29:23.134114 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:23.134085    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="760f8de6-914d-42f7-a40f-a66ff4f95700" path="/var/lib/kubelet/pods/760f8de6-914d-42f7-a40f-a66ff4f95700/volumes"
Jul 27 21:29:24.868643 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:24.868611    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:24.901142 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:24.901111    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-94tg9"]
Jul 27 21:29:24.901398 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:24.901386    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="760f8de6-914d-42f7-a40f-a66ff4f95700" containerName="kube-rbac-proxy"
Jul 27 21:29:24.901605 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:24.901400    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="760f8de6-914d-42f7-a40f-a66ff4f95700" containerName="kube-rbac-proxy"
Jul 27 21:29:24.901605 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:24.901414    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="760f8de6-914d-42f7-a40f-a66ff4f95700" containerName="kserve-container"
Jul 27 21:29:24.901605 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:24.901420    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="760f8de6-914d-42f7-a40f-a66ff4f95700" containerName="kserve-container"
Jul 27 21:29:24.901605 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:24.901430    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="760f8de6-914d-42f7-a40f-a66ff4f95700" containerName="storage-initializer"
Jul 27 21:29:24.901605 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:24.901435    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="760f8de6-914d-42f7-a40f-a66ff4f95700" containerName="storage-initializer"
Jul 27 21:29:24.901605 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:24.901479    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="760f8de6-914d-42f7-a40f-a66ff4f95700" containerName="kserve-container"
Jul 27 21:29:24.901605 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:24.901489    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="760f8de6-914d-42f7-a40f-a66ff4f95700" containerName="kube-rbac-proxy"
Jul 27 21:29:24.906279 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:24.906262    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-94tg9"
Jul 27 21:29:24.908402 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:24.908376    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 21:29:24.908530 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:24.908516    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 21:29:24.908622 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:24.908601    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v7c66\""
Jul 27 21:29:24.908685 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:24.908637    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Jul 27 21:29:24.908744 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:24.908716    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 21:29:24.909362 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:24.909338    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Jul 27 21:29:24.909449 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:24.909412    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 21:29:24.917782 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:24.916992    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-94tg9"]
Jul 27 21:29:24.978318 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:24.978282    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:25.021530 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:25.021505    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e307f46c-51d7-4d93-8035-37ac48762ce2-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-94tg9\" (UID: \"e307f46c-51d7-4d93-8035-37ac48762ce2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-94tg9"
Jul 27 21:29:25.021618 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:25.021541    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e307f46c-51d7-4d93-8035-37ac48762ce2-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-94tg9\" (UID: \"e307f46c-51d7-4d93-8035-37ac48762ce2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-94tg9"
Jul 27 21:29:25.021618 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:25.021558    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpx9c\" (UniqueName: \"kubernetes.io/projected/e307f46c-51d7-4d93-8035-37ac48762ce2-kube-api-access-jpx9c\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-94tg9\" (UID: \"e307f46c-51d7-4d93-8035-37ac48762ce2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-94tg9"
Jul 27 21:29:25.021722 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:25.021648    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e307f46c-51d7-4d93-8035-37ac48762ce2-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-94tg9\" (UID: \"e307f46c-51d7-4d93-8035-37ac48762ce2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-94tg9"
Jul 27 21:29:25.021722 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:25.021706    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e307f46c-51d7-4d93-8035-37ac48762ce2-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-94tg9\" (UID: \"e307f46c-51d7-4d93-8035-37ac48762ce2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-94tg9"
Jul 27 21:29:25.122739 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:25.122674    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e307f46c-51d7-4d93-8035-37ac48762ce2-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-94tg9\" (UID: \"e307f46c-51d7-4d93-8035-37ac48762ce2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-94tg9"
Jul 27 21:29:25.122739 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:25.122716    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e307f46c-51d7-4d93-8035-37ac48762ce2-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-94tg9\" (UID: \"e307f46c-51d7-4d93-8035-37ac48762ce2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-94tg9"
Jul 27 21:29:25.122926 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:25.122787    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e307f46c-51d7-4d93-8035-37ac48762ce2-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-94tg9\" (UID: \"e307f46c-51d7-4d93-8035-37ac48762ce2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-94tg9"
Jul 27 21:29:25.122926 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:25.122830    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e307f46c-51d7-4d93-8035-37ac48762ce2-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-94tg9\" (UID: \"e307f46c-51d7-4d93-8035-37ac48762ce2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-94tg9"
Jul 27 21:29:25.122926 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:25.122854    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpx9c\" (UniqueName: \"kubernetes.io/projected/e307f46c-51d7-4d93-8035-37ac48762ce2-kube-api-access-jpx9c\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-94tg9\" (UID: \"e307f46c-51d7-4d93-8035-37ac48762ce2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-94tg9"
Jul 27 21:29:25.123105 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:25.123087    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e307f46c-51d7-4d93-8035-37ac48762ce2-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-94tg9\" (UID: \"e307f46c-51d7-4d93-8035-37ac48762ce2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-94tg9"
Jul 27 21:29:25.123518 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:25.123496    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e307f46c-51d7-4d93-8035-37ac48762ce2-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-94tg9\" (UID: \"e307f46c-51d7-4d93-8035-37ac48762ce2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-94tg9"
Jul 27 21:29:25.123561 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:25.123514    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e307f46c-51d7-4d93-8035-37ac48762ce2-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-94tg9\" (UID: \"e307f46c-51d7-4d93-8035-37ac48762ce2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-94tg9"
Jul 27 21:29:25.125094 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:25.125077    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e307f46c-51d7-4d93-8035-37ac48762ce2-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-94tg9\" (UID: \"e307f46c-51d7-4d93-8035-37ac48762ce2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-94tg9"
Jul 27 21:29:25.132490 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:25.132467    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpx9c\" (UniqueName: \"kubernetes.io/projected/e307f46c-51d7-4d93-8035-37ac48762ce2-kube-api-access-jpx9c\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-94tg9\" (UID: \"e307f46c-51d7-4d93-8035-37ac48762ce2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-94tg9"
Jul 27 21:29:25.222285 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:25.222257    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-94tg9"
Jul 27 21:29:25.346210 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:25.346177    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-94tg9"]
Jul 27 21:29:25.349794 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:29:25.349752    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode307f46c_51d7_4d93_8035_37ac48762ce2.slice/crio-0176000bbf34558e7b31fa3e78db59e02e68e1473de4b462d063d85840d5b8d8 WatchSource:0}: Error finding container 0176000bbf34558e7b31fa3e78db59e02e68e1473de4b462d063d85840d5b8d8: Status 404 returned error can't find the container with id 0176000bbf34558e7b31fa3e78db59e02e68e1473de4b462d063d85840d5b8d8
Jul 27 21:29:26.123726 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:26.123696    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-94tg9" event={"ID":"e307f46c-51d7-4d93-8035-37ac48762ce2","Type":"ContainerStarted","Data":"1d9de435c52c662c8daab5332a148c483ea01249d6810cc8c665418f5fe1f92d"}
Jul 27 21:29:26.124117 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:26.123750    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-94tg9" event={"ID":"e307f46c-51d7-4d93-8035-37ac48762ce2","Type":"ContainerStarted","Data":"0176000bbf34558e7b31fa3e78db59e02e68e1473de4b462d063d85840d5b8d8"}
Jul 27 21:29:30.140078 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:30.140048    2577 generic.go:356] "Generic (PLEG): container finished" podID="e307f46c-51d7-4d93-8035-37ac48762ce2" containerID="1d9de435c52c662c8daab5332a148c483ea01249d6810cc8c665418f5fe1f92d" exitCode=0
Jul 27 21:29:30.140403 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:30.140088    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-94tg9" event={"ID":"e307f46c-51d7-4d93-8035-37ac48762ce2","Type":"ContainerDied","Data":"1d9de435c52c662c8daab5332a148c483ea01249d6810cc8c665418f5fe1f92d"}
Jul 27 21:29:31.146712 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:31.146680    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-94tg9" event={"ID":"e307f46c-51d7-4d93-8035-37ac48762ce2","Type":"ContainerStarted","Data":"7ed645e1a460d1efbaf991acf53dc0afa68964f8b21e04a602ec2b55f54dc18c"}
Jul 27 21:29:31.147089 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:31.146737    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-94tg9" event={"ID":"e307f46c-51d7-4d93-8035-37ac48762ce2","Type":"ContainerStarted","Data":"545f628f5bc951fdcaeb21cc91593f7b5f3904b0fc1529abcadbce67ab762a05"}
Jul 27 21:29:31.147089 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:31.146960    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-94tg9"
Jul 27 21:29:31.147168 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:31.147100    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-94tg9"
Jul 27 21:29:31.165306 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:31.165260    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-94tg9" podStartSLOduration=2.591685332 podStartE2EDuration="7.165242854s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.573557522s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:29:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:29:31.164639516 +0000 UTC m=+1494.531829382" watchObservedRunningTime="2026-07-27 21:29:31.165242854 +0000 UTC m=+1494.532432721"
Jul 27 21:29:37.157657 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:29:37.157620    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-94tg9"
Jul 27 21:31:04.134091 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:04.134062    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-94tg9"
Jul 27 21:31:11.109593 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:11.109556    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-94tg9"]
Jul 27 21:31:11.110004 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:11.109848    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-94tg9" podUID="e307f46c-51d7-4d93-8035-37ac48762ce2" containerName="kserve-container" containerID="cri-o://545f628f5bc951fdcaeb21cc91593f7b5f3904b0fc1529abcadbce67ab762a05" gracePeriod=30
Jul 27 21:31:11.110004 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:11.109897    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-94tg9" podUID="e307f46c-51d7-4d93-8035-37ac48762ce2" containerName="kube-rbac-proxy" containerID="cri-o://7ed645e1a460d1efbaf991acf53dc0afa68964f8b21e04a602ec2b55f54dc18c" gracePeriod=30
Jul 27 21:31:11.524052 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:11.524018    2577 generic.go:356] "Generic (PLEG): container finished" podID="e307f46c-51d7-4d93-8035-37ac48762ce2" containerID="7ed645e1a460d1efbaf991acf53dc0afa68964f8b21e04a602ec2b55f54dc18c" exitCode=2
Jul 27 21:31:11.524231 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:11.524092    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-94tg9" event={"ID":"e307f46c-51d7-4d93-8035-37ac48762ce2","Type":"ContainerDied","Data":"7ed645e1a460d1efbaf991acf53dc0afa68964f8b21e04a602ec2b55f54dc18c"}
Jul 27 21:31:11.806721 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:11.806645    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:31:12.152459 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:12.152369    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-94tg9" podUID="e307f46c-51d7-4d93-8035-37ac48762ce2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Jul 27 21:31:12.453470 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:12.453447    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-94tg9"
Jul 27 21:31:12.519556 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:12.519524    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jpx9c\" (UniqueName: \"kubernetes.io/projected/e307f46c-51d7-4d93-8035-37ac48762ce2-kube-api-access-jpx9c\") pod \"e307f46c-51d7-4d93-8035-37ac48762ce2\" (UID: \"e307f46c-51d7-4d93-8035-37ac48762ce2\") "
Jul 27 21:31:12.519700 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:12.519570    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e307f46c-51d7-4d93-8035-37ac48762ce2-cabundle-cert\") pod \"e307f46c-51d7-4d93-8035-37ac48762ce2\" (UID: \"e307f46c-51d7-4d93-8035-37ac48762ce2\") "
Jul 27 21:31:12.519700 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:12.519600    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e307f46c-51d7-4d93-8035-37ac48762ce2-kube-rbac-proxy-sar-config\") pod \"e307f46c-51d7-4d93-8035-37ac48762ce2\" (UID: \"e307f46c-51d7-4d93-8035-37ac48762ce2\") "
Jul 27 21:31:12.519700 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:12.519620    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e307f46c-51d7-4d93-8035-37ac48762ce2-proxy-tls\") pod \"e307f46c-51d7-4d93-8035-37ac48762ce2\" (UID: \"e307f46c-51d7-4d93-8035-37ac48762ce2\") "
Jul 27 21:31:12.519700 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:12.519645    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e307f46c-51d7-4d93-8035-37ac48762ce2-kserve-provision-location\") pod \"e307f46c-51d7-4d93-8035-37ac48762ce2\" (UID: \"e307f46c-51d7-4d93-8035-37ac48762ce2\") "
Jul 27 21:31:12.520079 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:12.520037    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e307f46c-51d7-4d93-8035-37ac48762ce2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e307f46c-51d7-4d93-8035-37ac48762ce2" (UID: "e307f46c-51d7-4d93-8035-37ac48762ce2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:31:12.520079 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:12.520035    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e307f46c-51d7-4d93-8035-37ac48762ce2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e307f46c-51d7-4d93-8035-37ac48762ce2" (UID: "e307f46c-51d7-4d93-8035-37ac48762ce2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:31:12.521579 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:12.521553    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e307f46c-51d7-4d93-8035-37ac48762ce2-kube-api-access-jpx9c" (OuterVolumeSpecName: "kube-api-access-jpx9c") pod "e307f46c-51d7-4d93-8035-37ac48762ce2" (UID: "e307f46c-51d7-4d93-8035-37ac48762ce2"). InnerVolumeSpecName "kube-api-access-jpx9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:31:12.521694 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:12.521631    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e307f46c-51d7-4d93-8035-37ac48762ce2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e307f46c-51d7-4d93-8035-37ac48762ce2" (UID: "e307f46c-51d7-4d93-8035-37ac48762ce2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:31:12.524486 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:12.524462    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e307f46c-51d7-4d93-8035-37ac48762ce2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e307f46c-51d7-4d93-8035-37ac48762ce2" (UID: "e307f46c-51d7-4d93-8035-37ac48762ce2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:31:12.530089 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:12.530068    2577 generic.go:356] "Generic (PLEG): container finished" podID="e307f46c-51d7-4d93-8035-37ac48762ce2" containerID="545f628f5bc951fdcaeb21cc91593f7b5f3904b0fc1529abcadbce67ab762a05" exitCode=0
Jul 27 21:31:12.530188 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:12.530140    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-94tg9"
Jul 27 21:31:12.530188 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:12.530144    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-94tg9" event={"ID":"e307f46c-51d7-4d93-8035-37ac48762ce2","Type":"ContainerDied","Data":"545f628f5bc951fdcaeb21cc91593f7b5f3904b0fc1529abcadbce67ab762a05"}
Jul 27 21:31:12.530274 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:12.530206    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-94tg9" event={"ID":"e307f46c-51d7-4d93-8035-37ac48762ce2","Type":"ContainerDied","Data":"0176000bbf34558e7b31fa3e78db59e02e68e1473de4b462d063d85840d5b8d8"}
Jul 27 21:31:12.530274 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:12.530234    2577 scope.go:117] "RemoveContainer" containerID="7ed645e1a460d1efbaf991acf53dc0afa68964f8b21e04a602ec2b55f54dc18c"
Jul 27 21:31:12.538504 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:12.538465    2577 scope.go:117] "RemoveContainer" containerID="545f628f5bc951fdcaeb21cc91593f7b5f3904b0fc1529abcadbce67ab762a05"
Jul 27 21:31:12.545336 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:12.545321    2577 scope.go:117] "RemoveContainer" containerID="1d9de435c52c662c8daab5332a148c483ea01249d6810cc8c665418f5fe1f92d"
Jul 27 21:31:12.552220 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:12.552201    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-94tg9"]
Jul 27 21:31:12.552888 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:12.552873    2577 scope.go:117] "RemoveContainer" containerID="7ed645e1a460d1efbaf991acf53dc0afa68964f8b21e04a602ec2b55f54dc18c"
Jul 27 21:31:12.553121 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:31:12.553103    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ed645e1a460d1efbaf991acf53dc0afa68964f8b21e04a602ec2b55f54dc18c\": container with ID starting with 7ed645e1a460d1efbaf991acf53dc0afa68964f8b21e04a602ec2b55f54dc18c not found: ID does not exist" containerID="7ed645e1a460d1efbaf991acf53dc0afa68964f8b21e04a602ec2b55f54dc18c"
Jul 27 21:31:12.553171 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:12.553131    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ed645e1a460d1efbaf991acf53dc0afa68964f8b21e04a602ec2b55f54dc18c"} err="failed to get container status \"7ed645e1a460d1efbaf991acf53dc0afa68964f8b21e04a602ec2b55f54dc18c\": rpc error: code = NotFound desc = could not find container \"7ed645e1a460d1efbaf991acf53dc0afa68964f8b21e04a602ec2b55f54dc18c\": container with ID starting with 7ed645e1a460d1efbaf991acf53dc0afa68964f8b21e04a602ec2b55f54dc18c not found: ID does not exist"
Jul 27 21:31:12.553171 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:12.553149    2577 scope.go:117] "RemoveContainer" containerID="545f628f5bc951fdcaeb21cc91593f7b5f3904b0fc1529abcadbce67ab762a05"
Jul 27 21:31:12.553371 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:31:12.553356    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"545f628f5bc951fdcaeb21cc91593f7b5f3904b0fc1529abcadbce67ab762a05\": container with ID starting with 545f628f5bc951fdcaeb21cc91593f7b5f3904b0fc1529abcadbce67ab762a05 not found: ID does not exist" containerID="545f628f5bc951fdcaeb21cc91593f7b5f3904b0fc1529abcadbce67ab762a05"
Jul 27 21:31:12.553410 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:12.553377    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"545f628f5bc951fdcaeb21cc91593f7b5f3904b0fc1529abcadbce67ab762a05"} err="failed to get container status \"545f628f5bc951fdcaeb21cc91593f7b5f3904b0fc1529abcadbce67ab762a05\": rpc error: code = NotFound desc = could not find container \"545f628f5bc951fdcaeb21cc91593f7b5f3904b0fc1529abcadbce67ab762a05\": container with ID starting with 545f628f5bc951fdcaeb21cc91593f7b5f3904b0fc1529abcadbce67ab762a05 not found: ID does not exist"
Jul 27 21:31:12.553410 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:12.553393    2577 scope.go:117] "RemoveContainer" containerID="1d9de435c52c662c8daab5332a148c483ea01249d6810cc8c665418f5fe1f92d"
Jul 27 21:31:12.553581 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:31:12.553564    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d9de435c52c662c8daab5332a148c483ea01249d6810cc8c665418f5fe1f92d\": container with ID starting with 1d9de435c52c662c8daab5332a148c483ea01249d6810cc8c665418f5fe1f92d not found: ID does not exist" containerID="1d9de435c52c662c8daab5332a148c483ea01249d6810cc8c665418f5fe1f92d"
Jul 27 21:31:12.553617 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:12.553586    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d9de435c52c662c8daab5332a148c483ea01249d6810cc8c665418f5fe1f92d"} err="failed to get container status \"1d9de435c52c662c8daab5332a148c483ea01249d6810cc8c665418f5fe1f92d\": rpc error: code = NotFound desc = could not find container \"1d9de435c52c662c8daab5332a148c483ea01249d6810cc8c665418f5fe1f92d\": container with ID starting with 1d9de435c52c662c8daab5332a148c483ea01249d6810cc8c665418f5fe1f92d not found: ID does not exist"
Jul 27 21:31:12.558359 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:12.558341    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-94tg9"]
Jul 27 21:31:12.620372 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:12.620351    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpx9c\" (UniqueName: \"kubernetes.io/projected/e307f46c-51d7-4d93-8035-37ac48762ce2-kube-api-access-jpx9c\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:31:12.620372 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:12.620371    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e307f46c-51d7-4d93-8035-37ac48762ce2-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:31:12.620482 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:12.620381    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e307f46c-51d7-4d93-8035-37ac48762ce2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:31:12.620482 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:12.620389    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e307f46c-51d7-4d93-8035-37ac48762ce2-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:31:12.620482 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:12.620397    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e307f46c-51d7-4d93-8035-37ac48762ce2-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:31:13.134540 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:13.134504    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e307f46c-51d7-4d93-8035-37ac48762ce2" path="/var/lib/kubelet/pods/e307f46c-51d7-4d93-8035-37ac48762ce2/volumes"
Jul 27 21:31:16.029515 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:16.029481    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:31:16.084923 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:16.084893    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb"]
Jul 27 21:31:16.085193 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:16.085181    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e307f46c-51d7-4d93-8035-37ac48762ce2" containerName="storage-initializer"
Jul 27 21:31:16.085235 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:16.085194    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e307f46c-51d7-4d93-8035-37ac48762ce2" containerName="storage-initializer"
Jul 27 21:31:16.085235 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:16.085201    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e307f46c-51d7-4d93-8035-37ac48762ce2" containerName="kube-rbac-proxy"
Jul 27 21:31:16.085235 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:16.085207    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e307f46c-51d7-4d93-8035-37ac48762ce2" containerName="kube-rbac-proxy"
Jul 27 21:31:16.085235 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:16.085221    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e307f46c-51d7-4d93-8035-37ac48762ce2" containerName="kserve-container"
Jul 27 21:31:16.085235 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:16.085227    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e307f46c-51d7-4d93-8035-37ac48762ce2" containerName="kserve-container"
Jul 27 21:31:16.085377 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:16.085268    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e307f46c-51d7-4d93-8035-37ac48762ce2" containerName="kube-rbac-proxy"
Jul 27 21:31:16.085377 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:16.085277    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e307f46c-51d7-4d93-8035-37ac48762ce2" containerName="kserve-container"
Jul 27 21:31:16.089835 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:16.089819    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb"
Jul 27 21:31:16.091994 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:16.091970    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Jul 27 21:31:16.092130 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:16.092051    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 21:31:16.092196 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:16.092053    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 21:31:16.092196 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:16.092059    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v7c66\""
Jul 27 21:31:16.092196 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:16.092160    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 21:31:16.092865 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:16.092847    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 21:31:16.092951 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:16.092883    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Jul 27 21:31:16.098827 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:16.098802    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb"]
Jul 27 21:31:16.247612 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:16.247579    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bab59149-41b1-4ce7-b03e-074c8c13852a-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb\" (UID: \"bab59149-41b1-4ce7-b03e-074c8c13852a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb"
Jul 27 21:31:16.247612 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:16.247611    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8gmt\" (UniqueName: \"kubernetes.io/projected/bab59149-41b1-4ce7-b03e-074c8c13852a-kube-api-access-w8gmt\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb\" (UID: \"bab59149-41b1-4ce7-b03e-074c8c13852a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb"
Jul 27 21:31:16.247876 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:16.247677    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bab59149-41b1-4ce7-b03e-074c8c13852a-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb\" (UID: \"bab59149-41b1-4ce7-b03e-074c8c13852a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb"
Jul 27 21:31:16.247876 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:16.247754    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bab59149-41b1-4ce7-b03e-074c8c13852a-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb\" (UID: \"bab59149-41b1-4ce7-b03e-074c8c13852a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb"
Jul 27 21:31:16.247876 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:16.247820    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bab59149-41b1-4ce7-b03e-074c8c13852a-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb\" (UID: \"bab59149-41b1-4ce7-b03e-074c8c13852a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb"
Jul 27 21:31:16.275296 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:16.275268    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:31:16.348497 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:16.348440    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bab59149-41b1-4ce7-b03e-074c8c13852a-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb\" (UID: \"bab59149-41b1-4ce7-b03e-074c8c13852a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb"
Jul 27 21:31:16.348497 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:16.348484    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bab59149-41b1-4ce7-b03e-074c8c13852a-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb\" (UID: \"bab59149-41b1-4ce7-b03e-074c8c13852a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb"
Jul 27 21:31:16.348613 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:16.348517    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bab59149-41b1-4ce7-b03e-074c8c13852a-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb\" (UID: \"bab59149-41b1-4ce7-b03e-074c8c13852a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb"
Jul 27 21:31:16.348613 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:16.348559    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bab59149-41b1-4ce7-b03e-074c8c13852a-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb\" (UID: \"bab59149-41b1-4ce7-b03e-074c8c13852a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb"
Jul 27 21:31:16.348613 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:16.348586    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8gmt\" (UniqueName: \"kubernetes.io/projected/bab59149-41b1-4ce7-b03e-074c8c13852a-kube-api-access-w8gmt\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb\" (UID: \"bab59149-41b1-4ce7-b03e-074c8c13852a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb"
Jul 27 21:31:16.348892 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:16.348873    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bab59149-41b1-4ce7-b03e-074c8c13852a-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb\" (UID: \"bab59149-41b1-4ce7-b03e-074c8c13852a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb"
Jul 27 21:31:16.349201 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:16.349180    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bab59149-41b1-4ce7-b03e-074c8c13852a-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb\" (UID: \"bab59149-41b1-4ce7-b03e-074c8c13852a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb"
Jul 27 21:31:16.349239 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:16.349211    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bab59149-41b1-4ce7-b03e-074c8c13852a-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb\" (UID: \"bab59149-41b1-4ce7-b03e-074c8c13852a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb"
Jul 27 21:31:16.350840 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:16.350825    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bab59149-41b1-4ce7-b03e-074c8c13852a-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb\" (UID: \"bab59149-41b1-4ce7-b03e-074c8c13852a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb"
Jul 27 21:31:16.356713 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:16.356690    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8gmt\" (UniqueName: \"kubernetes.io/projected/bab59149-41b1-4ce7-b03e-074c8c13852a-kube-api-access-w8gmt\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb\" (UID: \"bab59149-41b1-4ce7-b03e-074c8c13852a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb"
Jul 27 21:31:16.401467 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:16.401445    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb"
Jul 27 21:31:16.527895 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:16.527740    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb"]
Jul 27 21:31:16.529910 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:31:16.529884    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbab59149_41b1_4ce7_b03e_074c8c13852a.slice/crio-51c5a735405b7c57e3762d59f63c8497ed01e259d5c10ca38b1405072be04e91 WatchSource:0}: Error finding container 51c5a735405b7c57e3762d59f63c8497ed01e259d5c10ca38b1405072be04e91: Status 404 returned error can't find the container with id 51c5a735405b7c57e3762d59f63c8497ed01e259d5c10ca38b1405072be04e91
Jul 27 21:31:16.546424 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:16.546403    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb" event={"ID":"bab59149-41b1-4ce7-b03e-074c8c13852a","Type":"ContainerStarted","Data":"51c5a735405b7c57e3762d59f63c8497ed01e259d5c10ca38b1405072be04e91"}
Jul 27 21:31:17.551792 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:17.551743    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb" event={"ID":"bab59149-41b1-4ce7-b03e-074c8c13852a","Type":"ContainerStarted","Data":"000f9f53eb52a6263ebf412e1a887dc0557f3885af62120726b831dac98f4f48"}
Jul 27 21:31:25.585607 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:25.585571    2577 generic.go:356] "Generic (PLEG): container finished" podID="bab59149-41b1-4ce7-b03e-074c8c13852a" containerID="000f9f53eb52a6263ebf412e1a887dc0557f3885af62120726b831dac98f4f48" exitCode=0
Jul 27 21:31:25.585999 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:25.585649    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb" event={"ID":"bab59149-41b1-4ce7-b03e-074c8c13852a","Type":"ContainerDied","Data":"000f9f53eb52a6263ebf412e1a887dc0557f3885af62120726b831dac98f4f48"}
Jul 27 21:31:26.591614 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:26.591580    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb" event={"ID":"bab59149-41b1-4ce7-b03e-074c8c13852a","Type":"ContainerStarted","Data":"0bb6d51cd3b29def7c455522e3bba74ee92c66631c0049c79c486b64e3efcb0e"}
Jul 27 21:31:26.592010 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:26.591667    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb" event={"ID":"bab59149-41b1-4ce7-b03e-074c8c13852a","Type":"ContainerStarted","Data":"79704f326aa917d2b50cc11db8f8f70c9b0afe3e462c435a84be4cb1c461f5aa"}
Jul 27 21:31:26.592010 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:26.591860    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb"
Jul 27 21:31:26.611199 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:26.611149    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb" podStartSLOduration=1.680678126 podStartE2EDuration="10.611134505s" totalImagesPullingTime="0s" totalInitContainerRuntime="8.930456379s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:31:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:31:26.609846173 +0000 UTC m=+1609.977036042" watchObservedRunningTime="2026-07-27 21:31:26.611134505 +0000 UTC m=+1609.978324372"
Jul 27 21:31:27.596425 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:27.596396    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb"
Jul 27 21:31:27.597914 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:27.597887    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb" podUID="bab59149-41b1-4ce7-b03e-074c8c13852a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 27 21:31:28.600571 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:28.600472    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb" podUID="bab59149-41b1-4ce7-b03e-074c8c13852a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 27 21:31:29.280421 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:29.280383    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:31:33.604865 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:33.604833    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb"
Jul 27 21:31:33.605478 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:33.605450    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb" podUID="bab59149-41b1-4ce7-b03e-074c8c13852a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 27 21:31:43.606882 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:43.606854    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb"
Jul 27 21:31:58.088681 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:58.088646    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb"]
Jul 27 21:31:58.089104 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:58.088896    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb" podUID="bab59149-41b1-4ce7-b03e-074c8c13852a" containerName="kserve-container" containerID="cri-o://79704f326aa917d2b50cc11db8f8f70c9b0afe3e462c435a84be4cb1c461f5aa" gracePeriod=30
Jul 27 21:31:58.089104 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:58.088922    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb" podUID="bab59149-41b1-4ce7-b03e-074c8c13852a" containerName="kube-rbac-proxy" containerID="cri-o://0bb6d51cd3b29def7c455522e3bba74ee92c66631c0049c79c486b64e3efcb0e" gracePeriod=30
Jul 27 21:31:58.118069 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:58.118044    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:31:58.601242 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:58.601204    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb" podUID="bab59149-41b1-4ce7-b03e-074c8c13852a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Jul 27 21:31:58.728418 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:58.728383    2577 generic.go:356] "Generic (PLEG): container finished" podID="bab59149-41b1-4ce7-b03e-074c8c13852a" containerID="0bb6d51cd3b29def7c455522e3bba74ee92c66631c0049c79c486b64e3efcb0e" exitCode=2
Jul 27 21:31:58.728592 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:58.728460    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb" event={"ID":"bab59149-41b1-4ce7-b03e-074c8c13852a","Type":"ContainerDied","Data":"0bb6d51cd3b29def7c455522e3bba74ee92c66631c0049c79c486b64e3efcb0e"}
Jul 27 21:31:59.637631 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:59.637602    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb"
Jul 27 21:31:59.692162 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:59.692100    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bab59149-41b1-4ce7-b03e-074c8c13852a-kserve-provision-location\") pod \"bab59149-41b1-4ce7-b03e-074c8c13852a\" (UID: \"bab59149-41b1-4ce7-b03e-074c8c13852a\") "
Jul 27 21:31:59.692162 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:59.692136    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bab59149-41b1-4ce7-b03e-074c8c13852a-proxy-tls\") pod \"bab59149-41b1-4ce7-b03e-074c8c13852a\" (UID: \"bab59149-41b1-4ce7-b03e-074c8c13852a\") "
Jul 27 21:31:59.692316 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:59.692172    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bab59149-41b1-4ce7-b03e-074c8c13852a-kube-rbac-proxy-sar-config\") pod \"bab59149-41b1-4ce7-b03e-074c8c13852a\" (UID: \"bab59149-41b1-4ce7-b03e-074c8c13852a\") "
Jul 27 21:31:59.692316 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:59.692210    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w8gmt\" (UniqueName: \"kubernetes.io/projected/bab59149-41b1-4ce7-b03e-074c8c13852a-kube-api-access-w8gmt\") pod \"bab59149-41b1-4ce7-b03e-074c8c13852a\" (UID: \"bab59149-41b1-4ce7-b03e-074c8c13852a\") "
Jul 27 21:31:59.692316 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:59.692255    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bab59149-41b1-4ce7-b03e-074c8c13852a-cabundle-cert\") pod \"bab59149-41b1-4ce7-b03e-074c8c13852a\" (UID: \"bab59149-41b1-4ce7-b03e-074c8c13852a\") "
Jul 27 21:31:59.692624 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:59.692516    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bab59149-41b1-4ce7-b03e-074c8c13852a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bab59149-41b1-4ce7-b03e-074c8c13852a" (UID: "bab59149-41b1-4ce7-b03e-074c8c13852a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:31:59.692714 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:59.692659    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bab59149-41b1-4ce7-b03e-074c8c13852a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bab59149-41b1-4ce7-b03e-074c8c13852a" (UID: "bab59149-41b1-4ce7-b03e-074c8c13852a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:31:59.693116 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:59.693094    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bab59149-41b1-4ce7-b03e-074c8c13852a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bab59149-41b1-4ce7-b03e-074c8c13852a" (UID: "bab59149-41b1-4ce7-b03e-074c8c13852a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:31:59.694134 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:59.694113    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bab59149-41b1-4ce7-b03e-074c8c13852a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bab59149-41b1-4ce7-b03e-074c8c13852a" (UID: "bab59149-41b1-4ce7-b03e-074c8c13852a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:31:59.694218 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:59.694197    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bab59149-41b1-4ce7-b03e-074c8c13852a-kube-api-access-w8gmt" (OuterVolumeSpecName: "kube-api-access-w8gmt") pod "bab59149-41b1-4ce7-b03e-074c8c13852a" (UID: "bab59149-41b1-4ce7-b03e-074c8c13852a"). InnerVolumeSpecName "kube-api-access-w8gmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:31:59.733514 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:59.733487    2577 generic.go:356] "Generic (PLEG): container finished" podID="bab59149-41b1-4ce7-b03e-074c8c13852a" containerID="79704f326aa917d2b50cc11db8f8f70c9b0afe3e462c435a84be4cb1c461f5aa" exitCode=0
Jul 27 21:31:59.733633 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:59.733522    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb" event={"ID":"bab59149-41b1-4ce7-b03e-074c8c13852a","Type":"ContainerDied","Data":"79704f326aa917d2b50cc11db8f8f70c9b0afe3e462c435a84be4cb1c461f5aa"}
Jul 27 21:31:59.733633 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:59.733548    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb"
Jul 27 21:31:59.733633 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:59.733578    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb" event={"ID":"bab59149-41b1-4ce7-b03e-074c8c13852a","Type":"ContainerDied","Data":"51c5a735405b7c57e3762d59f63c8497ed01e259d5c10ca38b1405072be04e91"}
Jul 27 21:31:59.733633 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:59.733607    2577 scope.go:117] "RemoveContainer" containerID="0bb6d51cd3b29def7c455522e3bba74ee92c66631c0049c79c486b64e3efcb0e"
Jul 27 21:31:59.741488 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:59.741472    2577 scope.go:117] "RemoveContainer" containerID="79704f326aa917d2b50cc11db8f8f70c9b0afe3e462c435a84be4cb1c461f5aa"
Jul 27 21:31:59.748436 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:59.748414    2577 scope.go:117] "RemoveContainer" containerID="000f9f53eb52a6263ebf412e1a887dc0557f3885af62120726b831dac98f4f48"
Jul 27 21:31:59.753626 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:59.753599    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb"]
Jul 27 21:31:59.755300 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:59.755270    2577 scope.go:117] "RemoveContainer" containerID="0bb6d51cd3b29def7c455522e3bba74ee92c66631c0049c79c486b64e3efcb0e"
Jul 27 21:31:59.755645 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:31:59.755603    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bb6d51cd3b29def7c455522e3bba74ee92c66631c0049c79c486b64e3efcb0e\": container with ID starting with 0bb6d51cd3b29def7c455522e3bba74ee92c66631c0049c79c486b64e3efcb0e not found: ID does not exist" containerID="0bb6d51cd3b29def7c455522e3bba74ee92c66631c0049c79c486b64e3efcb0e"
Jul 27 21:31:59.755816 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:59.755638    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bb6d51cd3b29def7c455522e3bba74ee92c66631c0049c79c486b64e3efcb0e"} err="failed to get container status \"0bb6d51cd3b29def7c455522e3bba74ee92c66631c0049c79c486b64e3efcb0e\": rpc error: code = NotFound desc = could not find container \"0bb6d51cd3b29def7c455522e3bba74ee92c66631c0049c79c486b64e3efcb0e\": container with ID starting with 0bb6d51cd3b29def7c455522e3bba74ee92c66631c0049c79c486b64e3efcb0e not found: ID does not exist"
Jul 27 21:31:59.755816 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:59.755659    2577 scope.go:117] "RemoveContainer" containerID="79704f326aa917d2b50cc11db8f8f70c9b0afe3e462c435a84be4cb1c461f5aa"
Jul 27 21:31:59.755935 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:31:59.755916    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79704f326aa917d2b50cc11db8f8f70c9b0afe3e462c435a84be4cb1c461f5aa\": container with ID starting with 79704f326aa917d2b50cc11db8f8f70c9b0afe3e462c435a84be4cb1c461f5aa not found: ID does not exist" containerID="79704f326aa917d2b50cc11db8f8f70c9b0afe3e462c435a84be4cb1c461f5aa"
Jul 27 21:31:59.755970 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:59.755939    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79704f326aa917d2b50cc11db8f8f70c9b0afe3e462c435a84be4cb1c461f5aa"} err="failed to get container status \"79704f326aa917d2b50cc11db8f8f70c9b0afe3e462c435a84be4cb1c461f5aa\": rpc error: code = NotFound desc = could not find container \"79704f326aa917d2b50cc11db8f8f70c9b0afe3e462c435a84be4cb1c461f5aa\": container with ID starting with 79704f326aa917d2b50cc11db8f8f70c9b0afe3e462c435a84be4cb1c461f5aa not found: ID does not exist"
Jul 27 21:31:59.755970 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:59.755954    2577 scope.go:117] "RemoveContainer" containerID="000f9f53eb52a6263ebf412e1a887dc0557f3885af62120726b831dac98f4f48"
Jul 27 21:31:59.756218 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:31:59.756200    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"000f9f53eb52a6263ebf412e1a887dc0557f3885af62120726b831dac98f4f48\": container with ID starting with 000f9f53eb52a6263ebf412e1a887dc0557f3885af62120726b831dac98f4f48 not found: ID does not exist" containerID="000f9f53eb52a6263ebf412e1a887dc0557f3885af62120726b831dac98f4f48"
Jul 27 21:31:59.756260 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:59.756222    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"000f9f53eb52a6263ebf412e1a887dc0557f3885af62120726b831dac98f4f48"} err="failed to get container status \"000f9f53eb52a6263ebf412e1a887dc0557f3885af62120726b831dac98f4f48\": rpc error: code = NotFound desc = could not find container \"000f9f53eb52a6263ebf412e1a887dc0557f3885af62120726b831dac98f4f48\": container with ID starting with 000f9f53eb52a6263ebf412e1a887dc0557f3885af62120726b831dac98f4f48 not found: ID does not exist"
Jul 27 21:31:59.757318 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:59.757297    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-gl8nb"]
Jul 27 21:31:59.793735 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:59.793711    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bab59149-41b1-4ce7-b03e-074c8c13852a-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:31:59.793735 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:59.793733    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bab59149-41b1-4ce7-b03e-074c8c13852a-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:31:59.793885 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:59.793752    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bab59149-41b1-4ce7-b03e-074c8c13852a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:31:59.793885 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:59.793789    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8gmt\" (UniqueName: \"kubernetes.io/projected/bab59149-41b1-4ce7-b03e-074c8c13852a-kube-api-access-w8gmt\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:31:59.793885 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:31:59.793804    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bab59149-41b1-4ce7-b03e-074c8c13852a-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:32:01.135110 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:01.135077    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bab59149-41b1-4ce7-b03e-074c8c13852a" path="/var/lib/kubelet/pods/bab59149-41b1-4ce7-b03e-074c8c13852a/volumes"
Jul 27 21:32:03.246148 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:03.246119    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:32:03.264844 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:03.264818    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w"]
Jul 27 21:32:03.265100 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:03.265081    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bab59149-41b1-4ce7-b03e-074c8c13852a" containerName="kserve-container"
Jul 27 21:32:03.265100 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:03.265096    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bab59149-41b1-4ce7-b03e-074c8c13852a" containerName="kserve-container"
Jul 27 21:32:03.265233 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:03.265106    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bab59149-41b1-4ce7-b03e-074c8c13852a" containerName="kube-rbac-proxy"
Jul 27 21:32:03.265233 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:03.265113    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bab59149-41b1-4ce7-b03e-074c8c13852a" containerName="kube-rbac-proxy"
Jul 27 21:32:03.265233 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:03.265133    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bab59149-41b1-4ce7-b03e-074c8c13852a" containerName="storage-initializer"
Jul 27 21:32:03.265233 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:03.265139    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bab59149-41b1-4ce7-b03e-074c8c13852a" containerName="storage-initializer"
Jul 27 21:32:03.265233 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:03.265194    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bab59149-41b1-4ce7-b03e-074c8c13852a" containerName="kube-rbac-proxy"
Jul 27 21:32:03.265233 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:03.265202    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bab59149-41b1-4ce7-b03e-074c8c13852a" containerName="kserve-container"
Jul 27 21:32:03.270020 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:03.270004    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w"
Jul 27 21:32:03.272576 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:03.272549    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 21:32:03.272751 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:03.272616    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v7c66\""
Jul 27 21:32:03.272751 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:03.272626    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 21:32:03.272751 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:03.272679    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-predictor-serving-cert\""
Jul 27 21:32:03.272751 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:03.272686    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-kube-rbac-proxy-sar-config\""
Jul 27 21:32:03.273010 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:03.272754    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 21:32:03.273010 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:03.272952    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 21:32:03.278950 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:03.278930    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w"]
Jul 27 21:32:03.319926 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:03.319902    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89b03fe7-285b-49b0-ae43-b49f9c4ca280-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w\" (UID: \"89b03fe7-285b-49b0-ae43-b49f9c4ca280\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w"
Jul 27 21:32:03.320022 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:03.319934    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvfzn\" (UniqueName: \"kubernetes.io/projected/89b03fe7-285b-49b0-ae43-b49f9c4ca280-kube-api-access-nvfzn\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w\" (UID: \"89b03fe7-285b-49b0-ae43-b49f9c4ca280\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w"
Jul 27 21:32:03.320022 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:03.319959    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89b03fe7-285b-49b0-ae43-b49f9c4ca280-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w\" (UID: \"89b03fe7-285b-49b0-ae43-b49f9c4ca280\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w"
Jul 27 21:32:03.320022 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:03.319975    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89b03fe7-285b-49b0-ae43-b49f9c4ca280-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w\" (UID: \"89b03fe7-285b-49b0-ae43-b49f9c4ca280\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w"
Jul 27 21:32:03.320143 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:03.320100    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89b03fe7-285b-49b0-ae43-b49f9c4ca280-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w\" (UID: \"89b03fe7-285b-49b0-ae43-b49f9c4ca280\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w"
Jul 27 21:32:03.421338 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:03.421318    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89b03fe7-285b-49b0-ae43-b49f9c4ca280-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w\" (UID: \"89b03fe7-285b-49b0-ae43-b49f9c4ca280\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w"
Jul 27 21:32:03.421433 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:03.421345    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89b03fe7-285b-49b0-ae43-b49f9c4ca280-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w\" (UID: \"89b03fe7-285b-49b0-ae43-b49f9c4ca280\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w"
Jul 27 21:32:03.421433 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:03.421381    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89b03fe7-285b-49b0-ae43-b49f9c4ca280-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w\" (UID: \"89b03fe7-285b-49b0-ae43-b49f9c4ca280\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w"
Jul 27 21:32:03.421524 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:32:03.421462    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-serving-cert: secret "isvc-autogluon-ts-v1-noslash-predictor-serving-cert" not found
Jul 27 21:32:03.421524 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:03.421496    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89b03fe7-285b-49b0-ae43-b49f9c4ca280-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w\" (UID: \"89b03fe7-285b-49b0-ae43-b49f9c4ca280\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w"
Jul 27 21:32:03.421524 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:32:03.421523    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89b03fe7-285b-49b0-ae43-b49f9c4ca280-proxy-tls podName:89b03fe7-285b-49b0-ae43-b49f9c4ca280 nodeName:}" failed. No retries permitted until 2026-07-27 21:32:03.921502445 +0000 UTC m=+1647.288692292 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/89b03fe7-285b-49b0-ae43-b49f9c4ca280-proxy-tls") pod "isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w" (UID: "89b03fe7-285b-49b0-ae43-b49f9c4ca280") : secret "isvc-autogluon-ts-v1-noslash-predictor-serving-cert" not found
Jul 27 21:32:03.421686 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:03.421573    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvfzn\" (UniqueName: \"kubernetes.io/projected/89b03fe7-285b-49b0-ae43-b49f9c4ca280-kube-api-access-nvfzn\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w\" (UID: \"89b03fe7-285b-49b0-ae43-b49f9c4ca280\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w"
Jul 27 21:32:03.421787 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:03.421747    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89b03fe7-285b-49b0-ae43-b49f9c4ca280-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w\" (UID: \"89b03fe7-285b-49b0-ae43-b49f9c4ca280\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w"
Jul 27 21:32:03.422073 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:03.422056    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89b03fe7-285b-49b0-ae43-b49f9c4ca280-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w\" (UID: \"89b03fe7-285b-49b0-ae43-b49f9c4ca280\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w"
Jul 27 21:32:03.422128 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:03.422112    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89b03fe7-285b-49b0-ae43-b49f9c4ca280-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w\" (UID: \"89b03fe7-285b-49b0-ae43-b49f9c4ca280\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w"
Jul 27 21:32:03.429312 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:03.429293    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvfzn\" (UniqueName: \"kubernetes.io/projected/89b03fe7-285b-49b0-ae43-b49f9c4ca280-kube-api-access-nvfzn\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w\" (UID: \"89b03fe7-285b-49b0-ae43-b49f9c4ca280\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w"
Jul 27 21:32:03.924275 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:03.924249    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89b03fe7-285b-49b0-ae43-b49f9c4ca280-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w\" (UID: \"89b03fe7-285b-49b0-ae43-b49f9c4ca280\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w"
Jul 27 21:32:03.926475 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:03.926450    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89b03fe7-285b-49b0-ae43-b49f9c4ca280-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w\" (UID: \"89b03fe7-285b-49b0-ae43-b49f9c4ca280\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w"
Jul 27 21:32:04.182076 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:04.182024    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w"
Jul 27 21:32:04.312477 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:04.312385    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w"]
Jul 27 21:32:04.315053 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:32:04.315017    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89b03fe7_285b_49b0_ae43_b49f9c4ca280.slice/crio-8bc0f58acb0312460f1bba10e58366c07094700de40373135e2b1b528772d6b1 WatchSource:0}: Error finding container 8bc0f58acb0312460f1bba10e58366c07094700de40373135e2b1b528772d6b1: Status 404 returned error can't find the container with id 8bc0f58acb0312460f1bba10e58366c07094700de40373135e2b1b528772d6b1
Jul 27 21:32:04.650263 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:04.650177    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:32:04.756237 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:04.756205    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w" event={"ID":"89b03fe7-285b-49b0-ae43-b49f9c4ca280","Type":"ContainerStarted","Data":"95045e1396512ca14edcb1707051feef7fc83255bf7c5151919abdf3d04777d2"}
Jul 27 21:32:04.756386 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:04.756258    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w" event={"ID":"89b03fe7-285b-49b0-ae43-b49f9c4ca280","Type":"ContainerStarted","Data":"8bc0f58acb0312460f1bba10e58366c07094700de40373135e2b1b528772d6b1"}
Jul 27 21:32:13.791064 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:13.791029    2577 generic.go:356] "Generic (PLEG): container finished" podID="89b03fe7-285b-49b0-ae43-b49f9c4ca280" containerID="95045e1396512ca14edcb1707051feef7fc83255bf7c5151919abdf3d04777d2" exitCode=0
Jul 27 21:32:13.791465 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:13.791097    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w" event={"ID":"89b03fe7-285b-49b0-ae43-b49f9c4ca280","Type":"ContainerDied","Data":"95045e1396512ca14edcb1707051feef7fc83255bf7c5151919abdf3d04777d2"}
Jul 27 21:32:14.796895 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:14.796855    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w" event={"ID":"89b03fe7-285b-49b0-ae43-b49f9c4ca280","Type":"ContainerStarted","Data":"0f18d09079869aead43735b4fbe8241913faf4c64769bb10caba2ddf384d90f3"}
Jul 27 21:32:14.797355 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:14.796929    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w" event={"ID":"89b03fe7-285b-49b0-ae43-b49f9c4ca280","Type":"ContainerStarted","Data":"6bef2fd2ba2c44d0a10a26f0432c32a83721eba63d83ba4335415a267e67308d"}
Jul 27 21:32:14.797355 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:14.797156    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w"
Jul 27 21:32:14.816767 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:14.816723    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w" podStartSLOduration=3.064068623 podStartE2EDuration="11.816708618s" totalImagesPullingTime="0s" totalInitContainerRuntime="8.752639995s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:32:14.815890762 +0000 UTC m=+1658.183080629" watchObservedRunningTime="2026-07-27 21:32:14.816708618 +0000 UTC m=+1658.183898484"
Jul 27 21:32:15.801095 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:15.801066    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w"
Jul 27 21:32:15.802206 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:15.802177    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w" podUID="89b03fe7-285b-49b0-ae43-b49f9c4ca280" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 27 21:32:16.805562 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:16.805517    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w" podUID="89b03fe7-285b-49b0-ae43-b49f9c4ca280" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 27 21:32:21.809938 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:21.809909    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w"
Jul 27 21:32:21.810424 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:21.810393    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w" podUID="89b03fe7-285b-49b0-ae43-b49f9c4ca280" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 27 21:32:31.810961 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:31.810928    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w"
Jul 27 21:32:38.308837 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:38.308800    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w"]
Jul 27 21:32:38.309270 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:38.309102    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w" podUID="89b03fe7-285b-49b0-ae43-b49f9c4ca280" containerName="kserve-container" containerID="cri-o://6bef2fd2ba2c44d0a10a26f0432c32a83721eba63d83ba4335415a267e67308d" gracePeriod=30
Jul 27 21:32:38.309270 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:38.309130    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w" podUID="89b03fe7-285b-49b0-ae43-b49f9c4ca280" containerName="kube-rbac-proxy" containerID="cri-o://0f18d09079869aead43735b4fbe8241913faf4c64769bb10caba2ddf384d90f3" gracePeriod=30
Jul 27 21:32:38.332058 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:38.332027    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:32:38.898801 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:38.898752    2577 generic.go:356] "Generic (PLEG): container finished" podID="89b03fe7-285b-49b0-ae43-b49f9c4ca280" containerID="0f18d09079869aead43735b4fbe8241913faf4c64769bb10caba2ddf384d90f3" exitCode=2
Jul 27 21:32:38.898985 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:38.898830    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w" event={"ID":"89b03fe7-285b-49b0-ae43-b49f9c4ca280","Type":"ContainerDied","Data":"0f18d09079869aead43735b4fbe8241913faf4c64769bb10caba2ddf384d90f3"}
Jul 27 21:32:39.551956 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:39.551932    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w"
Jul 27 21:32:39.683757 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:39.683736    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nvfzn\" (UniqueName: \"kubernetes.io/projected/89b03fe7-285b-49b0-ae43-b49f9c4ca280-kube-api-access-nvfzn\") pod \"89b03fe7-285b-49b0-ae43-b49f9c4ca280\" (UID: \"89b03fe7-285b-49b0-ae43-b49f9c4ca280\") "
Jul 27 21:32:39.683884 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:39.683803    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89b03fe7-285b-49b0-ae43-b49f9c4ca280-proxy-tls\") pod \"89b03fe7-285b-49b0-ae43-b49f9c4ca280\" (UID: \"89b03fe7-285b-49b0-ae43-b49f9c4ca280\") "
Jul 27 21:32:39.683927 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:39.683896    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89b03fe7-285b-49b0-ae43-b49f9c4ca280-cabundle-cert\") pod \"89b03fe7-285b-49b0-ae43-b49f9c4ca280\" (UID: \"89b03fe7-285b-49b0-ae43-b49f9c4ca280\") "
Jul 27 21:32:39.683967 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:39.683932    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89b03fe7-285b-49b0-ae43-b49f9c4ca280-kube-rbac-proxy-sar-config\") pod \"89b03fe7-285b-49b0-ae43-b49f9c4ca280\" (UID: \"89b03fe7-285b-49b0-ae43-b49f9c4ca280\") "
Jul 27 21:32:39.684013 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:39.683971    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89b03fe7-285b-49b0-ae43-b49f9c4ca280-kserve-provision-location\") pod \"89b03fe7-285b-49b0-ae43-b49f9c4ca280\" (UID: \"89b03fe7-285b-49b0-ae43-b49f9c4ca280\") "
Jul 27 21:32:39.684303 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:39.684268    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89b03fe7-285b-49b0-ae43-b49f9c4ca280-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "89b03fe7-285b-49b0-ae43-b49f9c4ca280" (UID: "89b03fe7-285b-49b0-ae43-b49f9c4ca280"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:32:39.684426 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:39.684308    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89b03fe7-285b-49b0-ae43-b49f9c4ca280-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "89b03fe7-285b-49b0-ae43-b49f9c4ca280" (UID: "89b03fe7-285b-49b0-ae43-b49f9c4ca280"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:32:39.684922 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:39.684902    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b03fe7-285b-49b0-ae43-b49f9c4ca280-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "89b03fe7-285b-49b0-ae43-b49f9c4ca280" (UID: "89b03fe7-285b-49b0-ae43-b49f9c4ca280"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:32:39.685714 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:39.685683    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89b03fe7-285b-49b0-ae43-b49f9c4ca280-kube-api-access-nvfzn" (OuterVolumeSpecName: "kube-api-access-nvfzn") pod "89b03fe7-285b-49b0-ae43-b49f9c4ca280" (UID: "89b03fe7-285b-49b0-ae43-b49f9c4ca280"). InnerVolumeSpecName "kube-api-access-nvfzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:32:39.685837 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:39.685739    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89b03fe7-285b-49b0-ae43-b49f9c4ca280-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "89b03fe7-285b-49b0-ae43-b49f9c4ca280" (UID: "89b03fe7-285b-49b0-ae43-b49f9c4ca280"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:32:39.784795 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:39.784750    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/89b03fe7-285b-49b0-ae43-b49f9c4ca280-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:32:39.784795 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:39.784794    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/89b03fe7-285b-49b0-ae43-b49f9c4ca280-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:32:39.784935 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:39.784811    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvfzn\" (UniqueName: \"kubernetes.io/projected/89b03fe7-285b-49b0-ae43-b49f9c4ca280-kube-api-access-nvfzn\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:32:39.784935 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:39.784832    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/89b03fe7-285b-49b0-ae43-b49f9c4ca280-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:32:39.784935 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:39.784840    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/89b03fe7-285b-49b0-ae43-b49f9c4ca280-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:32:39.904541 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:39.904516    2577 generic.go:356] "Generic (PLEG): container finished" podID="89b03fe7-285b-49b0-ae43-b49f9c4ca280" containerID="6bef2fd2ba2c44d0a10a26f0432c32a83721eba63d83ba4335415a267e67308d" exitCode=0
Jul 27 21:32:39.904642 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:39.904552    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w" event={"ID":"89b03fe7-285b-49b0-ae43-b49f9c4ca280","Type":"ContainerDied","Data":"6bef2fd2ba2c44d0a10a26f0432c32a83721eba63d83ba4335415a267e67308d"}
Jul 27 21:32:39.904642 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:39.904567    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w"
Jul 27 21:32:39.904642 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:39.904602    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w" event={"ID":"89b03fe7-285b-49b0-ae43-b49f9c4ca280","Type":"ContainerDied","Data":"8bc0f58acb0312460f1bba10e58366c07094700de40373135e2b1b528772d6b1"}
Jul 27 21:32:39.904642 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:39.904628    2577 scope.go:117] "RemoveContainer" containerID="0f18d09079869aead43735b4fbe8241913faf4c64769bb10caba2ddf384d90f3"
Jul 27 21:32:39.913260 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:39.913246    2577 scope.go:117] "RemoveContainer" containerID="6bef2fd2ba2c44d0a10a26f0432c32a83721eba63d83ba4335415a267e67308d"
Jul 27 21:32:39.920268 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:39.920253    2577 scope.go:117] "RemoveContainer" containerID="95045e1396512ca14edcb1707051feef7fc83255bf7c5151919abdf3d04777d2"
Jul 27 21:32:39.925341 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:39.925320    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w"]
Jul 27 21:32:39.927248 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:39.927234    2577 scope.go:117] "RemoveContainer" containerID="0f18d09079869aead43735b4fbe8241913faf4c64769bb10caba2ddf384d90f3"
Jul 27 21:32:39.927489 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:32:39.927468    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f18d09079869aead43735b4fbe8241913faf4c64769bb10caba2ddf384d90f3\": container with ID starting with 0f18d09079869aead43735b4fbe8241913faf4c64769bb10caba2ddf384d90f3 not found: ID does not exist" containerID="0f18d09079869aead43735b4fbe8241913faf4c64769bb10caba2ddf384d90f3"
Jul 27 21:32:39.927566 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:39.927499    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f18d09079869aead43735b4fbe8241913faf4c64769bb10caba2ddf384d90f3"} err="failed to get container status \"0f18d09079869aead43735b4fbe8241913faf4c64769bb10caba2ddf384d90f3\": rpc error: code = NotFound desc = could not find container \"0f18d09079869aead43735b4fbe8241913faf4c64769bb10caba2ddf384d90f3\": container with ID starting with 0f18d09079869aead43735b4fbe8241913faf4c64769bb10caba2ddf384d90f3 not found: ID does not exist"
Jul 27 21:32:39.927566 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:39.927556    2577 scope.go:117] "RemoveContainer" containerID="6bef2fd2ba2c44d0a10a26f0432c32a83721eba63d83ba4335415a267e67308d"
Jul 27 21:32:39.928154 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:32:39.928132    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bef2fd2ba2c44d0a10a26f0432c32a83721eba63d83ba4335415a267e67308d\": container with ID starting with 6bef2fd2ba2c44d0a10a26f0432c32a83721eba63d83ba4335415a267e67308d not found: ID does not exist" containerID="6bef2fd2ba2c44d0a10a26f0432c32a83721eba63d83ba4335415a267e67308d"
Jul 27 21:32:39.928254 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:39.928160    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bef2fd2ba2c44d0a10a26f0432c32a83721eba63d83ba4335415a267e67308d"} err="failed to get container status \"6bef2fd2ba2c44d0a10a26f0432c32a83721eba63d83ba4335415a267e67308d\": rpc error: code = NotFound desc = could not find container \"6bef2fd2ba2c44d0a10a26f0432c32a83721eba63d83ba4335415a267e67308d\": container with ID starting with 6bef2fd2ba2c44d0a10a26f0432c32a83721eba63d83ba4335415a267e67308d not found: ID does not exist"
Jul 27 21:32:39.928254 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:39.928181    2577 scope.go:117] "RemoveContainer" containerID="95045e1396512ca14edcb1707051feef7fc83255bf7c5151919abdf3d04777d2"
Jul 27 21:32:39.928435 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:32:39.928416    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95045e1396512ca14edcb1707051feef7fc83255bf7c5151919abdf3d04777d2\": container with ID starting with 95045e1396512ca14edcb1707051feef7fc83255bf7c5151919abdf3d04777d2 not found: ID does not exist" containerID="95045e1396512ca14edcb1707051feef7fc83255bf7c5151919abdf3d04777d2"
Jul 27 21:32:39.928495 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:39.928438    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95045e1396512ca14edcb1707051feef7fc83255bf7c5151919abdf3d04777d2"} err="failed to get container status \"95045e1396512ca14edcb1707051feef7fc83255bf7c5151919abdf3d04777d2\": rpc error: code = NotFound desc = could not find container \"95045e1396512ca14edcb1707051feef7fc83255bf7c5151919abdf3d04777d2\": container with ID starting with 95045e1396512ca14edcb1707051feef7fc83255bf7c5151919abdf3d04777d2 not found: ID does not exist"
Jul 27 21:32:39.929393 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:39.929374    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-n8c9w"]
Jul 27 21:32:41.134853 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:41.134822    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89b03fe7-285b-49b0-ae43-b49f9c4ca280" path="/var/lib/kubelet/pods/89b03fe7-285b-49b0-ae43-b49f9c4ca280/volumes"
Jul 27 21:32:43.486928 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.486883    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:32:43.537133 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.537094    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-msl2b"]
Jul 27 21:32:43.537481 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.537467    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89b03fe7-285b-49b0-ae43-b49f9c4ca280" containerName="kserve-container"
Jul 27 21:32:43.537534 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.537484    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="89b03fe7-285b-49b0-ae43-b49f9c4ca280" containerName="kserve-container"
Jul 27 21:32:43.537534 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.537496    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89b03fe7-285b-49b0-ae43-b49f9c4ca280" containerName="kube-rbac-proxy"
Jul 27 21:32:43.537534 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.537505    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="89b03fe7-285b-49b0-ae43-b49f9c4ca280" containerName="kube-rbac-proxy"
Jul 27 21:32:43.537645 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.537541    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89b03fe7-285b-49b0-ae43-b49f9c4ca280" containerName="storage-initializer"
Jul 27 21:32:43.537645 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.537550    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="89b03fe7-285b-49b0-ae43-b49f9c4ca280" containerName="storage-initializer"
Jul 27 21:32:43.537645 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.537615    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="89b03fe7-285b-49b0-ae43-b49f9c4ca280" containerName="kube-rbac-proxy"
Jul 27 21:32:43.537645 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.537632    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="89b03fe7-285b-49b0-ae43-b49f9c4ca280" containerName="kserve-container"
Jul 27 21:32:43.542823 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.542803    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-msl2b"
Jul 27 21:32:43.543487 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.543465    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-25vpb"]
Jul 27 21:32:43.545178 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.545156    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Jul 27 21:32:43.545310 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.545281    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v7c66\""
Jul 27 21:32:43.545451 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.545368    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 21:32:43.545521 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.545487    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Jul 27 21:32:43.546064 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.546038    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 21:32:43.546174 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.546064    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 21:32:43.546174 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.546131    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 21:32:43.546550 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.546530    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-25vpb"
Jul 27 21:32:43.550142 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.550127    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Jul 27 21:32:43.554302 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.554282    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-msl2b"]
Jul 27 21:32:43.561992 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.561969    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-25vpb"]
Jul 27 21:32:43.570326 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.570310    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:32:43.669005 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.668974    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:32:43.709795 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.709750    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2ced923-c791-4a9a-bae9-17ecadd68c2c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-cb85cf55c-msl2b\" (UID: \"c2ced923-c791-4a9a-bae9-17ecadd68c2c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-msl2b"
Jul 27 21:32:43.709911 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.709802    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2ced923-c791-4a9a-bae9-17ecadd68c2c-proxy-tls\") pod \"isvc-canary-create-predictor-cb85cf55c-msl2b\" (UID: \"c2ced923-c791-4a9a-bae9-17ecadd68c2c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-msl2b"
Jul 27 21:32:43.709911 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.709818    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdsht\" (UniqueName: \"kubernetes.io/projected/c2ced923-c791-4a9a-bae9-17ecadd68c2c-kube-api-access-kdsht\") pod \"isvc-canary-create-predictor-cb85cf55c-msl2b\" (UID: \"c2ced923-c791-4a9a-bae9-17ecadd68c2c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-msl2b"
Jul 27 21:32:43.709911 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.709878    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e81f537c-a850-4658-886c-c6480ef951d1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-25vpb\" (UID: \"e81f537c-a850-4658-886c-c6480ef951d1\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-25vpb"
Jul 27 21:32:43.709911 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.709904    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e81f537c-a850-4658-886c-c6480ef951d1-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-25vpb\" (UID: \"e81f537c-a850-4658-886c-c6480ef951d1\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-25vpb"
Jul 27 21:32:43.710051 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.709931    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g2dl\" (UniqueName: \"kubernetes.io/projected/e81f537c-a850-4658-886c-c6480ef951d1-kube-api-access-6g2dl\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-25vpb\" (UID: \"e81f537c-a850-4658-886c-c6480ef951d1\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-25vpb"
Jul 27 21:32:43.710051 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.709961    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e81f537c-a850-4658-886c-c6480ef951d1-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-25vpb\" (UID: \"e81f537c-a850-4658-886c-c6480ef951d1\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-25vpb"
Jul 27 21:32:43.710051 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.710010    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2ced923-c791-4a9a-bae9-17ecadd68c2c-kserve-provision-location\") pod \"isvc-canary-create-predictor-cb85cf55c-msl2b\" (UID: \"c2ced923-c791-4a9a-bae9-17ecadd68c2c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-msl2b"
Jul 27 21:32:43.710051 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.710039    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e81f537c-a850-4658-886c-c6480ef951d1-proxy-tls\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-25vpb\" (UID: \"e81f537c-a850-4658-886c-c6480ef951d1\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-25vpb"
Jul 27 21:32:43.710173 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.710057    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c2ced923-c791-4a9a-bae9-17ecadd68c2c-cabundle-cert\") pod \"isvc-canary-create-predictor-cb85cf55c-msl2b\" (UID: \"c2ced923-c791-4a9a-bae9-17ecadd68c2c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-msl2b"
Jul 27 21:32:43.804250 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.804177    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:32:43.811336 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.811316    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2ced923-c791-4a9a-bae9-17ecadd68c2c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-cb85cf55c-msl2b\" (UID: \"c2ced923-c791-4a9a-bae9-17ecadd68c2c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-msl2b"
Jul 27 21:32:43.811423 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.811350    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2ced923-c791-4a9a-bae9-17ecadd68c2c-proxy-tls\") pod \"isvc-canary-create-predictor-cb85cf55c-msl2b\" (UID: \"c2ced923-c791-4a9a-bae9-17ecadd68c2c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-msl2b"
Jul 27 21:32:43.811423 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.811368    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdsht\" (UniqueName: \"kubernetes.io/projected/c2ced923-c791-4a9a-bae9-17ecadd68c2c-kube-api-access-kdsht\") pod \"isvc-canary-create-predictor-cb85cf55c-msl2b\" (UID: \"c2ced923-c791-4a9a-bae9-17ecadd68c2c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-msl2b"
Jul 27 21:32:43.811577 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.811561    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e81f537c-a850-4658-886c-c6480ef951d1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-25vpb\" (UID: \"e81f537c-a850-4658-886c-c6480ef951d1\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-25vpb"
Jul 27 21:32:43.811660 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.811626    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e81f537c-a850-4658-886c-c6480ef951d1-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-25vpb\" (UID: \"e81f537c-a850-4658-886c-c6480ef951d1\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-25vpb"
Jul 27 21:32:43.811714 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.811661    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g2dl\" (UniqueName: \"kubernetes.io/projected/e81f537c-a850-4658-886c-c6480ef951d1-kube-api-access-6g2dl\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-25vpb\" (UID: \"e81f537c-a850-4658-886c-c6480ef951d1\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-25vpb"
Jul 27 21:32:43.811779 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.811747    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e81f537c-a850-4658-886c-c6480ef951d1-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-25vpb\" (UID: \"e81f537c-a850-4658-886c-c6480ef951d1\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-25vpb"
Jul 27 21:32:43.811993 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.811968    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2ced923-c791-4a9a-bae9-17ecadd68c2c-kserve-provision-location\") pod \"isvc-canary-create-predictor-cb85cf55c-msl2b\" (UID: \"c2ced923-c791-4a9a-bae9-17ecadd68c2c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-msl2b"
Jul 27 21:32:43.812106 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.812048    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e81f537c-a850-4658-886c-c6480ef951d1-proxy-tls\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-25vpb\" (UID: \"e81f537c-a850-4658-886c-c6480ef951d1\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-25vpb"
Jul 27 21:32:43.812106 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.812087    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e81f537c-a850-4658-886c-c6480ef951d1-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-25vpb\" (UID: \"e81f537c-a850-4658-886c-c6480ef951d1\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-25vpb"
Jul 27 21:32:43.812106 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.812082    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c2ced923-c791-4a9a-bae9-17ecadd68c2c-cabundle-cert\") pod \"isvc-canary-create-predictor-cb85cf55c-msl2b\" (UID: \"c2ced923-c791-4a9a-bae9-17ecadd68c2c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-msl2b"
Jul 27 21:32:43.812288 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.812233    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2ced923-c791-4a9a-bae9-17ecadd68c2c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-cb85cf55c-msl2b\" (UID: \"c2ced923-c791-4a9a-bae9-17ecadd68c2c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-msl2b"
Jul 27 21:32:43.812288 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.812271    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e81f537c-a850-4658-886c-c6480ef951d1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-25vpb\" (UID: \"e81f537c-a850-4658-886c-c6480ef951d1\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-25vpb"
Jul 27 21:32:43.812367 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.812292    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e81f537c-a850-4658-886c-c6480ef951d1-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-25vpb\" (UID: \"e81f537c-a850-4658-886c-c6480ef951d1\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-25vpb"
Jul 27 21:32:43.812367 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.812357    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2ced923-c791-4a9a-bae9-17ecadd68c2c-kserve-provision-location\") pod \"isvc-canary-create-predictor-cb85cf55c-msl2b\" (UID: \"c2ced923-c791-4a9a-bae9-17ecadd68c2c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-msl2b"
Jul 27 21:32:43.812777 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.812735    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c2ced923-c791-4a9a-bae9-17ecadd68c2c-cabundle-cert\") pod \"isvc-canary-create-predictor-cb85cf55c-msl2b\" (UID: \"c2ced923-c791-4a9a-bae9-17ecadd68c2c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-msl2b"
Jul 27 21:32:43.814003 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.813985    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2ced923-c791-4a9a-bae9-17ecadd68c2c-proxy-tls\") pod \"isvc-canary-create-predictor-cb85cf55c-msl2b\" (UID: \"c2ced923-c791-4a9a-bae9-17ecadd68c2c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-msl2b"
Jul 27 21:32:43.814417 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.814392    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e81f537c-a850-4658-886c-c6480ef951d1-proxy-tls\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-25vpb\" (UID: \"e81f537c-a850-4658-886c-c6480ef951d1\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-25vpb"
Jul 27 21:32:43.819451 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.819433    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdsht\" (UniqueName: \"kubernetes.io/projected/c2ced923-c791-4a9a-bae9-17ecadd68c2c-kube-api-access-kdsht\") pod \"isvc-canary-create-predictor-cb85cf55c-msl2b\" (UID: \"c2ced923-c791-4a9a-bae9-17ecadd68c2c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-msl2b"
Jul 27 21:32:43.819614 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.819596    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g2dl\" (UniqueName: \"kubernetes.io/projected/e81f537c-a850-4658-886c-c6480ef951d1-kube-api-access-6g2dl\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-25vpb\" (UID: \"e81f537c-a850-4658-886c-c6480ef951d1\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-25vpb"
Jul 27 21:32:43.856549 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.856530    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-msl2b"
Jul 27 21:32:43.863755 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.863734    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-25vpb"
Jul 27 21:32:43.994170 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:43.994147    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-msl2b"]
Jul 27 21:32:43.996236 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:32:43.996205    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2ced923_c791_4a9a_bae9_17ecadd68c2c.slice/crio-a032749fcbd260a8d9d6b9bf26b7e9974385ba4af257a5e8cce183015c1de136 WatchSource:0}: Error finding container a032749fcbd260a8d9d6b9bf26b7e9974385ba4af257a5e8cce183015c1de136: Status 404 returned error can't find the container with id a032749fcbd260a8d9d6b9bf26b7e9974385ba4af257a5e8cce183015c1de136
Jul 27 21:32:44.016026 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:44.015999    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-25vpb"]
Jul 27 21:32:44.027682 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:32:44.027654    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode81f537c_a850_4658_886c_c6480ef951d1.slice/crio-6bc26fc60c63ac60120a28126c1a46d7082287d5318ee335599d4dc3a0285e4e WatchSource:0}: Error finding container 6bc26fc60c63ac60120a28126c1a46d7082287d5318ee335599d4dc3a0285e4e: Status 404 returned error can't find the container with id 6bc26fc60c63ac60120a28126c1a46d7082287d5318ee335599d4dc3a0285e4e
Jul 27 21:32:44.933303 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:44.933271    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-25vpb" event={"ID":"e81f537c-a850-4658-886c-c6480ef951d1","Type":"ContainerStarted","Data":"50be5d19ca791d9da2e22dac2dd7ba04acb31558d2ddd0d7c12524bdb99a5769"}
Jul 27 21:32:44.933834 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:44.933335    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-25vpb" event={"ID":"e81f537c-a850-4658-886c-c6480ef951d1","Type":"ContainerStarted","Data":"6bc26fc60c63ac60120a28126c1a46d7082287d5318ee335599d4dc3a0285e4e"}
Jul 27 21:32:44.934812 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:44.934788    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-msl2b" event={"ID":"c2ced923-c791-4a9a-bae9-17ecadd68c2c","Type":"ContainerStarted","Data":"523bbdcef8920ad1c07efa46760592d7fd090e957989bf99340eaf688bee141b"}
Jul 27 21:32:44.934928 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:44.934838    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-msl2b" event={"ID":"c2ced923-c791-4a9a-bae9-17ecadd68c2c","Type":"ContainerStarted","Data":"a032749fcbd260a8d9d6b9bf26b7e9974385ba4af257a5e8cce183015c1de136"}
Jul 27 21:32:47.947247 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:47.947216    2577 generic.go:356] "Generic (PLEG): container finished" podID="e81f537c-a850-4658-886c-c6480ef951d1" containerID="50be5d19ca791d9da2e22dac2dd7ba04acb31558d2ddd0d7c12524bdb99a5769" exitCode=0
Jul 27 21:32:47.947562 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:47.947286    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-25vpb" event={"ID":"e81f537c-a850-4658-886c-c6480ef951d1","Type":"ContainerDied","Data":"50be5d19ca791d9da2e22dac2dd7ba04acb31558d2ddd0d7c12524bdb99a5769"}
Jul 27 21:32:47.948674 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:47.948656    2577 generic.go:356] "Generic (PLEG): container finished" podID="c2ced923-c791-4a9a-bae9-17ecadd68c2c" containerID="523bbdcef8920ad1c07efa46760592d7fd090e957989bf99340eaf688bee141b" exitCode=0
Jul 27 21:32:47.948745 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:47.948725    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-msl2b" event={"ID":"c2ced923-c791-4a9a-bae9-17ecadd68c2c","Type":"ContainerDied","Data":"523bbdcef8920ad1c07efa46760592d7fd090e957989bf99340eaf688bee141b"}
Jul 27 21:32:48.954860 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:48.954826    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-25vpb" event={"ID":"e81f537c-a850-4658-886c-c6480ef951d1","Type":"ContainerStarted","Data":"b243548519d00b729b2bc7d017e7949b48da359f29229d4383b0113115033bc3"}
Jul 27 21:32:48.955314 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:48.954898    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-25vpb" event={"ID":"e81f537c-a850-4658-886c-c6480ef951d1","Type":"ContainerStarted","Data":"864b9fe9eaa0839da8fe1e36fc431e936eed2c46d498130a242e8ffc5bee918e"}
Jul 27 21:32:48.955314 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:48.955189    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-25vpb"
Jul 27 21:32:48.955314 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:48.955241    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-25vpb"
Jul 27 21:32:48.956805 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:48.956774    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-25vpb" podUID="e81f537c-a850-4658-886c-c6480ef951d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 27 21:32:48.956930 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:48.956811    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-msl2b" event={"ID":"c2ced923-c791-4a9a-bae9-17ecadd68c2c","Type":"ContainerStarted","Data":"246e8a5ebfaa37c992be7d4ba6b22cb02465a5d66bbd3987c5caa68aa9f7a4a2"}
Jul 27 21:32:48.956930 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:48.956852    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-msl2b" event={"ID":"c2ced923-c791-4a9a-bae9-17ecadd68c2c","Type":"ContainerStarted","Data":"f5f1bc1309fd0fe435fc1e5e02da9e3060a9808b8eb561da738de50af0fa77c5"}
Jul 27 21:32:48.957102 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:48.957084    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-msl2b"
Jul 27 21:32:48.957156 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:48.957136    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-msl2b"
Jul 27 21:32:48.958058 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:48.958037    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-msl2b" podUID="c2ced923-c791-4a9a-bae9-17ecadd68c2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 27 21:32:48.973655 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:48.973614    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-25vpb" podStartSLOduration=2.145441737 podStartE2EDuration="5.973594858s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.828153121s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:32:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:32:48.972276819 +0000 UTC m=+1692.339466698" watchObservedRunningTime="2026-07-27 21:32:48.973594858 +0000 UTC m=+1692.340784723"
Jul 27 21:32:48.991239 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:48.991204    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-msl2b" podStartSLOduration=2.4972313059999998 podStartE2EDuration="5.991192456s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.49396115s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:32:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:32:48.990488653 +0000 UTC m=+1692.357678519" watchObservedRunningTime="2026-07-27 21:32:48.991192456 +0000 UTC m=+1692.358382392"
Jul 27 21:32:49.960599 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:49.960553    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-msl2b" podUID="c2ced923-c791-4a9a-bae9-17ecadd68c2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 27 21:32:49.961001 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:49.960667    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-25vpb" podUID="e81f537c-a850-4658-886c-c6480ef951d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 27 21:32:54.964525 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:54.964496    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-msl2b"
Jul 27 21:32:54.964995 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:54.964976    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-25vpb"
Jul 27 21:32:54.965059 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:54.965030    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-msl2b" podUID="c2ced923-c791-4a9a-bae9-17ecadd68c2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 27 21:32:54.965433 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:32:54.965406    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-25vpb" podUID="e81f537c-a850-4658-886c-c6480ef951d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 27 21:33:04.964976 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:33:04.964899    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-msl2b" podUID="c2ced923-c791-4a9a-bae9-17ecadd68c2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 27 21:33:04.965336 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:33:04.965311    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-25vpb" podUID="e81f537c-a850-4658-886c-c6480ef951d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 27 21:33:14.965493 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:33:14.965454    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-25vpb" podUID="e81f537c-a850-4658-886c-c6480ef951d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 27 21:33:14.965493 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:33:14.965486    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-msl2b" podUID="c2ced923-c791-4a9a-bae9-17ecadd68c2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 27 21:33:24.965722 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:33:24.965679    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-25vpb" podUID="e81f537c-a850-4658-886c-c6480ef951d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 27 21:33:24.966163 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:33:24.965715    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-msl2b" podUID="c2ced923-c791-4a9a-bae9-17ecadd68c2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 27 21:33:34.965531 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:33:34.965494    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-25vpb" podUID="e81f537c-a850-4658-886c-c6480ef951d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 27 21:33:34.965950 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:33:34.965495    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-msl2b" podUID="c2ced923-c791-4a9a-bae9-17ecadd68c2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 27 21:33:44.965754 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:33:44.965705    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-msl2b" podUID="c2ced923-c791-4a9a-bae9-17ecadd68c2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 27 21:33:44.965754 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:33:44.965722    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-25vpb" podUID="e81f537c-a850-4658-886c-c6480ef951d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 27 21:33:48.262303 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:33:48.262270    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:33:54.965064 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:33:54.964995    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-msl2b" podUID="c2ced923-c791-4a9a-bae9-17ecadd68c2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 27 21:33:54.965603 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:33:54.965360    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-25vpb" podUID="e81f537c-a850-4658-886c-c6480ef951d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 27 21:34:00.130951 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:00.130922    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-25vpb"
Jul 27 21:34:04.965469 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:04.965445    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-msl2b"
Jul 27 21:34:13.607268 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:13.607227    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-msl2b"]
Jul 27 21:34:13.607705 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:13.607491    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-msl2b" podUID="c2ced923-c791-4a9a-bae9-17ecadd68c2c" containerName="kserve-container" containerID="cri-o://f5f1bc1309fd0fe435fc1e5e02da9e3060a9808b8eb561da738de50af0fa77c5" gracePeriod=30
Jul 27 21:34:13.607705 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:13.607517    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-msl2b" podUID="c2ced923-c791-4a9a-bae9-17ecadd68c2c" containerName="kube-rbac-proxy" containerID="cri-o://246e8a5ebfaa37c992be7d4ba6b22cb02465a5d66bbd3987c5caa68aa9f7a4a2" gracePeriod=30
Jul 27 21:34:13.612485 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:13.612454    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-25vpb"]
Jul 27 21:34:13.612838 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:13.612783    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-25vpb" podUID="e81f537c-a850-4658-886c-c6480ef951d1" containerName="kserve-container" containerID="cri-o://864b9fe9eaa0839da8fe1e36fc431e936eed2c46d498130a242e8ffc5bee918e" gracePeriod=30
Jul 27 21:34:13.612838 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:13.612826    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-25vpb" podUID="e81f537c-a850-4658-886c-c6480ef951d1" containerName="kube-rbac-proxy" containerID="cri-o://b243548519d00b729b2bc7d017e7949b48da359f29229d4383b0113115033bc3" gracePeriod=30
Jul 27 21:34:13.632130 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:13.632107    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:34:13.691587 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:13.691544    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:34:14.297660 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:14.297632    2577 generic.go:356] "Generic (PLEG): container finished" podID="e81f537c-a850-4658-886c-c6480ef951d1" containerID="b243548519d00b729b2bc7d017e7949b48da359f29229d4383b0113115033bc3" exitCode=2
Jul 27 21:34:14.297879 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:14.297697    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-25vpb" event={"ID":"e81f537c-a850-4658-886c-c6480ef951d1","Type":"ContainerDied","Data":"b243548519d00b729b2bc7d017e7949b48da359f29229d4383b0113115033bc3"}
Jul 27 21:34:14.299654 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:14.299630    2577 generic.go:356] "Generic (PLEG): container finished" podID="c2ced923-c791-4a9a-bae9-17ecadd68c2c" containerID="246e8a5ebfaa37c992be7d4ba6b22cb02465a5d66bbd3987c5caa68aa9f7a4a2" exitCode=2
Jul 27 21:34:14.299791 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:14.299679    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-msl2b" event={"ID":"c2ced923-c791-4a9a-bae9-17ecadd68c2c","Type":"ContainerDied","Data":"246e8a5ebfaa37c992be7d4ba6b22cb02465a5d66bbd3987c5caa68aa9f7a4a2"}
Jul 27 21:34:14.961038 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:14.960999    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-25vpb" podUID="e81f537c-a850-4658-886c-c6480ef951d1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Jul 27 21:34:14.961414 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:14.960999    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-msl2b" podUID="c2ced923-c791-4a9a-bae9-17ecadd68c2c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Jul 27 21:34:14.965264 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:14.965245    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-msl2b" podUID="c2ced923-c791-4a9a-bae9-17ecadd68c2c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 27 21:34:17.968030 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:17.967978    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-msl2b"
Jul 27 21:34:17.970574 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:17.970555    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-25vpb"
Jul 27 21:34:18.033732 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.033705    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e81f537c-a850-4658-886c-c6480ef951d1-proxy-tls\") pod \"e81f537c-a850-4658-886c-c6480ef951d1\" (UID: \"e81f537c-a850-4658-886c-c6480ef951d1\") "
Jul 27 21:34:18.033893 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.033740    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kdsht\" (UniqueName: \"kubernetes.io/projected/c2ced923-c791-4a9a-bae9-17ecadd68c2c-kube-api-access-kdsht\") pod \"c2ced923-c791-4a9a-bae9-17ecadd68c2c\" (UID: \"c2ced923-c791-4a9a-bae9-17ecadd68c2c\") "
Jul 27 21:34:18.033893 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.033773    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6g2dl\" (UniqueName: \"kubernetes.io/projected/e81f537c-a850-4658-886c-c6480ef951d1-kube-api-access-6g2dl\") pod \"e81f537c-a850-4658-886c-c6480ef951d1\" (UID: \"e81f537c-a850-4658-886c-c6480ef951d1\") "
Jul 27 21:34:18.033893 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.033811    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2ced923-c791-4a9a-bae9-17ecadd68c2c-kserve-provision-location\") pod \"c2ced923-c791-4a9a-bae9-17ecadd68c2c\" (UID: \"c2ced923-c791-4a9a-bae9-17ecadd68c2c\") "
Jul 27 21:34:18.033893 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.033832    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2ced923-c791-4a9a-bae9-17ecadd68c2c-proxy-tls\") pod \"c2ced923-c791-4a9a-bae9-17ecadd68c2c\" (UID: \"c2ced923-c791-4a9a-bae9-17ecadd68c2c\") "
Jul 27 21:34:18.033893 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.033866    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e81f537c-a850-4658-886c-c6480ef951d1-kube-rbac-proxy-sar-config\") pod \"e81f537c-a850-4658-886c-c6480ef951d1\" (UID: \"e81f537c-a850-4658-886c-c6480ef951d1\") "
Jul 27 21:34:18.033893 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.033889    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c2ced923-c791-4a9a-bae9-17ecadd68c2c-cabundle-cert\") pod \"c2ced923-c791-4a9a-bae9-17ecadd68c2c\" (UID: \"c2ced923-c791-4a9a-bae9-17ecadd68c2c\") "
Jul 27 21:34:18.034225 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.034162    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2ced923-c791-4a9a-bae9-17ecadd68c2c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c2ced923-c791-4a9a-bae9-17ecadd68c2c" (UID: "c2ced923-c791-4a9a-bae9-17ecadd68c2c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:34:18.034285 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.034226    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e81f537c-a850-4658-886c-c6480ef951d1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e81f537c-a850-4658-886c-c6480ef951d1" (UID: "e81f537c-a850-4658-886c-c6480ef951d1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:34:18.034285 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.034244    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2ced923-c791-4a9a-bae9-17ecadd68c2c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c2ced923-c791-4a9a-bae9-17ecadd68c2c" (UID: "c2ced923-c791-4a9a-bae9-17ecadd68c2c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:34:18.034397 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.034291    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e81f537c-a850-4658-886c-c6480ef951d1-cabundle-cert\") pod \"e81f537c-a850-4658-886c-c6480ef951d1\" (UID: \"e81f537c-a850-4658-886c-c6480ef951d1\") "
Jul 27 21:34:18.034397 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.034327    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e81f537c-a850-4658-886c-c6480ef951d1-kserve-provision-location\") pod \"e81f537c-a850-4658-886c-c6480ef951d1\" (UID: \"e81f537c-a850-4658-886c-c6480ef951d1\") "
Jul 27 21:34:18.034397 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.034357    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2ced923-c791-4a9a-bae9-17ecadd68c2c-kube-rbac-proxy-sar-config\") pod \"c2ced923-c791-4a9a-bae9-17ecadd68c2c\" (UID: \"c2ced923-c791-4a9a-bae9-17ecadd68c2c\") "
Jul 27 21:34:18.034922 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.034674    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2ced923-c791-4a9a-bae9-17ecadd68c2c-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:34:18.034922 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.034697    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e81f537c-a850-4658-886c-c6480ef951d1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:34:18.034922 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.034712    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c2ced923-c791-4a9a-bae9-17ecadd68c2c-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:34:18.034922 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.034740    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e81f537c-a850-4658-886c-c6480ef951d1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e81f537c-a850-4658-886c-c6480ef951d1" (UID: "e81f537c-a850-4658-886c-c6480ef951d1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:34:18.034922 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.034837    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e81f537c-a850-4658-886c-c6480ef951d1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e81f537c-a850-4658-886c-c6480ef951d1" (UID: "e81f537c-a850-4658-886c-c6480ef951d1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:34:18.034922 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.034888    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2ced923-c791-4a9a-bae9-17ecadd68c2c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c2ced923-c791-4a9a-bae9-17ecadd68c2c" (UID: "c2ced923-c791-4a9a-bae9-17ecadd68c2c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:34:18.036056 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.036038    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2ced923-c791-4a9a-bae9-17ecadd68c2c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c2ced923-c791-4a9a-bae9-17ecadd68c2c" (UID: "c2ced923-c791-4a9a-bae9-17ecadd68c2c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:34:18.036129 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.036090    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2ced923-c791-4a9a-bae9-17ecadd68c2c-kube-api-access-kdsht" (OuterVolumeSpecName: "kube-api-access-kdsht") pod "c2ced923-c791-4a9a-bae9-17ecadd68c2c" (UID: "c2ced923-c791-4a9a-bae9-17ecadd68c2c"). InnerVolumeSpecName "kube-api-access-kdsht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:34:18.036259 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.036233    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e81f537c-a850-4658-886c-c6480ef951d1-kube-api-access-6g2dl" (OuterVolumeSpecName: "kube-api-access-6g2dl") pod "e81f537c-a850-4658-886c-c6480ef951d1" (UID: "e81f537c-a850-4658-886c-c6480ef951d1"). InnerVolumeSpecName "kube-api-access-6g2dl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:34:18.036306 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.036234    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e81f537c-a850-4658-886c-c6480ef951d1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e81f537c-a850-4658-886c-c6480ef951d1" (UID: "e81f537c-a850-4658-886c-c6480ef951d1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:34:18.135244 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.135221    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2ced923-c791-4a9a-bae9-17ecadd68c2c-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:34:18.135244 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.135243    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e81f537c-a850-4658-886c-c6480ef951d1-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:34:18.135364 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.135256    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e81f537c-a850-4658-886c-c6480ef951d1-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:34:18.135364 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.135268    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2ced923-c791-4a9a-bae9-17ecadd68c2c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:34:18.135364 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.135281    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e81f537c-a850-4658-886c-c6480ef951d1-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:34:18.135364 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.135293    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdsht\" (UniqueName: \"kubernetes.io/projected/c2ced923-c791-4a9a-bae9-17ecadd68c2c-kube-api-access-kdsht\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:34:18.135364 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.135307    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g2dl\" (UniqueName: \"kubernetes.io/projected/e81f537c-a850-4658-886c-c6480ef951d1-kube-api-access-6g2dl\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:34:18.317943 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.317865    2577 generic.go:356] "Generic (PLEG): container finished" podID="e81f537c-a850-4658-886c-c6480ef951d1" containerID="864b9fe9eaa0839da8fe1e36fc431e936eed2c46d498130a242e8ffc5bee918e" exitCode=0
Jul 27 21:34:18.317943 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.317928    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-25vpb"
Jul 27 21:34:18.317943 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.317939    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-25vpb" event={"ID":"e81f537c-a850-4658-886c-c6480ef951d1","Type":"ContainerDied","Data":"864b9fe9eaa0839da8fe1e36fc431e936eed2c46d498130a242e8ffc5bee918e"}
Jul 27 21:34:18.318190 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.317994    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-25vpb" event={"ID":"e81f537c-a850-4658-886c-c6480ef951d1","Type":"ContainerDied","Data":"6bc26fc60c63ac60120a28126c1a46d7082287d5318ee335599d4dc3a0285e4e"}
Jul 27 21:34:18.318190 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.318029    2577 scope.go:117] "RemoveContainer" containerID="b243548519d00b729b2bc7d017e7949b48da359f29229d4383b0113115033bc3"
Jul 27 21:34:18.319837 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.319811    2577 generic.go:356] "Generic (PLEG): container finished" podID="c2ced923-c791-4a9a-bae9-17ecadd68c2c" containerID="f5f1bc1309fd0fe435fc1e5e02da9e3060a9808b8eb561da738de50af0fa77c5" exitCode=0
Jul 27 21:34:18.319949 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.319867    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-msl2b"
Jul 27 21:34:18.319949 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.319895    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-msl2b" event={"ID":"c2ced923-c791-4a9a-bae9-17ecadd68c2c","Type":"ContainerDied","Data":"f5f1bc1309fd0fe435fc1e5e02da9e3060a9808b8eb561da738de50af0fa77c5"}
Jul 27 21:34:18.320069 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.319958    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-msl2b" event={"ID":"c2ced923-c791-4a9a-bae9-17ecadd68c2c","Type":"ContainerDied","Data":"a032749fcbd260a8d9d6b9bf26b7e9974385ba4af257a5e8cce183015c1de136"}
Jul 27 21:34:18.326608 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.326597    2577 scope.go:117] "RemoveContainer" containerID="864b9fe9eaa0839da8fe1e36fc431e936eed2c46d498130a242e8ffc5bee918e"
Jul 27 21:34:18.334177 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.334163    2577 scope.go:117] "RemoveContainer" containerID="50be5d19ca791d9da2e22dac2dd7ba04acb31558d2ddd0d7c12524bdb99a5769"
Jul 27 21:34:18.341493 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.341476    2577 scope.go:117] "RemoveContainer" containerID="b243548519d00b729b2bc7d017e7949b48da359f29229d4383b0113115033bc3"
Jul 27 21:34:18.341698 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.341675    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-25vpb"]
Jul 27 21:34:18.341857 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:34:18.341739    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b243548519d00b729b2bc7d017e7949b48da359f29229d4383b0113115033bc3\": container with ID starting with b243548519d00b729b2bc7d017e7949b48da359f29229d4383b0113115033bc3 not found: ID does not exist" containerID="b243548519d00b729b2bc7d017e7949b48da359f29229d4383b0113115033bc3"
Jul 27 21:34:18.341857 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.341774    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b243548519d00b729b2bc7d017e7949b48da359f29229d4383b0113115033bc3"} err="failed to get container status \"b243548519d00b729b2bc7d017e7949b48da359f29229d4383b0113115033bc3\": rpc error: code = NotFound desc = could not find container \"b243548519d00b729b2bc7d017e7949b48da359f29229d4383b0113115033bc3\": container with ID starting with b243548519d00b729b2bc7d017e7949b48da359f29229d4383b0113115033bc3 not found: ID does not exist"
Jul 27 21:34:18.341857 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.341793    2577 scope.go:117] "RemoveContainer" containerID="864b9fe9eaa0839da8fe1e36fc431e936eed2c46d498130a242e8ffc5bee918e"
Jul 27 21:34:18.342046 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:34:18.342026    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"864b9fe9eaa0839da8fe1e36fc431e936eed2c46d498130a242e8ffc5bee918e\": container with ID starting with 864b9fe9eaa0839da8fe1e36fc431e936eed2c46d498130a242e8ffc5bee918e not found: ID does not exist" containerID="864b9fe9eaa0839da8fe1e36fc431e936eed2c46d498130a242e8ffc5bee918e"
Jul 27 21:34:18.342083 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.342054    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"864b9fe9eaa0839da8fe1e36fc431e936eed2c46d498130a242e8ffc5bee918e"} err="failed to get container status \"864b9fe9eaa0839da8fe1e36fc431e936eed2c46d498130a242e8ffc5bee918e\": rpc error: code = NotFound desc = could not find container \"864b9fe9eaa0839da8fe1e36fc431e936eed2c46d498130a242e8ffc5bee918e\": container with ID starting with 864b9fe9eaa0839da8fe1e36fc431e936eed2c46d498130a242e8ffc5bee918e not found: ID does not exist"
Jul 27 21:34:18.342083 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.342069    2577 scope.go:117] "RemoveContainer" containerID="50be5d19ca791d9da2e22dac2dd7ba04acb31558d2ddd0d7c12524bdb99a5769"
Jul 27 21:34:18.342288 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:34:18.342272    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50be5d19ca791d9da2e22dac2dd7ba04acb31558d2ddd0d7c12524bdb99a5769\": container with ID starting with 50be5d19ca791d9da2e22dac2dd7ba04acb31558d2ddd0d7c12524bdb99a5769 not found: ID does not exist" containerID="50be5d19ca791d9da2e22dac2dd7ba04acb31558d2ddd0d7c12524bdb99a5769"
Jul 27 21:34:18.342338 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.342293    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50be5d19ca791d9da2e22dac2dd7ba04acb31558d2ddd0d7c12524bdb99a5769"} err="failed to get container status \"50be5d19ca791d9da2e22dac2dd7ba04acb31558d2ddd0d7c12524bdb99a5769\": rpc error: code = NotFound desc = could not find container \"50be5d19ca791d9da2e22dac2dd7ba04acb31558d2ddd0d7c12524bdb99a5769\": container with ID starting with 50be5d19ca791d9da2e22dac2dd7ba04acb31558d2ddd0d7c12524bdb99a5769 not found: ID does not exist"
Jul 27 21:34:18.342338 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.342307    2577 scope.go:117] "RemoveContainer" containerID="246e8a5ebfaa37c992be7d4ba6b22cb02465a5d66bbd3987c5caa68aa9f7a4a2"
Jul 27 21:34:18.350828 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.350800    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-25vpb"]
Jul 27 21:34:18.354912 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.354896    2577 scope.go:117] "RemoveContainer" containerID="f5f1bc1309fd0fe435fc1e5e02da9e3060a9808b8eb561da738de50af0fa77c5"
Jul 27 21:34:18.361529 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.361511    2577 scope.go:117] "RemoveContainer" containerID="523bbdcef8920ad1c07efa46760592d7fd090e957989bf99340eaf688bee141b"
Jul 27 21:34:18.363293 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.363274    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-msl2b"]
Jul 27 21:34:18.368325 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.368304    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-msl2b"]
Jul 27 21:34:18.369494 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.369475    2577 scope.go:117] "RemoveContainer" containerID="246e8a5ebfaa37c992be7d4ba6b22cb02465a5d66bbd3987c5caa68aa9f7a4a2"
Jul 27 21:34:18.369732 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:34:18.369715    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"246e8a5ebfaa37c992be7d4ba6b22cb02465a5d66bbd3987c5caa68aa9f7a4a2\": container with ID starting with 246e8a5ebfaa37c992be7d4ba6b22cb02465a5d66bbd3987c5caa68aa9f7a4a2 not found: ID does not exist" containerID="246e8a5ebfaa37c992be7d4ba6b22cb02465a5d66bbd3987c5caa68aa9f7a4a2"
Jul 27 21:34:18.369792 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.369739    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"246e8a5ebfaa37c992be7d4ba6b22cb02465a5d66bbd3987c5caa68aa9f7a4a2"} err="failed to get container status \"246e8a5ebfaa37c992be7d4ba6b22cb02465a5d66bbd3987c5caa68aa9f7a4a2\": rpc error: code = NotFound desc = could not find container \"246e8a5ebfaa37c992be7d4ba6b22cb02465a5d66bbd3987c5caa68aa9f7a4a2\": container with ID starting with 246e8a5ebfaa37c992be7d4ba6b22cb02465a5d66bbd3987c5caa68aa9f7a4a2 not found: ID does not exist"
Jul 27 21:34:18.369792 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.369755    2577 scope.go:117] "RemoveContainer" containerID="f5f1bc1309fd0fe435fc1e5e02da9e3060a9808b8eb561da738de50af0fa77c5"
Jul 27 21:34:18.370023 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:34:18.370005    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5f1bc1309fd0fe435fc1e5e02da9e3060a9808b8eb561da738de50af0fa77c5\": container with ID starting with f5f1bc1309fd0fe435fc1e5e02da9e3060a9808b8eb561da738de50af0fa77c5 not found: ID does not exist" containerID="f5f1bc1309fd0fe435fc1e5e02da9e3060a9808b8eb561da738de50af0fa77c5"
Jul 27 21:34:18.370075 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.370031    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5f1bc1309fd0fe435fc1e5e02da9e3060a9808b8eb561da738de50af0fa77c5"} err="failed to get container status \"f5f1bc1309fd0fe435fc1e5e02da9e3060a9808b8eb561da738de50af0fa77c5\": rpc error: code = NotFound desc = could not find container \"f5f1bc1309fd0fe435fc1e5e02da9e3060a9808b8eb561da738de50af0fa77c5\": container with ID starting with f5f1bc1309fd0fe435fc1e5e02da9e3060a9808b8eb561da738de50af0fa77c5 not found: ID does not exist"
Jul 27 21:34:18.370075 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.370047    2577 scope.go:117] "RemoveContainer" containerID="523bbdcef8920ad1c07efa46760592d7fd090e957989bf99340eaf688bee141b"
Jul 27 21:34:18.370282 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:34:18.370266    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"523bbdcef8920ad1c07efa46760592d7fd090e957989bf99340eaf688bee141b\": container with ID starting with 523bbdcef8920ad1c07efa46760592d7fd090e957989bf99340eaf688bee141b not found: ID does not exist" containerID="523bbdcef8920ad1c07efa46760592d7fd090e957989bf99340eaf688bee141b"
Jul 27 21:34:18.370323 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.370287    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"523bbdcef8920ad1c07efa46760592d7fd090e957989bf99340eaf688bee141b"} err="failed to get container status \"523bbdcef8920ad1c07efa46760592d7fd090e957989bf99340eaf688bee141b\": rpc error: code = NotFound desc = could not find container \"523bbdcef8920ad1c07efa46760592d7fd090e957989bf99340eaf688bee141b\": container with ID starting with 523bbdcef8920ad1c07efa46760592d7fd090e957989bf99340eaf688bee141b not found: ID does not exist"
Jul 27 21:34:18.704485 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.704446    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:34:18.765890 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.765860    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-9f7ng"]
Jul 27 21:34:18.766332 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.766317    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2ced923-c791-4a9a-bae9-17ecadd68c2c" containerName="kserve-container"
Jul 27 21:34:18.766392 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.766335    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2ced923-c791-4a9a-bae9-17ecadd68c2c" containerName="kserve-container"
Jul 27 21:34:18.766392 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.766351    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e81f537c-a850-4658-886c-c6480ef951d1" containerName="storage-initializer"
Jul 27 21:34:18.766392 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.766360    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e81f537c-a850-4658-886c-c6480ef951d1" containerName="storage-initializer"
Jul 27 21:34:18.766392 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.766373    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e81f537c-a850-4658-886c-c6480ef951d1" containerName="kube-rbac-proxy"
Jul 27 21:34:18.766392 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.766383    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e81f537c-a850-4658-886c-c6480ef951d1" containerName="kube-rbac-proxy"
Jul 27 21:34:18.766632 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.766394    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e81f537c-a850-4658-886c-c6480ef951d1" containerName="kserve-container"
Jul 27 21:34:18.766632 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.766403    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e81f537c-a850-4658-886c-c6480ef951d1" containerName="kserve-container"
Jul 27 21:34:18.766632 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.766414    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2ced923-c791-4a9a-bae9-17ecadd68c2c" containerName="kube-rbac-proxy"
Jul 27 21:34:18.766632 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.766422    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2ced923-c791-4a9a-bae9-17ecadd68c2c" containerName="kube-rbac-proxy"
Jul 27 21:34:18.766632 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.766438    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2ced923-c791-4a9a-bae9-17ecadd68c2c" containerName="storage-initializer"
Jul 27 21:34:18.766632 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.766446    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2ced923-c791-4a9a-bae9-17ecadd68c2c" containerName="storage-initializer"
Jul 27 21:34:18.766632 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.766510    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e81f537c-a850-4658-886c-c6480ef951d1" containerName="kserve-container"
Jul 27 21:34:18.766632 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.766521    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e81f537c-a850-4658-886c-c6480ef951d1" containerName="kube-rbac-proxy"
Jul 27 21:34:18.766632 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.766530    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2ced923-c791-4a9a-bae9-17ecadd68c2c" containerName="kserve-container"
Jul 27 21:34:18.766632 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.766541    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2ced923-c791-4a9a-bae9-17ecadd68c2c" containerName="kube-rbac-proxy"
Jul 27 21:34:18.771779 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.771731    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-9f7ng"
Jul 27 21:34:18.773396 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.773371    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-n59xt"]
Jul 27 21:34:18.774505 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.774307    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Jul 27 21:34:18.774505 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.774350    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 21:34:18.774505 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.774390    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 21:34:18.774505 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.774409    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Jul 27 21:34:18.774854 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.774635    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 21:34:18.774854 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.774726    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 21:34:18.774854 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.774753    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v7c66\""
Jul 27 21:34:18.776579 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.776564    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-n59xt"
Jul 27 21:34:18.778815 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.778795    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Jul 27 21:34:18.780003 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.779833    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-9f7ng"]
Jul 27 21:34:18.785206 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.785178    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:34:18.793059 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.793039    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-n59xt"]
Jul 27 21:34:18.842026 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.841999    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a723442e-449c-40fd-9279-c24f2916e745-cabundle-cert\") pod \"isvc-canary-promote-predictor-d9759ffd5-9f7ng\" (UID: \"a723442e-449c-40fd-9279-c24f2916e745\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-9f7ng"
Jul 27 21:34:18.842171 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.842038    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8fab226-5fd8-4f9b-9b17-9168fa14aa7d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-86766c4958-n59xt\" (UID: \"b8fab226-5fd8-4f9b-9b17-9168fa14aa7d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-n59xt"
Jul 27 21:34:18.842171 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.842057    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a723442e-449c-40fd-9279-c24f2916e745-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-d9759ffd5-9f7ng\" (UID: \"a723442e-449c-40fd-9279-c24f2916e745\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-9f7ng"
Jul 27 21:34:18.842171 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.842083    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8fab226-5fd8-4f9b-9b17-9168fa14aa7d-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-86766c4958-n59xt\" (UID: \"b8fab226-5fd8-4f9b-9b17-9168fa14aa7d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-n59xt"
Jul 27 21:34:18.842171 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.842134    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8fab226-5fd8-4f9b-9b17-9168fa14aa7d-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-86766c4958-n59xt\" (UID: \"b8fab226-5fd8-4f9b-9b17-9168fa14aa7d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-n59xt"
Jul 27 21:34:18.842171 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.842158    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a723442e-449c-40fd-9279-c24f2916e745-proxy-tls\") pod \"isvc-canary-promote-predictor-d9759ffd5-9f7ng\" (UID: \"a723442e-449c-40fd-9279-c24f2916e745\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-9f7ng"
Jul 27 21:34:18.842386 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.842193    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-685vt\" (UniqueName: \"kubernetes.io/projected/a723442e-449c-40fd-9279-c24f2916e745-kube-api-access-685vt\") pod \"isvc-canary-promote-predictor-d9759ffd5-9f7ng\" (UID: \"a723442e-449c-40fd-9279-c24f2916e745\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-9f7ng"
Jul 27 21:34:18.842386 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.842241    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwf6v\" (UniqueName: \"kubernetes.io/projected/b8fab226-5fd8-4f9b-9b17-9168fa14aa7d-kube-api-access-jwf6v\") pod \"isvc-canary-promote-v2-predictor-86766c4958-n59xt\" (UID: \"b8fab226-5fd8-4f9b-9b17-9168fa14aa7d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-n59xt"
Jul 27 21:34:18.842386 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.842276    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a723442e-449c-40fd-9279-c24f2916e745-kserve-provision-location\") pod \"isvc-canary-promote-predictor-d9759ffd5-9f7ng\" (UID: \"a723442e-449c-40fd-9279-c24f2916e745\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-9f7ng"
Jul 27 21:34:18.842386 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.842301    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8fab226-5fd8-4f9b-9b17-9168fa14aa7d-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-86766c4958-n59xt\" (UID: \"b8fab226-5fd8-4f9b-9b17-9168fa14aa7d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-n59xt"
Jul 27 21:34:18.852212 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.852192    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:34:18.942983 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.942949    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8fab226-5fd8-4f9b-9b17-9168fa14aa7d-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-86766c4958-n59xt\" (UID: \"b8fab226-5fd8-4f9b-9b17-9168fa14aa7d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-n59xt"
Jul 27 21:34:18.942983 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.942979    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a723442e-449c-40fd-9279-c24f2916e745-cabundle-cert\") pod \"isvc-canary-promote-predictor-d9759ffd5-9f7ng\" (UID: \"a723442e-449c-40fd-9279-c24f2916e745\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-9f7ng"
Jul 27 21:34:18.943178 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.943001    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8fab226-5fd8-4f9b-9b17-9168fa14aa7d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-86766c4958-n59xt\" (UID: \"b8fab226-5fd8-4f9b-9b17-9168fa14aa7d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-n59xt"
Jul 27 21:34:18.943178 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.943017    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a723442e-449c-40fd-9279-c24f2916e745-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-d9759ffd5-9f7ng\" (UID: \"a723442e-449c-40fd-9279-c24f2916e745\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-9f7ng"
Jul 27 21:34:18.943178 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.943034    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8fab226-5fd8-4f9b-9b17-9168fa14aa7d-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-86766c4958-n59xt\" (UID: \"b8fab226-5fd8-4f9b-9b17-9168fa14aa7d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-n59xt"
Jul 27 21:34:18.943178 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.943066    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8fab226-5fd8-4f9b-9b17-9168fa14aa7d-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-86766c4958-n59xt\" (UID: \"b8fab226-5fd8-4f9b-9b17-9168fa14aa7d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-n59xt"
Jul 27 21:34:18.943178 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.943101    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a723442e-449c-40fd-9279-c24f2916e745-proxy-tls\") pod \"isvc-canary-promote-predictor-d9759ffd5-9f7ng\" (UID: \"a723442e-449c-40fd-9279-c24f2916e745\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-9f7ng"
Jul 27 21:34:18.943178 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.943148    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-685vt\" (UniqueName: \"kubernetes.io/projected/a723442e-449c-40fd-9279-c24f2916e745-kube-api-access-685vt\") pod \"isvc-canary-promote-predictor-d9759ffd5-9f7ng\" (UID: \"a723442e-449c-40fd-9279-c24f2916e745\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-9f7ng"
Jul 27 21:34:18.943484 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.943183    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwf6v\" (UniqueName: \"kubernetes.io/projected/b8fab226-5fd8-4f9b-9b17-9168fa14aa7d-kube-api-access-jwf6v\") pod \"isvc-canary-promote-v2-predictor-86766c4958-n59xt\" (UID: \"b8fab226-5fd8-4f9b-9b17-9168fa14aa7d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-n59xt"
Jul 27 21:34:18.943484 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.943213    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a723442e-449c-40fd-9279-c24f2916e745-kserve-provision-location\") pod \"isvc-canary-promote-predictor-d9759ffd5-9f7ng\" (UID: \"a723442e-449c-40fd-9279-c24f2916e745\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-9f7ng"
Jul 27 21:34:18.943590 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.943479    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8fab226-5fd8-4f9b-9b17-9168fa14aa7d-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-86766c4958-n59xt\" (UID: \"b8fab226-5fd8-4f9b-9b17-9168fa14aa7d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-n59xt"
Jul 27 21:34:18.943590 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.943512    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a723442e-449c-40fd-9279-c24f2916e745-kserve-provision-location\") pod \"isvc-canary-promote-predictor-d9759ffd5-9f7ng\" (UID: \"a723442e-449c-40fd-9279-c24f2916e745\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-9f7ng"
Jul 27 21:34:18.943802 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.943785    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a723442e-449c-40fd-9279-c24f2916e745-cabundle-cert\") pod \"isvc-canary-promote-predictor-d9759ffd5-9f7ng\" (UID: \"a723442e-449c-40fd-9279-c24f2916e745\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-9f7ng"
Jul 27 21:34:18.943882 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.943807    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a723442e-449c-40fd-9279-c24f2916e745-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-d9759ffd5-9f7ng\" (UID: \"a723442e-449c-40fd-9279-c24f2916e745\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-9f7ng"
Jul 27 21:34:18.944362 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.944318    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8fab226-5fd8-4f9b-9b17-9168fa14aa7d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-86766c4958-n59xt\" (UID: \"b8fab226-5fd8-4f9b-9b17-9168fa14aa7d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-n59xt"
Jul 27 21:34:18.944444 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.944406    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8fab226-5fd8-4f9b-9b17-9168fa14aa7d-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-86766c4958-n59xt\" (UID: \"b8fab226-5fd8-4f9b-9b17-9168fa14aa7d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-n59xt"
Jul 27 21:34:18.945507 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.945480    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8fab226-5fd8-4f9b-9b17-9168fa14aa7d-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-86766c4958-n59xt\" (UID: \"b8fab226-5fd8-4f9b-9b17-9168fa14aa7d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-n59xt"
Jul 27 21:34:18.946126 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.946103    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a723442e-449c-40fd-9279-c24f2916e745-proxy-tls\") pod \"isvc-canary-promote-predictor-d9759ffd5-9f7ng\" (UID: \"a723442e-449c-40fd-9279-c24f2916e745\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-9f7ng"
Jul 27 21:34:18.950544 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.950524    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwf6v\" (UniqueName: \"kubernetes.io/projected/b8fab226-5fd8-4f9b-9b17-9168fa14aa7d-kube-api-access-jwf6v\") pod \"isvc-canary-promote-v2-predictor-86766c4958-n59xt\" (UID: \"b8fab226-5fd8-4f9b-9b17-9168fa14aa7d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-n59xt"
Jul 27 21:34:18.950630 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.950593    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-685vt\" (UniqueName: \"kubernetes.io/projected/a723442e-449c-40fd-9279-c24f2916e745-kube-api-access-685vt\") pod \"isvc-canary-promote-predictor-d9759ffd5-9f7ng\" (UID: \"a723442e-449c-40fd-9279-c24f2916e745\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-9f7ng"
Jul 27 21:34:18.952862 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:18.952839    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:34:19.084636 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:19.084564    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-9f7ng"
Jul 27 21:34:19.092708 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:19.092683    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-n59xt"
Jul 27 21:34:19.139141 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:19.139104    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2ced923-c791-4a9a-bae9-17ecadd68c2c" path="/var/lib/kubelet/pods/c2ced923-c791-4a9a-bae9-17ecadd68c2c/volumes"
Jul 27 21:34:19.140707 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:19.140147    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e81f537c-a850-4658-886c-c6480ef951d1" path="/var/lib/kubelet/pods/e81f537c-a850-4658-886c-c6480ef951d1/volumes"
Jul 27 21:34:19.228621 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:19.228595    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-9f7ng"]
Jul 27 21:34:19.231148 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:34:19.231125    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda723442e_449c_40fd_9279_c24f2916e745.slice/crio-346f231b480d432768eae9a754e311bfe819df16e79e7370e881f578f55098f7 WatchSource:0}: Error finding container 346f231b480d432768eae9a754e311bfe819df16e79e7370e881f578f55098f7: Status 404 returned error can't find the container with id 346f231b480d432768eae9a754e311bfe819df16e79e7370e881f578f55098f7
Jul 27 21:34:19.248730 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:19.248695    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-n59xt"]
Jul 27 21:34:19.250897 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:34:19.250876    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8fab226_5fd8_4f9b_9b17_9168fa14aa7d.slice/crio-5b6dbbf71876cbc221c43912aed71a6de2cf6150f38855b1a105cee6d1e6d0bc WatchSource:0}: Error finding container 5b6dbbf71876cbc221c43912aed71a6de2cf6150f38855b1a105cee6d1e6d0bc: Status 404 returned error can't find the container with id 5b6dbbf71876cbc221c43912aed71a6de2cf6150f38855b1a105cee6d1e6d0bc
Jul 27 21:34:19.328907 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:19.328875    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-n59xt" event={"ID":"b8fab226-5fd8-4f9b-9b17-9168fa14aa7d","Type":"ContainerStarted","Data":"a4098e036c723650c95b7fc1bfba60fa1dd65bf146b7e90229e5b2ace94de6a1"}
Jul 27 21:34:19.329082 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:19.328936    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-n59xt" event={"ID":"b8fab226-5fd8-4f9b-9b17-9168fa14aa7d","Type":"ContainerStarted","Data":"5b6dbbf71876cbc221c43912aed71a6de2cf6150f38855b1a105cee6d1e6d0bc"}
Jul 27 21:34:19.330593 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:19.330559    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-9f7ng" event={"ID":"a723442e-449c-40fd-9279-c24f2916e745","Type":"ContainerStarted","Data":"25ad3c2e8c75a8fa364616378421923d617b154ee2b0c80769e3e12bb33d9570"}
Jul 27 21:34:19.330700 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:19.330613    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-9f7ng" event={"ID":"a723442e-449c-40fd-9279-c24f2916e745","Type":"ContainerStarted","Data":"346f231b480d432768eae9a754e311bfe819df16e79e7370e881f578f55098f7"}
Jul 27 21:34:23.348328 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:23.348281    2577 generic.go:356] "Generic (PLEG): container finished" podID="b8fab226-5fd8-4f9b-9b17-9168fa14aa7d" containerID="a4098e036c723650c95b7fc1bfba60fa1dd65bf146b7e90229e5b2ace94de6a1" exitCode=0
Jul 27 21:34:23.348800 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:23.348354    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-n59xt" event={"ID":"b8fab226-5fd8-4f9b-9b17-9168fa14aa7d","Type":"ContainerDied","Data":"a4098e036c723650c95b7fc1bfba60fa1dd65bf146b7e90229e5b2ace94de6a1"}
Jul 27 21:34:23.349858 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:23.349836    2577 generic.go:356] "Generic (PLEG): container finished" podID="a723442e-449c-40fd-9279-c24f2916e745" containerID="25ad3c2e8c75a8fa364616378421923d617b154ee2b0c80769e3e12bb33d9570" exitCode=0
Jul 27 21:34:23.349970 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:23.349879    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-9f7ng" event={"ID":"a723442e-449c-40fd-9279-c24f2916e745","Type":"ContainerDied","Data":"25ad3c2e8c75a8fa364616378421923d617b154ee2b0c80769e3e12bb33d9570"}
Jul 27 21:34:24.355572 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:24.355539    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-9f7ng" event={"ID":"a723442e-449c-40fd-9279-c24f2916e745","Type":"ContainerStarted","Data":"f0c58b8bdbd9dd35a2fd015f708a29ce09b680bbdaa2cab24d278b5c0f7be0a2"}
Jul 27 21:34:24.355963 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:24.355599    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-9f7ng" event={"ID":"a723442e-449c-40fd-9279-c24f2916e745","Type":"ContainerStarted","Data":"12b79ee1e201036a379739ee7a29802de8b94b952445d5ad67f3d1a73323068a"}
Jul 27 21:34:24.355963 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:24.355803    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-9f7ng"
Jul 27 21:34:24.357501 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:24.357480    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-n59xt" event={"ID":"b8fab226-5fd8-4f9b-9b17-9168fa14aa7d","Type":"ContainerStarted","Data":"e09b8598b158d6e8991fd1892496941abed39be17b9e7f0b0798318f0325fb04"}
Jul 27 21:34:24.357632 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:24.357524    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-n59xt" event={"ID":"b8fab226-5fd8-4f9b-9b17-9168fa14aa7d","Type":"ContainerStarted","Data":"622ee9fc52df85c19c6654fcb292920c372e29ffbee2dc8ee5a6e4e56848542f"}
Jul 27 21:34:24.357816 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:24.357800    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-n59xt"
Jul 27 21:34:24.357867 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:24.357848    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-n59xt"
Jul 27 21:34:24.359336 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:24.359306    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-n59xt" podUID="b8fab226-5fd8-4f9b-9b17-9168fa14aa7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 27 21:34:24.375271 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:24.375230    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-9f7ng" podStartSLOduration=2.811357084 podStartE2EDuration="6.375217258s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.563860174s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:34:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:34:24.373850861 +0000 UTC m=+1787.741040728" watchObservedRunningTime="2026-07-27 21:34:24.375217258 +0000 UTC m=+1787.742407123"
Jul 27 21:34:24.392991 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:24.392947    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-n59xt" podStartSLOduration=2.465326942 podStartE2EDuration="6.392933017s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.927606075s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:34:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:34:24.392178553 +0000 UTC m=+1787.759368422" watchObservedRunningTime="2026-07-27 21:34:24.392933017 +0000 UTC m=+1787.760122881"
Jul 27 21:34:25.361906 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:25.361866    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-9f7ng"
Jul 27 21:34:25.361906 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:25.361892    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-n59xt" podUID="b8fab226-5fd8-4f9b-9b17-9168fa14aa7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 27 21:34:25.363020 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:25.362997    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-9f7ng" podUID="a723442e-449c-40fd-9279-c24f2916e745" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 27 21:34:26.365253 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:26.365215    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-9f7ng" podUID="a723442e-449c-40fd-9279-c24f2916e745" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 27 21:34:30.366280 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:30.366203    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-n59xt"
Jul 27 21:34:30.366792 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:30.366744    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-n59xt" podUID="b8fab226-5fd8-4f9b-9b17-9168fa14aa7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 27 21:34:31.369539 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:31.369514    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-9f7ng"
Jul 27 21:34:31.370093 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:31.370067    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-9f7ng" podUID="a723442e-449c-40fd-9279-c24f2916e745" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 27 21:34:40.367577 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:40.367533    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-n59xt" podUID="b8fab226-5fd8-4f9b-9b17-9168fa14aa7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 27 21:34:41.370286 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:41.370245    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-9f7ng" podUID="a723442e-449c-40fd-9279-c24f2916e745" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 27 21:34:49.083355 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:49.083317    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:34:50.367412 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:50.367368    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-n59xt" podUID="b8fab226-5fd8-4f9b-9b17-9168fa14aa7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 27 21:34:51.370131 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:34:51.370094    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-9f7ng" podUID="a723442e-449c-40fd-9279-c24f2916e745" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 27 21:35:00.367209 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:35:00.367170    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-n59xt" podUID="b8fab226-5fd8-4f9b-9b17-9168fa14aa7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 27 21:35:01.370295 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:35:01.370260    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-9f7ng" podUID="a723442e-449c-40fd-9279-c24f2916e745" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 27 21:35:10.366748 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:35:10.366709    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-n59xt" podUID="b8fab226-5fd8-4f9b-9b17-9168fa14aa7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 27 21:35:11.370702 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:35:11.370660    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-9f7ng" podUID="a723442e-449c-40fd-9279-c24f2916e745" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 27 21:35:20.366994 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:35:20.366948    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-n59xt" podUID="b8fab226-5fd8-4f9b-9b17-9168fa14aa7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 27 21:35:21.370800 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:35:21.370731    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-9f7ng" podUID="a723442e-449c-40fd-9279-c24f2916e745" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 27 21:35:30.366742 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:35:30.366695    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-n59xt" podUID="b8fab226-5fd8-4f9b-9b17-9168fa14aa7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 27 21:35:31.370725 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:35:31.370689    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-9f7ng" podUID="a723442e-449c-40fd-9279-c24f2916e745" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 27 21:35:39.135661 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:35:39.135629    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-n59xt"
Jul 27 21:35:41.135830 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:35:41.135804    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-9f7ng"
Jul 27 21:35:48.834593 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:35:48.834563    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-9f7ng"]
Jul 27 21:35:48.835033 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:35:48.834829    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-9f7ng" podUID="a723442e-449c-40fd-9279-c24f2916e745" containerName="kserve-container" containerID="cri-o://12b79ee1e201036a379739ee7a29802de8b94b952445d5ad67f3d1a73323068a" gracePeriod=30
Jul 27 21:35:48.835033 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:35:48.834829    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-9f7ng" podUID="a723442e-449c-40fd-9279-c24f2916e745" containerName="kube-rbac-proxy" containerID="cri-o://f0c58b8bdbd9dd35a2fd015f708a29ce09b680bbdaa2cab24d278b5c0f7be0a2" gracePeriod=30
Jul 27 21:35:48.862411 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:35:48.862381    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:35:49.692131 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:35:49.692099    2577 generic.go:356] "Generic (PLEG): container finished" podID="a723442e-449c-40fd-9279-c24f2916e745" containerID="f0c58b8bdbd9dd35a2fd015f708a29ce09b680bbdaa2cab24d278b5c0f7be0a2" exitCode=2
Jul 27 21:35:49.692131 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:35:49.692132    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-9f7ng" event={"ID":"a723442e-449c-40fd-9279-c24f2916e745","Type":"ContainerDied","Data":"f0c58b8bdbd9dd35a2fd015f708a29ce09b680bbdaa2cab24d278b5c0f7be0a2"}
Jul 27 21:35:51.131324 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:35:51.131267    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-9f7ng" podUID="a723442e-449c-40fd-9279-c24f2916e745" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 27 21:35:51.365720 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:35:51.365682    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-9f7ng" podUID="a723442e-449c-40fd-9279-c24f2916e745" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Jul 27 21:35:52.877991 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:35:52.877966    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-9f7ng"
Jul 27 21:35:53.002218 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:35:53.002160    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a723442e-449c-40fd-9279-c24f2916e745-kube-rbac-proxy-sar-config\") pod \"a723442e-449c-40fd-9279-c24f2916e745\" (UID: \"a723442e-449c-40fd-9279-c24f2916e745\") "
Jul 27 21:35:53.002218 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:35:53.002193    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a723442e-449c-40fd-9279-c24f2916e745-kserve-provision-location\") pod \"a723442e-449c-40fd-9279-c24f2916e745\" (UID: \"a723442e-449c-40fd-9279-c24f2916e745\") "
Jul 27 21:35:53.002392 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:35:53.002241    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-685vt\" (UniqueName: \"kubernetes.io/projected/a723442e-449c-40fd-9279-c24f2916e745-kube-api-access-685vt\") pod \"a723442e-449c-40fd-9279-c24f2916e745\" (UID: \"a723442e-449c-40fd-9279-c24f2916e745\") "
Jul 27 21:35:53.002392 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:35:53.002269    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a723442e-449c-40fd-9279-c24f2916e745-proxy-tls\") pod \"a723442e-449c-40fd-9279-c24f2916e745\" (UID: \"a723442e-449c-40fd-9279-c24f2916e745\") "
Jul 27 21:35:53.002392 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:35:53.002298    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a723442e-449c-40fd-9279-c24f2916e745-cabundle-cert\") pod \"a723442e-449c-40fd-9279-c24f2916e745\" (UID: \"a723442e-449c-40fd-9279-c24f2916e745\") "
Jul 27 21:35:53.002532 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:35:53.002449    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a723442e-449c-40fd-9279-c24f2916e745-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a723442e-449c-40fd-9279-c24f2916e745" (UID: "a723442e-449c-40fd-9279-c24f2916e745"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:35:53.002532 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:35:53.002512    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a723442e-449c-40fd-9279-c24f2916e745-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a723442e-449c-40fd-9279-c24f2916e745" (UID: "a723442e-449c-40fd-9279-c24f2916e745"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:35:53.002781 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:35:53.002739    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a723442e-449c-40fd-9279-c24f2916e745-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a723442e-449c-40fd-9279-c24f2916e745" (UID: "a723442e-449c-40fd-9279-c24f2916e745"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:35:53.004272 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:35:53.004253    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a723442e-449c-40fd-9279-c24f2916e745-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a723442e-449c-40fd-9279-c24f2916e745" (UID: "a723442e-449c-40fd-9279-c24f2916e745"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:35:53.004336 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:35:53.004315    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a723442e-449c-40fd-9279-c24f2916e745-kube-api-access-685vt" (OuterVolumeSpecName: "kube-api-access-685vt") pod "a723442e-449c-40fd-9279-c24f2916e745" (UID: "a723442e-449c-40fd-9279-c24f2916e745"). InnerVolumeSpecName "kube-api-access-685vt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:35:53.102882 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:35:53.102859    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-685vt\" (UniqueName: \"kubernetes.io/projected/a723442e-449c-40fd-9279-c24f2916e745-kube-api-access-685vt\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:35:53.102882 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:35:53.102880    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a723442e-449c-40fd-9279-c24f2916e745-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:35:53.103004 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:35:53.102889    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a723442e-449c-40fd-9279-c24f2916e745-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:35:53.103004 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:35:53.102897    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a723442e-449c-40fd-9279-c24f2916e745-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:35:53.103004 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:35:53.102905    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a723442e-449c-40fd-9279-c24f2916e745-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:35:53.709711 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:35:53.709679    2577 generic.go:356] "Generic (PLEG): container finished" podID="a723442e-449c-40fd-9279-c24f2916e745" containerID="12b79ee1e201036a379739ee7a29802de8b94b952445d5ad67f3d1a73323068a" exitCode=0
Jul 27 21:35:53.709907 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:35:53.709733    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-9f7ng"
Jul 27 21:35:53.709907 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:35:53.709746    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-9f7ng" event={"ID":"a723442e-449c-40fd-9279-c24f2916e745","Type":"ContainerDied","Data":"12b79ee1e201036a379739ee7a29802de8b94b952445d5ad67f3d1a73323068a"}
Jul 27 21:35:53.709907 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:35:53.709821    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-9f7ng" event={"ID":"a723442e-449c-40fd-9279-c24f2916e745","Type":"ContainerDied","Data":"346f231b480d432768eae9a754e311bfe819df16e79e7370e881f578f55098f7"}
Jul 27 21:35:53.709907 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:35:53.709849    2577 scope.go:117] "RemoveContainer" containerID="f0c58b8bdbd9dd35a2fd015f708a29ce09b680bbdaa2cab24d278b5c0f7be0a2"
Jul 27 21:35:53.717812 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:35:53.717792    2577 scope.go:117] "RemoveContainer" containerID="12b79ee1e201036a379739ee7a29802de8b94b952445d5ad67f3d1a73323068a"
Jul 27 21:35:53.725007 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:35:53.724989    2577 scope.go:117] "RemoveContainer" containerID="25ad3c2e8c75a8fa364616378421923d617b154ee2b0c80769e3e12bb33d9570"
Jul 27 21:35:53.728026 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:35:53.728003    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-9f7ng"]
Jul 27 21:35:53.732873 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:35:53.732856    2577 scope.go:117] "RemoveContainer" containerID="f0c58b8bdbd9dd35a2fd015f708a29ce09b680bbdaa2cab24d278b5c0f7be0a2"
Jul 27 21:35:53.733158 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:35:53.733137    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0c58b8bdbd9dd35a2fd015f708a29ce09b680bbdaa2cab24d278b5c0f7be0a2\": container with ID starting with f0c58b8bdbd9dd35a2fd015f708a29ce09b680bbdaa2cab24d278b5c0f7be0a2 not found: ID does not exist" containerID="f0c58b8bdbd9dd35a2fd015f708a29ce09b680bbdaa2cab24d278b5c0f7be0a2"
Jul 27 21:35:53.733253 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:35:53.733163    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0c58b8bdbd9dd35a2fd015f708a29ce09b680bbdaa2cab24d278b5c0f7be0a2"} err="failed to get container status \"f0c58b8bdbd9dd35a2fd015f708a29ce09b680bbdaa2cab24d278b5c0f7be0a2\": rpc error: code = NotFound desc = could not find container \"f0c58b8bdbd9dd35a2fd015f708a29ce09b680bbdaa2cab24d278b5c0f7be0a2\": container with ID starting with f0c58b8bdbd9dd35a2fd015f708a29ce09b680bbdaa2cab24d278b5c0f7be0a2 not found: ID does not exist"
Jul 27 21:35:53.733253 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:35:53.733186    2577 scope.go:117] "RemoveContainer" containerID="12b79ee1e201036a379739ee7a29802de8b94b952445d5ad67f3d1a73323068a"
Jul 27 21:35:53.733438 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:35:53.733421    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12b79ee1e201036a379739ee7a29802de8b94b952445d5ad67f3d1a73323068a\": container with ID starting with 12b79ee1e201036a379739ee7a29802de8b94b952445d5ad67f3d1a73323068a not found: ID does not exist" containerID="12b79ee1e201036a379739ee7a29802de8b94b952445d5ad67f3d1a73323068a"
Jul 27 21:35:53.733481 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:35:53.733452    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12b79ee1e201036a379739ee7a29802de8b94b952445d5ad67f3d1a73323068a"} err="failed to get container status \"12b79ee1e201036a379739ee7a29802de8b94b952445d5ad67f3d1a73323068a\": rpc error: code = NotFound desc = could not find container \"12b79ee1e201036a379739ee7a29802de8b94b952445d5ad67f3d1a73323068a\": container with ID starting with 12b79ee1e201036a379739ee7a29802de8b94b952445d5ad67f3d1a73323068a not found: ID does not exist"
Jul 27 21:35:53.733481 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:35:53.733468    2577 scope.go:117] "RemoveContainer" containerID="25ad3c2e8c75a8fa364616378421923d617b154ee2b0c80769e3e12bb33d9570"
Jul 27 21:35:53.733725 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:35:53.733707    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25ad3c2e8c75a8fa364616378421923d617b154ee2b0c80769e3e12bb33d9570\": container with ID starting with 25ad3c2e8c75a8fa364616378421923d617b154ee2b0c80769e3e12bb33d9570 not found: ID does not exist" containerID="25ad3c2e8c75a8fa364616378421923d617b154ee2b0c80769e3e12bb33d9570"
Jul 27 21:35:53.733806 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:35:53.733736    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25ad3c2e8c75a8fa364616378421923d617b154ee2b0c80769e3e12bb33d9570"} err="failed to get container status \"25ad3c2e8c75a8fa364616378421923d617b154ee2b0c80769e3e12bb33d9570\": rpc error: code = NotFound desc = could not find container \"25ad3c2e8c75a8fa364616378421923d617b154ee2b0c80769e3e12bb33d9570\": container with ID starting with 25ad3c2e8c75a8fa364616378421923d617b154ee2b0c80769e3e12bb33d9570 not found: ID does not exist"
Jul 27 21:35:53.735932 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:35:53.735910    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-9f7ng"]
Jul 27 21:35:55.135488 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:35:55.135455    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a723442e-449c-40fd-9279-c24f2916e745" path="/var/lib/kubelet/pods/a723442e-449c-40fd-9279-c24f2916e745/volumes"
Jul 27 21:37:39.050844 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:39.050742    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-n59xt"]
Jul 27 21:37:39.051342 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:39.051106    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-n59xt" podUID="b8fab226-5fd8-4f9b-9b17-9168fa14aa7d" containerName="kserve-container" containerID="cri-o://622ee9fc52df85c19c6654fcb292920c372e29ffbee2dc8ee5a6e4e56848542f" gracePeriod=30
Jul 27 21:37:39.051342 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:39.051150    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-n59xt" podUID="b8fab226-5fd8-4f9b-9b17-9168fa14aa7d" containerName="kube-rbac-proxy" containerID="cri-o://e09b8598b158d6e8991fd1892496941abed39be17b9e7f0b0798318f0325fb04" gracePeriod=30
Jul 27 21:37:39.103857 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:39.103810    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:37:39.131953 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:39.131920    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-n59xt" podUID="b8fab226-5fd8-4f9b-9b17-9168fa14aa7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 27 21:37:40.129172 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:40.129138    2577 generic.go:356] "Generic (PLEG): container finished" podID="b8fab226-5fd8-4f9b-9b17-9168fa14aa7d" containerID="e09b8598b158d6e8991fd1892496941abed39be17b9e7f0b0798318f0325fb04" exitCode=2
Jul 27 21:37:40.129527 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:40.129204    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-n59xt" event={"ID":"b8fab226-5fd8-4f9b-9b17-9168fa14aa7d","Type":"ContainerDied","Data":"e09b8598b158d6e8991fd1892496941abed39be17b9e7f0b0798318f0325fb04"}
Jul 27 21:37:40.362289 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:40.362244    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-n59xt" podUID="b8fab226-5fd8-4f9b-9b17-9168fa14aa7d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Jul 27 21:37:43.290307 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:43.290286    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-n59xt"
Jul 27 21:37:43.314244 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:43.314219    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8fab226-5fd8-4f9b-9b17-9168fa14aa7d-proxy-tls\") pod \"b8fab226-5fd8-4f9b-9b17-9168fa14aa7d\" (UID: \"b8fab226-5fd8-4f9b-9b17-9168fa14aa7d\") "
Jul 27 21:37:43.314358 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:43.314295    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8fab226-5fd8-4f9b-9b17-9168fa14aa7d-cabundle-cert\") pod \"b8fab226-5fd8-4f9b-9b17-9168fa14aa7d\" (UID: \"b8fab226-5fd8-4f9b-9b17-9168fa14aa7d\") "
Jul 27 21:37:43.314358 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:43.314327    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8fab226-5fd8-4f9b-9b17-9168fa14aa7d-kube-rbac-proxy-sar-config\") pod \"b8fab226-5fd8-4f9b-9b17-9168fa14aa7d\" (UID: \"b8fab226-5fd8-4f9b-9b17-9168fa14aa7d\") "
Jul 27 21:37:43.314473 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:43.314357    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8fab226-5fd8-4f9b-9b17-9168fa14aa7d-kserve-provision-location\") pod \"b8fab226-5fd8-4f9b-9b17-9168fa14aa7d\" (UID: \"b8fab226-5fd8-4f9b-9b17-9168fa14aa7d\") "
Jul 27 21:37:43.314473 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:43.314425    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwf6v\" (UniqueName: \"kubernetes.io/projected/b8fab226-5fd8-4f9b-9b17-9168fa14aa7d-kube-api-access-jwf6v\") pod \"b8fab226-5fd8-4f9b-9b17-9168fa14aa7d\" (UID: \"b8fab226-5fd8-4f9b-9b17-9168fa14aa7d\") "
Jul 27 21:37:43.314670 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:43.314646    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8fab226-5fd8-4f9b-9b17-9168fa14aa7d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b8fab226-5fd8-4f9b-9b17-9168fa14aa7d" (UID: "b8fab226-5fd8-4f9b-9b17-9168fa14aa7d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:37:43.314732 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:43.314669    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8fab226-5fd8-4f9b-9b17-9168fa14aa7d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b8fab226-5fd8-4f9b-9b17-9168fa14aa7d" (UID: "b8fab226-5fd8-4f9b-9b17-9168fa14aa7d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:37:43.314732 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:43.314677    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8fab226-5fd8-4f9b-9b17-9168fa14aa7d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b8fab226-5fd8-4f9b-9b17-9168fa14aa7d" (UID: "b8fab226-5fd8-4f9b-9b17-9168fa14aa7d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:37:43.316274 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:43.316249    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8fab226-5fd8-4f9b-9b17-9168fa14aa7d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b8fab226-5fd8-4f9b-9b17-9168fa14aa7d" (UID: "b8fab226-5fd8-4f9b-9b17-9168fa14aa7d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:37:43.316431 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:43.316410    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8fab226-5fd8-4f9b-9b17-9168fa14aa7d-kube-api-access-jwf6v" (OuterVolumeSpecName: "kube-api-access-jwf6v") pod "b8fab226-5fd8-4f9b-9b17-9168fa14aa7d" (UID: "b8fab226-5fd8-4f9b-9b17-9168fa14aa7d"). InnerVolumeSpecName "kube-api-access-jwf6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:37:43.415669 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:43.415606    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwf6v\" (UniqueName: \"kubernetes.io/projected/b8fab226-5fd8-4f9b-9b17-9168fa14aa7d-kube-api-access-jwf6v\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:37:43.415669 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:43.415630    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8fab226-5fd8-4f9b-9b17-9168fa14aa7d-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:37:43.415669 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:43.415640    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8fab226-5fd8-4f9b-9b17-9168fa14aa7d-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:37:43.415669 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:43.415647    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8fab226-5fd8-4f9b-9b17-9168fa14aa7d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:37:43.415669 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:43.415656    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8fab226-5fd8-4f9b-9b17-9168fa14aa7d-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:37:44.149081 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.148985    2577 generic.go:356] "Generic (PLEG): container finished" podID="b8fab226-5fd8-4f9b-9b17-9168fa14aa7d" containerID="622ee9fc52df85c19c6654fcb292920c372e29ffbee2dc8ee5a6e4e56848542f" exitCode=0
Jul 27 21:37:44.149081 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.149055    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-n59xt"
Jul 27 21:37:44.149412 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.149048    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-n59xt" event={"ID":"b8fab226-5fd8-4f9b-9b17-9168fa14aa7d","Type":"ContainerDied","Data":"622ee9fc52df85c19c6654fcb292920c372e29ffbee2dc8ee5a6e4e56848542f"}
Jul 27 21:37:44.149412 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.149217    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-n59xt" event={"ID":"b8fab226-5fd8-4f9b-9b17-9168fa14aa7d","Type":"ContainerDied","Data":"5b6dbbf71876cbc221c43912aed71a6de2cf6150f38855b1a105cee6d1e6d0bc"}
Jul 27 21:37:44.149412 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.149265    2577 scope.go:117] "RemoveContainer" containerID="e09b8598b158d6e8991fd1892496941abed39be17b9e7f0b0798318f0325fb04"
Jul 27 21:37:44.166670 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.166254    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:37:44.170187 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.170004    2577 scope.go:117] "RemoveContainer" containerID="622ee9fc52df85c19c6654fcb292920c372e29ffbee2dc8ee5a6e4e56848542f"
Jul 27 21:37:44.177328 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.177308    2577 scope.go:117] "RemoveContainer" containerID="a4098e036c723650c95b7fc1bfba60fa1dd65bf146b7e90229e5b2ace94de6a1"
Jul 27 21:37:44.181884 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.181862    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-n59xt"]
Jul 27 21:37:44.184651 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.184632    2577 scope.go:117] "RemoveContainer" containerID="e09b8598b158d6e8991fd1892496941abed39be17b9e7f0b0798318f0325fb04"
Jul 27 21:37:44.184976 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:37:44.184943    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e09b8598b158d6e8991fd1892496941abed39be17b9e7f0b0798318f0325fb04\": container with ID starting with e09b8598b158d6e8991fd1892496941abed39be17b9e7f0b0798318f0325fb04 not found: ID does not exist" containerID="e09b8598b158d6e8991fd1892496941abed39be17b9e7f0b0798318f0325fb04"
Jul 27 21:37:44.185078 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.184986    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e09b8598b158d6e8991fd1892496941abed39be17b9e7f0b0798318f0325fb04"} err="failed to get container status \"e09b8598b158d6e8991fd1892496941abed39be17b9e7f0b0798318f0325fb04\": rpc error: code = NotFound desc = could not find container \"e09b8598b158d6e8991fd1892496941abed39be17b9e7f0b0798318f0325fb04\": container with ID starting with e09b8598b158d6e8991fd1892496941abed39be17b9e7f0b0798318f0325fb04 not found: ID does not exist"
Jul 27 21:37:44.185078 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.185008    2577 scope.go:117] "RemoveContainer" containerID="622ee9fc52df85c19c6654fcb292920c372e29ffbee2dc8ee5a6e4e56848542f"
Jul 27 21:37:44.185441 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:37:44.185292    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"622ee9fc52df85c19c6654fcb292920c372e29ffbee2dc8ee5a6e4e56848542f\": container with ID starting with 622ee9fc52df85c19c6654fcb292920c372e29ffbee2dc8ee5a6e4e56848542f not found: ID does not exist" containerID="622ee9fc52df85c19c6654fcb292920c372e29ffbee2dc8ee5a6e4e56848542f"
Jul 27 21:37:44.185441 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.185324    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"622ee9fc52df85c19c6654fcb292920c372e29ffbee2dc8ee5a6e4e56848542f"} err="failed to get container status \"622ee9fc52df85c19c6654fcb292920c372e29ffbee2dc8ee5a6e4e56848542f\": rpc error: code = NotFound desc = could not find container \"622ee9fc52df85c19c6654fcb292920c372e29ffbee2dc8ee5a6e4e56848542f\": container with ID starting with 622ee9fc52df85c19c6654fcb292920c372e29ffbee2dc8ee5a6e4e56848542f not found: ID does not exist"
Jul 27 21:37:44.185441 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.185347    2577 scope.go:117] "RemoveContainer" containerID="a4098e036c723650c95b7fc1bfba60fa1dd65bf146b7e90229e5b2ace94de6a1"
Jul 27 21:37:44.185719 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:37:44.185701    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4098e036c723650c95b7fc1bfba60fa1dd65bf146b7e90229e5b2ace94de6a1\": container with ID starting with a4098e036c723650c95b7fc1bfba60fa1dd65bf146b7e90229e5b2ace94de6a1 not found: ID does not exist" containerID="a4098e036c723650c95b7fc1bfba60fa1dd65bf146b7e90229e5b2ace94de6a1"
Jul 27 21:37:44.185848 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.185727    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4098e036c723650c95b7fc1bfba60fa1dd65bf146b7e90229e5b2ace94de6a1"} err="failed to get container status \"a4098e036c723650c95b7fc1bfba60fa1dd65bf146b7e90229e5b2ace94de6a1\": rpc error: code = NotFound desc = could not find container \"a4098e036c723650c95b7fc1bfba60fa1dd65bf146b7e90229e5b2ace94de6a1\": container with ID starting with a4098e036c723650c95b7fc1bfba60fa1dd65bf146b7e90229e5b2ace94de6a1 not found: ID does not exist"
Jul 27 21:37:44.187035 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.187018    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-n59xt"]
Jul 27 21:37:44.222337 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.222310    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d"]
Jul 27 21:37:44.222610 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.222598    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8fab226-5fd8-4f9b-9b17-9168fa14aa7d" containerName="kserve-container"
Jul 27 21:37:44.222651 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.222612    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8fab226-5fd8-4f9b-9b17-9168fa14aa7d" containerName="kserve-container"
Jul 27 21:37:44.222651 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.222623    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8fab226-5fd8-4f9b-9b17-9168fa14aa7d" containerName="kube-rbac-proxy"
Jul 27 21:37:44.222651 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.222630    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8fab226-5fd8-4f9b-9b17-9168fa14aa7d" containerName="kube-rbac-proxy"
Jul 27 21:37:44.222651 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.222644    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8fab226-5fd8-4f9b-9b17-9168fa14aa7d" containerName="storage-initializer"
Jul 27 21:37:44.222651 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.222650    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8fab226-5fd8-4f9b-9b17-9168fa14aa7d" containerName="storage-initializer"
Jul 27 21:37:44.222833 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.222657    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a723442e-449c-40fd-9279-c24f2916e745" containerName="kube-rbac-proxy"
Jul 27 21:37:44.222833 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.222663    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a723442e-449c-40fd-9279-c24f2916e745" containerName="kube-rbac-proxy"
Jul 27 21:37:44.222833 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.222668    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a723442e-449c-40fd-9279-c24f2916e745" containerName="storage-initializer"
Jul 27 21:37:44.222833 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.222673    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a723442e-449c-40fd-9279-c24f2916e745" containerName="storage-initializer"
Jul 27 21:37:44.222833 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.222679    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a723442e-449c-40fd-9279-c24f2916e745" containerName="kserve-container"
Jul 27 21:37:44.222833 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.222684    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a723442e-449c-40fd-9279-c24f2916e745" containerName="kserve-container"
Jul 27 21:37:44.222833 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.222726    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a723442e-449c-40fd-9279-c24f2916e745" containerName="kube-rbac-proxy"
Jul 27 21:37:44.222833 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.222734    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8fab226-5fd8-4f9b-9b17-9168fa14aa7d" containerName="kserve-container"
Jul 27 21:37:44.222833 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.222740    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a723442e-449c-40fd-9279-c24f2916e745" containerName="kserve-container"
Jul 27 21:37:44.222833 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.222745    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8fab226-5fd8-4f9b-9b17-9168fa14aa7d" containerName="kube-rbac-proxy"
Jul 27 21:37:44.226888 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.226874    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d"
Jul 27 21:37:44.229713 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.229506    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 21:37:44.229713 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.229567    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v7c66\""
Jul 27 21:37:44.229713 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.229594    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Jul 27 21:37:44.229930 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.229744    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 21:37:44.229930 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.229885    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 21:37:44.230046 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.229963    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 21:37:44.230198 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.230170    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Jul 27 21:37:44.239330 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.239310    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d"]
Jul 27 21:37:44.253499 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.253482    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:37:44.323212 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.323185    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8dc831ee-6044-452f-9857-c906bfe7310d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d\" (UID: \"8dc831ee-6044-452f-9857-c906bfe7310d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d"
Jul 27 21:37:44.323581 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.323219    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8dc831ee-6044-452f-9857-c906bfe7310d-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d\" (UID: \"8dc831ee-6044-452f-9857-c906bfe7310d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d"
Jul 27 21:37:44.323581 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.323253    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8dc831ee-6044-452f-9857-c906bfe7310d-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d\" (UID: \"8dc831ee-6044-452f-9857-c906bfe7310d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d"
Jul 27 21:37:44.323581 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.323299    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8dc831ee-6044-452f-9857-c906bfe7310d-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d\" (UID: \"8dc831ee-6044-452f-9857-c906bfe7310d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d"
Jul 27 21:37:44.323581 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.323367    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88xmk\" (UniqueName: \"kubernetes.io/projected/8dc831ee-6044-452f-9857-c906bfe7310d-kube-api-access-88xmk\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d\" (UID: \"8dc831ee-6044-452f-9857-c906bfe7310d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d"
Jul 27 21:37:44.411436 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.411377    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:37:44.424465 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.424438    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88xmk\" (UniqueName: \"kubernetes.io/projected/8dc831ee-6044-452f-9857-c906bfe7310d-kube-api-access-88xmk\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d\" (UID: \"8dc831ee-6044-452f-9857-c906bfe7310d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d"
Jul 27 21:37:44.424589 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.424485    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8dc831ee-6044-452f-9857-c906bfe7310d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d\" (UID: \"8dc831ee-6044-452f-9857-c906bfe7310d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d"
Jul 27 21:37:44.424589 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.424504    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8dc831ee-6044-452f-9857-c906bfe7310d-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d\" (UID: \"8dc831ee-6044-452f-9857-c906bfe7310d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d"
Jul 27 21:37:44.424589 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.424526    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8dc831ee-6044-452f-9857-c906bfe7310d-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d\" (UID: \"8dc831ee-6044-452f-9857-c906bfe7310d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d"
Jul 27 21:37:44.424589 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.424558    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8dc831ee-6044-452f-9857-c906bfe7310d-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d\" (UID: \"8dc831ee-6044-452f-9857-c906bfe7310d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d"
Jul 27 21:37:44.424830 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:37:44.424643    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Jul 27 21:37:44.424830 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:37:44.424697    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dc831ee-6044-452f-9857-c906bfe7310d-proxy-tls podName:8dc831ee-6044-452f-9857-c906bfe7310d nodeName:}" failed. No retries permitted until 2026-07-27 21:37:44.924680559 +0000 UTC m=+1988.291870405 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8dc831ee-6044-452f-9857-c906bfe7310d-proxy-tls") pod "isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d" (UID: "8dc831ee-6044-452f-9857-c906bfe7310d") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Jul 27 21:37:44.424947 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.424882    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8dc831ee-6044-452f-9857-c906bfe7310d-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d\" (UID: \"8dc831ee-6044-452f-9857-c906bfe7310d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d"
Jul 27 21:37:44.425208 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.425188    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8dc831ee-6044-452f-9857-c906bfe7310d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d\" (UID: \"8dc831ee-6044-452f-9857-c906bfe7310d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d"
Jul 27 21:37:44.425278 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.425260    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8dc831ee-6044-452f-9857-c906bfe7310d-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d\" (UID: \"8dc831ee-6044-452f-9857-c906bfe7310d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d"
Jul 27 21:37:44.434736 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.434718    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88xmk\" (UniqueName: \"kubernetes.io/projected/8dc831ee-6044-452f-9857-c906bfe7310d-kube-api-access-88xmk\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d\" (UID: \"8dc831ee-6044-452f-9857-c906bfe7310d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d"
Jul 27 21:37:44.552756 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.552731    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:37:44.927891 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.927863    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8dc831ee-6044-452f-9857-c906bfe7310d-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d\" (UID: \"8dc831ee-6044-452f-9857-c906bfe7310d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d"
Jul 27 21:37:44.930304 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:44.930282    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8dc831ee-6044-452f-9857-c906bfe7310d-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d\" (UID: \"8dc831ee-6044-452f-9857-c906bfe7310d\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d"
Jul 27 21:37:45.135167 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:45.135136    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8fab226-5fd8-4f9b-9b17-9168fa14aa7d" path="/var/lib/kubelet/pods/b8fab226-5fd8-4f9b-9b17-9168fa14aa7d/volumes"
Jul 27 21:37:45.136837 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:45.136824    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d"
Jul 27 21:37:45.259254 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:45.259231    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d"]
Jul 27 21:37:45.261232 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:37:45.261205    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dc831ee_6044_452f_9857_c906bfe7310d.slice/crio-4ca922e1627c8460c756d6472f146604082d61ea12f10a219d3038d5b994979f WatchSource:0}: Error finding container 4ca922e1627c8460c756d6472f146604082d61ea12f10a219d3038d5b994979f: Status 404 returned error can't find the container with id 4ca922e1627c8460c756d6472f146604082d61ea12f10a219d3038d5b994979f
Jul 27 21:37:46.160982 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:46.160951    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d" event={"ID":"8dc831ee-6044-452f-9857-c906bfe7310d","Type":"ContainerStarted","Data":"a5abdda8683bd67c8a61e2d7a619d22ac561e550c2777798885afdd21d9b5d39"}
Jul 27 21:37:46.161385 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:46.161008    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d" event={"ID":"8dc831ee-6044-452f-9857-c906bfe7310d","Type":"ContainerStarted","Data":"4ca922e1627c8460c756d6472f146604082d61ea12f10a219d3038d5b994979f"}
Jul 27 21:37:49.173854 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:49.173827    2577 generic.go:356] "Generic (PLEG): container finished" podID="8dc831ee-6044-452f-9857-c906bfe7310d" containerID="a5abdda8683bd67c8a61e2d7a619d22ac561e550c2777798885afdd21d9b5d39" exitCode=0
Jul 27 21:37:49.174208 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:49.173893    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d" event={"ID":"8dc831ee-6044-452f-9857-c906bfe7310d","Type":"ContainerDied","Data":"a5abdda8683bd67c8a61e2d7a619d22ac561e550c2777798885afdd21d9b5d39"}
Jul 27 21:37:50.180010 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:50.179974    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d" event={"ID":"8dc831ee-6044-452f-9857-c906bfe7310d","Type":"ContainerStarted","Data":"ab30114f43b55b5fb43b4cadd834b0d82e40d01847eda0b346b6949b73a8fd2e"}
Jul 27 21:37:50.180469 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:50.180065    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d" event={"ID":"8dc831ee-6044-452f-9857-c906bfe7310d","Type":"ContainerStarted","Data":"aaa094e4bda63958bfb0e901283aede67734e56930f2057de337041e90051675"}
Jul 27 21:37:50.180469 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:50.180278    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d"
Jul 27 21:37:50.201410 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:50.201370    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d" podStartSLOduration=2.380264411 podStartE2EDuration="6.201357859s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.821093448s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:37:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:37:50.199604536 +0000 UTC m=+1993.566794400" watchObservedRunningTime="2026-07-27 21:37:50.201357859 +0000 UTC m=+1993.568547722"
Jul 27 21:37:51.184611 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:51.184576    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d"
Jul 27 21:37:51.185856 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:51.185829    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d" podUID="8dc831ee-6044-452f-9857-c906bfe7310d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 27 21:37:52.189301 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:52.189264    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d" podUID="8dc831ee-6044-452f-9857-c906bfe7310d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 27 21:37:57.194408 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:57.194373    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d"
Jul 27 21:37:57.194985 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:37:57.194956    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d" podUID="8dc831ee-6044-452f-9857-c906bfe7310d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 27 21:38:07.195704 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:38:07.195665    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d" podUID="8dc831ee-6044-452f-9857-c906bfe7310d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 27 21:38:17.195206 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:38:17.195162    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d" podUID="8dc831ee-6044-452f-9857-c906bfe7310d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 27 21:38:27.195269 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:38:27.195216    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d" podUID="8dc831ee-6044-452f-9857-c906bfe7310d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 27 21:38:37.195212 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:38:37.195171    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d" podUID="8dc831ee-6044-452f-9857-c906bfe7310d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 27 21:38:47.195707 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:38:47.195668    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d" podUID="8dc831ee-6044-452f-9857-c906bfe7310d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 27 21:38:57.195034 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:38:57.194995    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d" podUID="8dc831ee-6044-452f-9857-c906bfe7310d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 27 21:39:03.986986 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:39:03.986884    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:39:07.195906 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:39:07.195875    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d"
Jul 27 21:39:14.272883 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:39:14.272850    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:40:32.327911 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:32.327836    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:40:44.444279 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:44.444242    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d"]
Jul 27 21:40:44.444669 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:44.444545    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d" podUID="8dc831ee-6044-452f-9857-c906bfe7310d" containerName="kserve-container" containerID="cri-o://aaa094e4bda63958bfb0e901283aede67734e56930f2057de337041e90051675" gracePeriod=30
Jul 27 21:40:44.444669 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:44.444627    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d" podUID="8dc831ee-6044-452f-9857-c906bfe7310d" containerName="kube-rbac-proxy" containerID="cri-o://ab30114f43b55b5fb43b4cadd834b0d82e40d01847eda0b346b6949b73a8fd2e" gracePeriod=30
Jul 27 21:40:44.493949 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:44.493910    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:40:44.848426 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:44.848330    2577 generic.go:356] "Generic (PLEG): container finished" podID="8dc831ee-6044-452f-9857-c906bfe7310d" containerID="ab30114f43b55b5fb43b4cadd834b0d82e40d01847eda0b346b6949b73a8fd2e" exitCode=2
Jul 27 21:40:44.848426 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:44.848372    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d" event={"ID":"8dc831ee-6044-452f-9857-c906bfe7310d","Type":"ContainerDied","Data":"ab30114f43b55b5fb43b4cadd834b0d82e40d01847eda0b346b6949b73a8fd2e"}
Jul 27 21:40:44.942252 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:44.942217    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:40:45.001860 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:45.001827    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:40:47.190258 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:47.190222    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d" podUID="8dc831ee-6044-452f-9857-c906bfe7310d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused"
Jul 27 21:40:47.195712 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:47.195684    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d" podUID="8dc831ee-6044-452f-9857-c906bfe7310d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 27 21:40:48.781488 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:48.781466    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d"
Jul 27 21:40:48.836357 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:48.836335    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88xmk\" (UniqueName: \"kubernetes.io/projected/8dc831ee-6044-452f-9857-c906bfe7310d-kube-api-access-88xmk\") pod \"8dc831ee-6044-452f-9857-c906bfe7310d\" (UID: \"8dc831ee-6044-452f-9857-c906bfe7310d\") "
Jul 27 21:40:48.836488 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:48.836386    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8dc831ee-6044-452f-9857-c906bfe7310d-cabundle-cert\") pod \"8dc831ee-6044-452f-9857-c906bfe7310d\" (UID: \"8dc831ee-6044-452f-9857-c906bfe7310d\") "
Jul 27 21:40:48.836488 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:48.836412    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8dc831ee-6044-452f-9857-c906bfe7310d-proxy-tls\") pod \"8dc831ee-6044-452f-9857-c906bfe7310d\" (UID: \"8dc831ee-6044-452f-9857-c906bfe7310d\") "
Jul 27 21:40:48.836615 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:48.836594    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8dc831ee-6044-452f-9857-c906bfe7310d-kserve-provision-location\") pod \"8dc831ee-6044-452f-9857-c906bfe7310d\" (UID: \"8dc831ee-6044-452f-9857-c906bfe7310d\") "
Jul 27 21:40:48.836669 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:48.836649    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8dc831ee-6044-452f-9857-c906bfe7310d-kube-rbac-proxy-sar-config\") pod \"8dc831ee-6044-452f-9857-c906bfe7310d\" (UID: \"8dc831ee-6044-452f-9857-c906bfe7310d\") "
Jul 27 21:40:48.836795 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:48.836748    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dc831ee-6044-452f-9857-c906bfe7310d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8dc831ee-6044-452f-9857-c906bfe7310d" (UID: "8dc831ee-6044-452f-9857-c906bfe7310d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:40:48.836939 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:48.836921    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8dc831ee-6044-452f-9857-c906bfe7310d-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:40:48.837048 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:48.836939    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dc831ee-6044-452f-9857-c906bfe7310d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8dc831ee-6044-452f-9857-c906bfe7310d" (UID: "8dc831ee-6044-452f-9857-c906bfe7310d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:40:48.837105 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:48.837068    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dc831ee-6044-452f-9857-c906bfe7310d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8dc831ee-6044-452f-9857-c906bfe7310d" (UID: "8dc831ee-6044-452f-9857-c906bfe7310d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:40:48.838497 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:48.838477    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dc831ee-6044-452f-9857-c906bfe7310d-kube-api-access-88xmk" (OuterVolumeSpecName: "kube-api-access-88xmk") pod "8dc831ee-6044-452f-9857-c906bfe7310d" (UID: "8dc831ee-6044-452f-9857-c906bfe7310d"). InnerVolumeSpecName "kube-api-access-88xmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:40:48.838554 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:48.838495    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dc831ee-6044-452f-9857-c906bfe7310d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8dc831ee-6044-452f-9857-c906bfe7310d" (UID: "8dc831ee-6044-452f-9857-c906bfe7310d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:40:48.868238 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:48.868182    2577 generic.go:356] "Generic (PLEG): container finished" podID="8dc831ee-6044-452f-9857-c906bfe7310d" containerID="aaa094e4bda63958bfb0e901283aede67734e56930f2057de337041e90051675" exitCode=0
Jul 27 21:40:48.868238 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:48.868222    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d" event={"ID":"8dc831ee-6044-452f-9857-c906bfe7310d","Type":"ContainerDied","Data":"aaa094e4bda63958bfb0e901283aede67734e56930f2057de337041e90051675"}
Jul 27 21:40:48.868405 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:48.868242    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d"
Jul 27 21:40:48.868405 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:48.868280    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d" event={"ID":"8dc831ee-6044-452f-9857-c906bfe7310d","Type":"ContainerDied","Data":"4ca922e1627c8460c756d6472f146604082d61ea12f10a219d3038d5b994979f"}
Jul 27 21:40:48.868405 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:48.868306    2577 scope.go:117] "RemoveContainer" containerID="ab30114f43b55b5fb43b4cadd834b0d82e40d01847eda0b346b6949b73a8fd2e"
Jul 27 21:40:48.876177 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:48.876081    2577 scope.go:117] "RemoveContainer" containerID="aaa094e4bda63958bfb0e901283aede67734e56930f2057de337041e90051675"
Jul 27 21:40:48.883294 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:48.883279    2577 scope.go:117] "RemoveContainer" containerID="a5abdda8683bd67c8a61e2d7a619d22ac561e550c2777798885afdd21d9b5d39"
Jul 27 21:40:48.889926 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:48.889901    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d"]
Jul 27 21:40:48.890613 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:48.890594    2577 scope.go:117] "RemoveContainer" containerID="ab30114f43b55b5fb43b4cadd834b0d82e40d01847eda0b346b6949b73a8fd2e"
Jul 27 21:40:48.890885 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:40:48.890864    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab30114f43b55b5fb43b4cadd834b0d82e40d01847eda0b346b6949b73a8fd2e\": container with ID starting with ab30114f43b55b5fb43b4cadd834b0d82e40d01847eda0b346b6949b73a8fd2e not found: ID does not exist" containerID="ab30114f43b55b5fb43b4cadd834b0d82e40d01847eda0b346b6949b73a8fd2e"
Jul 27 21:40:48.890949 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:48.890892    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab30114f43b55b5fb43b4cadd834b0d82e40d01847eda0b346b6949b73a8fd2e"} err="failed to get container status \"ab30114f43b55b5fb43b4cadd834b0d82e40d01847eda0b346b6949b73a8fd2e\": rpc error: code = NotFound desc = could not find container \"ab30114f43b55b5fb43b4cadd834b0d82e40d01847eda0b346b6949b73a8fd2e\": container with ID starting with ab30114f43b55b5fb43b4cadd834b0d82e40d01847eda0b346b6949b73a8fd2e not found: ID does not exist"
Jul 27 21:40:48.890949 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:48.890909    2577 scope.go:117] "RemoveContainer" containerID="aaa094e4bda63958bfb0e901283aede67734e56930f2057de337041e90051675"
Jul 27 21:40:48.891142 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:40:48.891126    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaa094e4bda63958bfb0e901283aede67734e56930f2057de337041e90051675\": container with ID starting with aaa094e4bda63958bfb0e901283aede67734e56930f2057de337041e90051675 not found: ID does not exist" containerID="aaa094e4bda63958bfb0e901283aede67734e56930f2057de337041e90051675"
Jul 27 21:40:48.891181 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:48.891148    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaa094e4bda63958bfb0e901283aede67734e56930f2057de337041e90051675"} err="failed to get container status \"aaa094e4bda63958bfb0e901283aede67734e56930f2057de337041e90051675\": rpc error: code = NotFound desc = could not find container \"aaa094e4bda63958bfb0e901283aede67734e56930f2057de337041e90051675\": container with ID starting with aaa094e4bda63958bfb0e901283aede67734e56930f2057de337041e90051675 not found: ID does not exist"
Jul 27 21:40:48.891181 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:48.891162    2577 scope.go:117] "RemoveContainer" containerID="a5abdda8683bd67c8a61e2d7a619d22ac561e550c2777798885afdd21d9b5d39"
Jul 27 21:40:48.891369 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:40:48.891352    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5abdda8683bd67c8a61e2d7a619d22ac561e550c2777798885afdd21d9b5d39\": container with ID starting with a5abdda8683bd67c8a61e2d7a619d22ac561e550c2777798885afdd21d9b5d39 not found: ID does not exist" containerID="a5abdda8683bd67c8a61e2d7a619d22ac561e550c2777798885afdd21d9b5d39"
Jul 27 21:40:48.891405 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:48.891374    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5abdda8683bd67c8a61e2d7a619d22ac561e550c2777798885afdd21d9b5d39"} err="failed to get container status \"a5abdda8683bd67c8a61e2d7a619d22ac561e550c2777798885afdd21d9b5d39\": rpc error: code = NotFound desc = could not find container \"a5abdda8683bd67c8a61e2d7a619d22ac561e550c2777798885afdd21d9b5d39\": container with ID starting with a5abdda8683bd67c8a61e2d7a619d22ac561e550c2777798885afdd21d9b5d39 not found: ID does not exist"
Jul 27 21:40:48.894789 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:48.894753    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-x2k8d"]
Jul 27 21:40:48.938211 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:48.938193    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8dc831ee-6044-452f-9857-c906bfe7310d-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:40:48.938211 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:48.938212    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8dc831ee-6044-452f-9857-c906bfe7310d-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:40:48.938326 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:48.938222    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8dc831ee-6044-452f-9857-c906bfe7310d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:40:48.938326 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:48.938232    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88xmk\" (UniqueName: \"kubernetes.io/projected/8dc831ee-6044-452f-9857-c906bfe7310d-kube-api-access-88xmk\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:40:49.135343 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:49.135276    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8dc831ee-6044-452f-9857-c906bfe7310d" path="/var/lib/kubelet/pods/8dc831ee-6044-452f-9857-c906bfe7310d/volumes"
Jul 27 21:40:49.552356 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:49.552319    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:40:49.625868 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:49.625835    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-rtptb"]
Jul 27 21:40:49.626109 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:49.626094    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dc831ee-6044-452f-9857-c906bfe7310d" containerName="storage-initializer"
Jul 27 21:40:49.626173 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:49.626110    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dc831ee-6044-452f-9857-c906bfe7310d" containerName="storage-initializer"
Jul 27 21:40:49.626173 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:49.626126    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dc831ee-6044-452f-9857-c906bfe7310d" containerName="kserve-container"
Jul 27 21:40:49.626173 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:49.626132    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dc831ee-6044-452f-9857-c906bfe7310d" containerName="kserve-container"
Jul 27 21:40:49.626173 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:49.626146    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dc831ee-6044-452f-9857-c906bfe7310d" containerName="kube-rbac-proxy"
Jul 27 21:40:49.626173 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:49.626152    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dc831ee-6044-452f-9857-c906bfe7310d" containerName="kube-rbac-proxy"
Jul 27 21:40:49.626423 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:49.626195    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8dc831ee-6044-452f-9857-c906bfe7310d" containerName="kserve-container"
Jul 27 21:40:49.626423 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:49.626206    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8dc831ee-6044-452f-9857-c906bfe7310d" containerName="kube-rbac-proxy"
Jul 27 21:40:49.630822 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:49.630803    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-rtptb"
Jul 27 21:40:49.630968 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:49.630946    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:40:49.633277 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:49.633257    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 21:40:49.633392 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:49.633347    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Jul 27 21:40:49.633470 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:49.633347    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 21:40:49.634197 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:49.634173    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 21:40:49.634305 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:49.634238    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Jul 27 21:40:49.634305 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:49.634245    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v7c66\""
Jul 27 21:40:49.634607 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:49.634589    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 21:40:49.644179 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:49.644157    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-rtptb"]
Jul 27 21:40:49.744037 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:49.744008    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/280747d9-9219-4d4f-a4b0-b442260d38b4-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-77f778d864-rtptb\" (UID: \"280747d9-9219-4d4f-a4b0-b442260d38b4\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-rtptb"
Jul 27 21:40:49.744151 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:49.744060    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/280747d9-9219-4d4f-a4b0-b442260d38b4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-77f778d864-rtptb\" (UID: \"280747d9-9219-4d4f-a4b0-b442260d38b4\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-rtptb"
Jul 27 21:40:49.744151 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:49.744085    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/280747d9-9219-4d4f-a4b0-b442260d38b4-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-77f778d864-rtptb\" (UID: \"280747d9-9219-4d4f-a4b0-b442260d38b4\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-rtptb"
Jul 27 21:40:49.744151 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:49.744135    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/280747d9-9219-4d4f-a4b0-b442260d38b4-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-77f778d864-rtptb\" (UID: \"280747d9-9219-4d4f-a4b0-b442260d38b4\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-rtptb"
Jul 27 21:40:49.744252 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:49.744173    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpcz6\" (UniqueName: \"kubernetes.io/projected/280747d9-9219-4d4f-a4b0-b442260d38b4-kube-api-access-hpcz6\") pod \"isvc-canary-stop-v2-predictor-77f778d864-rtptb\" (UID: \"280747d9-9219-4d4f-a4b0-b442260d38b4\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-rtptb"
Jul 27 21:40:49.833467 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:49.833390    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:40:49.845418 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:49.845393    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/280747d9-9219-4d4f-a4b0-b442260d38b4-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-77f778d864-rtptb\" (UID: \"280747d9-9219-4d4f-a4b0-b442260d38b4\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-rtptb"
Jul 27 21:40:49.845521 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:49.845437    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/280747d9-9219-4d4f-a4b0-b442260d38b4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-77f778d864-rtptb\" (UID: \"280747d9-9219-4d4f-a4b0-b442260d38b4\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-rtptb"
Jul 27 21:40:49.845521 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:49.845461    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/280747d9-9219-4d4f-a4b0-b442260d38b4-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-77f778d864-rtptb\" (UID: \"280747d9-9219-4d4f-a4b0-b442260d38b4\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-rtptb"
Jul 27 21:40:49.845521 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:49.845517    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/280747d9-9219-4d4f-a4b0-b442260d38b4-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-77f778d864-rtptb\" (UID: \"280747d9-9219-4d4f-a4b0-b442260d38b4\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-rtptb"
Jul 27 21:40:49.845709 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:49.845555    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpcz6\" (UniqueName: \"kubernetes.io/projected/280747d9-9219-4d4f-a4b0-b442260d38b4-kube-api-access-hpcz6\") pod \"isvc-canary-stop-v2-predictor-77f778d864-rtptb\" (UID: \"280747d9-9219-4d4f-a4b0-b442260d38b4\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-rtptb"
Jul 27 21:40:49.845922 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:49.845903    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/280747d9-9219-4d4f-a4b0-b442260d38b4-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-77f778d864-rtptb\" (UID: \"280747d9-9219-4d4f-a4b0-b442260d38b4\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-rtptb"
Jul 27 21:40:49.846207 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:49.846185    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/280747d9-9219-4d4f-a4b0-b442260d38b4-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-77f778d864-rtptb\" (UID: \"280747d9-9219-4d4f-a4b0-b442260d38b4\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-rtptb"
Jul 27 21:40:49.846312 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:49.846208    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/280747d9-9219-4d4f-a4b0-b442260d38b4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-77f778d864-rtptb\" (UID: \"280747d9-9219-4d4f-a4b0-b442260d38b4\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-rtptb"
Jul 27 21:40:49.847923 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:49.847902    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/280747d9-9219-4d4f-a4b0-b442260d38b4-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-77f778d864-rtptb\" (UID: \"280747d9-9219-4d4f-a4b0-b442260d38b4\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-rtptb"
Jul 27 21:40:49.854162 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:49.854141    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpcz6\" (UniqueName: \"kubernetes.io/projected/280747d9-9219-4d4f-a4b0-b442260d38b4-kube-api-access-hpcz6\") pod \"isvc-canary-stop-v2-predictor-77f778d864-rtptb\" (UID: \"280747d9-9219-4d4f-a4b0-b442260d38b4\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-rtptb"
Jul 27 21:40:49.943314 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:49.943293    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-rtptb"
Jul 27 21:40:49.970903 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:49.970876    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:40:50.069793 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:50.069744    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-rtptb"]
Jul 27 21:40:50.072343 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:40:50.072317    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod280747d9_9219_4d4f_a4b0_b442260d38b4.slice/crio-6d5999813dc04506cf74a20e095332448e4ad739776027c184548a243c887333 WatchSource:0}: Error finding container 6d5999813dc04506cf74a20e095332448e4ad739776027c184548a243c887333: Status 404 returned error can't find the container with id 6d5999813dc04506cf74a20e095332448e4ad739776027c184548a243c887333
Jul 27 21:40:50.878203 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:50.878166    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-rtptb" event={"ID":"280747d9-9219-4d4f-a4b0-b442260d38b4","Type":"ContainerStarted","Data":"9a84af0b07268ab571b8abc50ff6fb34117168a3f2fbc520be2677079b581c42"}
Jul 27 21:40:50.878563 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:50.878253    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-rtptb" event={"ID":"280747d9-9219-4d4f-a4b0-b442260d38b4","Type":"ContainerStarted","Data":"6d5999813dc04506cf74a20e095332448e4ad739776027c184548a243c887333"}
Jul 27 21:40:54.894557 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:54.894526    2577 generic.go:356] "Generic (PLEG): container finished" podID="280747d9-9219-4d4f-a4b0-b442260d38b4" containerID="9a84af0b07268ab571b8abc50ff6fb34117168a3f2fbc520be2677079b581c42" exitCode=0
Jul 27 21:40:54.895021 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:54.894599    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-rtptb" event={"ID":"280747d9-9219-4d4f-a4b0-b442260d38b4","Type":"ContainerDied","Data":"9a84af0b07268ab571b8abc50ff6fb34117168a3f2fbc520be2677079b581c42"}
Jul 27 21:40:55.900942 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:55.900907    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-rtptb" event={"ID":"280747d9-9219-4d4f-a4b0-b442260d38b4","Type":"ContainerStarted","Data":"fa9bc3ee884dbd46d58039e7371bc6f29435b701f683f5fec65ce8869781745b"}
Jul 27 21:40:55.901420 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:55.900978    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-rtptb" event={"ID":"280747d9-9219-4d4f-a4b0-b442260d38b4","Type":"ContainerStarted","Data":"19ffb8f454f621a5b32e445455d85f3c995d6adcc30c77c6098e388da9e6455b"}
Jul 27 21:40:55.901420 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:55.901186    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-rtptb"
Jul 27 21:40:55.920330 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:55.920277    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-rtptb" podStartSLOduration=3.148441758 podStartE2EDuration="6.92025685s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.771815092s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:40:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:40:55.919088719 +0000 UTC m=+2179.286278587" watchObservedRunningTime="2026-07-27 21:40:55.92025685 +0000 UTC m=+2179.287446776"
Jul 27 21:40:56.904885 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:56.904852    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-rtptb"
Jul 27 21:40:56.906106 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:56.906075    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-rtptb" podUID="280747d9-9219-4d4f-a4b0-b442260d38b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 27 21:40:57.908438 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:40:57.908390    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-rtptb" podUID="280747d9-9219-4d4f-a4b0-b442260d38b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 27 21:41:02.913406 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:41:02.913377    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-rtptb"
Jul 27 21:41:02.913913 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:41:02.913882    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-rtptb" podUID="280747d9-9219-4d4f-a4b0-b442260d38b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 27 21:41:12.913917 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:41:12.913867    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-rtptb" podUID="280747d9-9219-4d4f-a4b0-b442260d38b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 27 21:41:22.914455 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:41:22.914416    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-rtptb" podUID="280747d9-9219-4d4f-a4b0-b442260d38b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 27 21:41:32.914627 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:41:32.914573    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-rtptb" podUID="280747d9-9219-4d4f-a4b0-b442260d38b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 27 21:41:42.914559 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:41:42.914515    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-rtptb" podUID="280747d9-9219-4d4f-a4b0-b442260d38b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 27 21:41:52.914225 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:41:52.914172    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-rtptb" podUID="280747d9-9219-4d4f-a4b0-b442260d38b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 27 21:42:02.914007 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:42:02.913925    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-rtptb" podUID="280747d9-9219-4d4f-a4b0-b442260d38b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 27 21:42:09.134896 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:42:09.134866    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-rtptb"
Jul 27 21:42:19.630471 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:42:19.630436    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:42:19.644031 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:42:19.644004    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-rtptb"]
Jul 27 21:42:19.644256 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:42:19.644220    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-rtptb" podUID="280747d9-9219-4d4f-a4b0-b442260d38b4" containerName="kserve-container" containerID="cri-o://19ffb8f454f621a5b32e445455d85f3c995d6adcc30c77c6098e388da9e6455b" gracePeriod=30
Jul 27 21:42:19.644342 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:42:19.644253    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-rtptb" podUID="280747d9-9219-4d4f-a4b0-b442260d38b4" containerName="kube-rbac-proxy" containerID="cri-o://fa9bc3ee884dbd46d58039e7371bc6f29435b701f683f5fec65ce8869781745b" gracePeriod=30
Jul 27 21:42:19.730153 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:42:19.730127    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:42:20.220397 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:42:20.220367    2577 generic.go:356] "Generic (PLEG): container finished" podID="280747d9-9219-4d4f-a4b0-b442260d38b4" containerID="fa9bc3ee884dbd46d58039e7371bc6f29435b701f683f5fec65ce8869781745b" exitCode=2
Jul 27 21:42:20.220574 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:42:20.220414    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-rtptb" event={"ID":"280747d9-9219-4d4f-a4b0-b442260d38b4","Type":"ContainerDied","Data":"fa9bc3ee884dbd46d58039e7371bc6f29435b701f683f5fec65ce8869781745b"}
Jul 27 21:42:22.909287 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:42:22.909241    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-rtptb" podUID="280747d9-9219-4d4f-a4b0-b442260d38b4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Jul 27 21:42:24.081652 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:42:24.081622    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-rtptb"
Jul 27 21:42:24.171558 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:42:24.171530    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/280747d9-9219-4d4f-a4b0-b442260d38b4-kube-rbac-proxy-sar-config\") pod \"280747d9-9219-4d4f-a4b0-b442260d38b4\" (UID: \"280747d9-9219-4d4f-a4b0-b442260d38b4\") "
Jul 27 21:42:24.171693 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:42:24.171577    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/280747d9-9219-4d4f-a4b0-b442260d38b4-cabundle-cert\") pod \"280747d9-9219-4d4f-a4b0-b442260d38b4\" (UID: \"280747d9-9219-4d4f-a4b0-b442260d38b4\") "
Jul 27 21:42:24.171693 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:42:24.171594    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/280747d9-9219-4d4f-a4b0-b442260d38b4-proxy-tls\") pod \"280747d9-9219-4d4f-a4b0-b442260d38b4\" (UID: \"280747d9-9219-4d4f-a4b0-b442260d38b4\") "
Jul 27 21:42:24.171693 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:42:24.171614    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/280747d9-9219-4d4f-a4b0-b442260d38b4-kserve-provision-location\") pod \"280747d9-9219-4d4f-a4b0-b442260d38b4\" (UID: \"280747d9-9219-4d4f-a4b0-b442260d38b4\") "
Jul 27 21:42:24.171693 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:42:24.171666    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hpcz6\" (UniqueName: \"kubernetes.io/projected/280747d9-9219-4d4f-a4b0-b442260d38b4-kube-api-access-hpcz6\") pod \"280747d9-9219-4d4f-a4b0-b442260d38b4\" (UID: \"280747d9-9219-4d4f-a4b0-b442260d38b4\") "
Jul 27 21:42:24.171952 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:42:24.171929    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/280747d9-9219-4d4f-a4b0-b442260d38b4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "280747d9-9219-4d4f-a4b0-b442260d38b4" (UID: "280747d9-9219-4d4f-a4b0-b442260d38b4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:42:24.171990 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:42:24.171931    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/280747d9-9219-4d4f-a4b0-b442260d38b4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "280747d9-9219-4d4f-a4b0-b442260d38b4" (UID: "280747d9-9219-4d4f-a4b0-b442260d38b4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:42:24.172150 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:42:24.172126    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/280747d9-9219-4d4f-a4b0-b442260d38b4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "280747d9-9219-4d4f-a4b0-b442260d38b4" (UID: "280747d9-9219-4d4f-a4b0-b442260d38b4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:42:24.173685 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:42:24.173648    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/280747d9-9219-4d4f-a4b0-b442260d38b4-kube-api-access-hpcz6" (OuterVolumeSpecName: "kube-api-access-hpcz6") pod "280747d9-9219-4d4f-a4b0-b442260d38b4" (UID: "280747d9-9219-4d4f-a4b0-b442260d38b4"). InnerVolumeSpecName "kube-api-access-hpcz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:42:24.173777 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:42:24.173709    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/280747d9-9219-4d4f-a4b0-b442260d38b4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "280747d9-9219-4d4f-a4b0-b442260d38b4" (UID: "280747d9-9219-4d4f-a4b0-b442260d38b4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:42:24.237776 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:42:24.237730    2577 generic.go:356] "Generic (PLEG): container finished" podID="280747d9-9219-4d4f-a4b0-b442260d38b4" containerID="19ffb8f454f621a5b32e445455d85f3c995d6adcc30c77c6098e388da9e6455b" exitCode=0
Jul 27 21:42:24.237917 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:42:24.237801    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-rtptb"
Jul 27 21:42:24.237917 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:42:24.237805    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-rtptb" event={"ID":"280747d9-9219-4d4f-a4b0-b442260d38b4","Type":"ContainerDied","Data":"19ffb8f454f621a5b32e445455d85f3c995d6adcc30c77c6098e388da9e6455b"}
Jul 27 21:42:24.237917 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:42:24.237870    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-rtptb" event={"ID":"280747d9-9219-4d4f-a4b0-b442260d38b4","Type":"ContainerDied","Data":"6d5999813dc04506cf74a20e095332448e4ad739776027c184548a243c887333"}
Jul 27 21:42:24.237917 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:42:24.237898    2577 scope.go:117] "RemoveContainer" containerID="fa9bc3ee884dbd46d58039e7371bc6f29435b701f683f5fec65ce8869781745b"
Jul 27 21:42:24.248055 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:42:24.248039    2577 scope.go:117] "RemoveContainer" containerID="19ffb8f454f621a5b32e445455d85f3c995d6adcc30c77c6098e388da9e6455b"
Jul 27 21:42:24.254990 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:42:24.254972    2577 scope.go:117] "RemoveContainer" containerID="9a84af0b07268ab571b8abc50ff6fb34117168a3f2fbc520be2677079b581c42"
Jul 27 21:42:24.260425 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:42:24.260404    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-rtptb"]
Jul 27 21:42:24.262426 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:42:24.262404    2577 scope.go:117] "RemoveContainer" containerID="fa9bc3ee884dbd46d58039e7371bc6f29435b701f683f5fec65ce8869781745b"
Jul 27 21:42:24.262806 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:42:24.262743    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa9bc3ee884dbd46d58039e7371bc6f29435b701f683f5fec65ce8869781745b\": container with ID starting with fa9bc3ee884dbd46d58039e7371bc6f29435b701f683f5fec65ce8869781745b not found: ID does not exist" containerID="fa9bc3ee884dbd46d58039e7371bc6f29435b701f683f5fec65ce8869781745b"
Jul 27 21:42:24.262905 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:42:24.262799    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa9bc3ee884dbd46d58039e7371bc6f29435b701f683f5fec65ce8869781745b"} err="failed to get container status \"fa9bc3ee884dbd46d58039e7371bc6f29435b701f683f5fec65ce8869781745b\": rpc error: code = NotFound desc = could not find container \"fa9bc3ee884dbd46d58039e7371bc6f29435b701f683f5fec65ce8869781745b\": container with ID starting with fa9bc3ee884dbd46d58039e7371bc6f29435b701f683f5fec65ce8869781745b not found: ID does not exist"
Jul 27 21:42:24.262905 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:42:24.262825    2577 scope.go:117] "RemoveContainer" containerID="19ffb8f454f621a5b32e445455d85f3c995d6adcc30c77c6098e388da9e6455b"
Jul 27 21:42:24.263154 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:42:24.263127    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19ffb8f454f621a5b32e445455d85f3c995d6adcc30c77c6098e388da9e6455b\": container with ID starting with 19ffb8f454f621a5b32e445455d85f3c995d6adcc30c77c6098e388da9e6455b not found: ID does not exist" containerID="19ffb8f454f621a5b32e445455d85f3c995d6adcc30c77c6098e388da9e6455b"
Jul 27 21:42:24.263261 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:42:24.263162    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19ffb8f454f621a5b32e445455d85f3c995d6adcc30c77c6098e388da9e6455b"} err="failed to get container status \"19ffb8f454f621a5b32e445455d85f3c995d6adcc30c77c6098e388da9e6455b\": rpc error: code = NotFound desc = could not find container \"19ffb8f454f621a5b32e445455d85f3c995d6adcc30c77c6098e388da9e6455b\": container with ID starting with 19ffb8f454f621a5b32e445455d85f3c995d6adcc30c77c6098e388da9e6455b not found: ID does not exist"
Jul 27 21:42:24.263261 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:42:24.263182    2577 scope.go:117] "RemoveContainer" containerID="9a84af0b07268ab571b8abc50ff6fb34117168a3f2fbc520be2677079b581c42"
Jul 27 21:42:24.263417 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:42:24.263402    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a84af0b07268ab571b8abc50ff6fb34117168a3f2fbc520be2677079b581c42\": container with ID starting with 9a84af0b07268ab571b8abc50ff6fb34117168a3f2fbc520be2677079b581c42 not found: ID does not exist" containerID="9a84af0b07268ab571b8abc50ff6fb34117168a3f2fbc520be2677079b581c42"
Jul 27 21:42:24.263480 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:42:24.263419    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a84af0b07268ab571b8abc50ff6fb34117168a3f2fbc520be2677079b581c42"} err="failed to get container status \"9a84af0b07268ab571b8abc50ff6fb34117168a3f2fbc520be2677079b581c42\": rpc error: code = NotFound desc = could not find container \"9a84af0b07268ab571b8abc50ff6fb34117168a3f2fbc520be2677079b581c42\": container with ID starting with 9a84af0b07268ab571b8abc50ff6fb34117168a3f2fbc520be2677079b581c42 not found: ID does not exist"
Jul 27 21:42:24.264416 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:42:24.264395    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-rtptb"]
Jul 27 21:42:24.272806 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:42:24.272789    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/280747d9-9219-4d4f-a4b0-b442260d38b4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:42:24.272878 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:42:24.272807    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/280747d9-9219-4d4f-a4b0-b442260d38b4-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:42:24.272878 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:42:24.272816    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/280747d9-9219-4d4f-a4b0-b442260d38b4-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:42:24.272878 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:42:24.272825    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/280747d9-9219-4d4f-a4b0-b442260d38b4-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:42:24.272878 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:42:24.272834    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpcz6\" (UniqueName: \"kubernetes.io/projected/280747d9-9219-4d4f-a4b0-b442260d38b4-kube-api-access-hpcz6\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:42:25.135655 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:42:25.135618    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="280747d9-9219-4d4f-a4b0-b442260d38b4" path="/var/lib/kubelet/pods/280747d9-9219-4d4f-a4b0-b442260d38b4/volumes"
Jul 27 21:42:29.847733 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:42:29.847683    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:42:30.143244 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:42:30.143164    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:43:12.749607 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:43:12.749567    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:44:00.137793 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:44:00.137736    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:44:05.211448 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:44:05.211385    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:44:05.506289 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:44:05.506188    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:45:35.921936 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:45:35.921898    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:45:40.928544 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:45:40.928505    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:45:40.980191 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:45:40.980158    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-llwtf"]
Jul 27 21:45:40.980503 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:45:40.980489    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="280747d9-9219-4d4f-a4b0-b442260d38b4" containerName="kserve-container"
Jul 27 21:45:40.980572 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:45:40.980504    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="280747d9-9219-4d4f-a4b0-b442260d38b4" containerName="kserve-container"
Jul 27 21:45:40.980572 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:45:40.980513    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="280747d9-9219-4d4f-a4b0-b442260d38b4" containerName="kube-rbac-proxy"
Jul 27 21:45:40.980572 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:45:40.980520    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="280747d9-9219-4d4f-a4b0-b442260d38b4" containerName="kube-rbac-proxy"
Jul 27 21:45:40.980572 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:45:40.980528    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="280747d9-9219-4d4f-a4b0-b442260d38b4" containerName="storage-initializer"
Jul 27 21:45:40.980572 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:45:40.980534    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="280747d9-9219-4d4f-a4b0-b442260d38b4" containerName="storage-initializer"
Jul 27 21:45:40.980869 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:45:40.980577    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="280747d9-9219-4d4f-a4b0-b442260d38b4" containerName="kserve-container"
Jul 27 21:45:40.980869 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:45:40.980586    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="280747d9-9219-4d4f-a4b0-b442260d38b4" containerName="kube-rbac-proxy"
Jul 27 21:45:40.983667 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:45:40.983651    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-llwtf"
Jul 27 21:45:40.985987 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:45:40.985958    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
Jul 27 21:45:40.986134 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:45:40.986111    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 21:45:40.986222 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:45:40.986193    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 21:45:40.986222 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:45:40.986201    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 21:45:40.986344 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:45:40.986216    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 21:45:40.986808 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:45:40.986792    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 27 21:45:40.986893 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:45:40.986795    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v7c66\""
Jul 27 21:45:40.994151 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:45:40.994131    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-llwtf"]
Jul 27 21:45:41.007710 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:45:41.007687    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2766c17-f61e-4855-b810-6036c5167962-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-llwtf\" (UID: \"a2766c17-f61e-4855-b810-6036c5167962\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-llwtf"
Jul 27 21:45:41.007820 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:45:41.007724    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms9c4\" (UniqueName: \"kubernetes.io/projected/a2766c17-f61e-4855-b810-6036c5167962-kube-api-access-ms9c4\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-llwtf\" (UID: \"a2766c17-f61e-4855-b810-6036c5167962\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-llwtf"
Jul 27 21:45:41.007820 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:45:41.007757    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2766c17-f61e-4855-b810-6036c5167962-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-llwtf\" (UID: \"a2766c17-f61e-4855-b810-6036c5167962\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-llwtf"
Jul 27 21:45:41.007820 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:45:41.007813    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2766c17-f61e-4855-b810-6036c5167962-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-llwtf\" (UID: \"a2766c17-f61e-4855-b810-6036c5167962\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-llwtf"
Jul 27 21:45:41.007957 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:45:41.007834    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2766c17-f61e-4855-b810-6036c5167962-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-llwtf\" (UID: \"a2766c17-f61e-4855-b810-6036c5167962\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-llwtf"
Jul 27 21:45:41.109151 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:45:41.109121    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms9c4\" (UniqueName: \"kubernetes.io/projected/a2766c17-f61e-4855-b810-6036c5167962-kube-api-access-ms9c4\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-llwtf\" (UID: \"a2766c17-f61e-4855-b810-6036c5167962\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-llwtf"
Jul 27 21:45:41.109290 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:45:41.109172    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2766c17-f61e-4855-b810-6036c5167962-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-llwtf\" (UID: \"a2766c17-f61e-4855-b810-6036c5167962\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-llwtf"
Jul 27 21:45:41.109290 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:45:41.109213    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2766c17-f61e-4855-b810-6036c5167962-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-llwtf\" (UID: \"a2766c17-f61e-4855-b810-6036c5167962\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-llwtf"
Jul 27 21:45:41.109290 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:45:41.109244    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2766c17-f61e-4855-b810-6036c5167962-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-llwtf\" (UID: \"a2766c17-f61e-4855-b810-6036c5167962\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-llwtf"
Jul 27 21:45:41.109290 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:45:41.109285    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2766c17-f61e-4855-b810-6036c5167962-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-llwtf\" (UID: \"a2766c17-f61e-4855-b810-6036c5167962\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-llwtf"
Jul 27 21:45:41.109515 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:45:41.109400    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-serving-cert: secret "isvc-lightgbm-v2-runtime-predictor-serving-cert" not found
Jul 27 21:45:41.109515 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:45:41.109457    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2766c17-f61e-4855-b810-6036c5167962-proxy-tls podName:a2766c17-f61e-4855-b810-6036c5167962 nodeName:}" failed. No retries permitted until 2026-07-27 21:45:41.609436375 +0000 UTC m=+2464.976626232 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a2766c17-f61e-4855-b810-6036c5167962-proxy-tls") pod "isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-llwtf" (UID: "a2766c17-f61e-4855-b810-6036c5167962") : secret "isvc-lightgbm-v2-runtime-predictor-serving-cert" not found
Jul 27 21:45:41.109719 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:45:41.109697    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2766c17-f61e-4855-b810-6036c5167962-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-llwtf\" (UID: \"a2766c17-f61e-4855-b810-6036c5167962\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-llwtf"
Jul 27 21:45:41.109950 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:45:41.109934    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2766c17-f61e-4855-b810-6036c5167962-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-llwtf\" (UID: \"a2766c17-f61e-4855-b810-6036c5167962\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-llwtf"
Jul 27 21:45:41.110077 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:45:41.110058    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2766c17-f61e-4855-b810-6036c5167962-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-llwtf\" (UID: \"a2766c17-f61e-4855-b810-6036c5167962\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-llwtf"
Jul 27 21:45:41.117840 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:45:41.117819    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms9c4\" (UniqueName: \"kubernetes.io/projected/a2766c17-f61e-4855-b810-6036c5167962-kube-api-access-ms9c4\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-llwtf\" (UID: \"a2766c17-f61e-4855-b810-6036c5167962\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-llwtf"
Jul 27 21:45:41.304517 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:45:41.304432    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:45:41.614178 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:45:41.614098    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2766c17-f61e-4855-b810-6036c5167962-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-llwtf\" (UID: \"a2766c17-f61e-4855-b810-6036c5167962\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-llwtf"
Jul 27 21:45:41.616484 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:45:41.616463    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2766c17-f61e-4855-b810-6036c5167962-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-llwtf\" (UID: \"a2766c17-f61e-4855-b810-6036c5167962\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-llwtf"
Jul 27 21:45:41.895165 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:45:41.895081    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-llwtf"
Jul 27 21:45:42.021465 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:45:42.021439    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-llwtf"]
Jul 27 21:45:42.023642 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:45:42.023606    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2766c17_f61e_4855_b810_6036c5167962.slice/crio-c5f6d0af22ba3249ef9cdd771b7dd3525ac44a1d4b8792abee99160901b16360 WatchSource:0}: Error finding container c5f6d0af22ba3249ef9cdd771b7dd3525ac44a1d4b8792abee99160901b16360: Status 404 returned error can't find the container with id c5f6d0af22ba3249ef9cdd771b7dd3525ac44a1d4b8792abee99160901b16360
Jul 27 21:45:42.974781 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:45:42.974724    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-llwtf" event={"ID":"a2766c17-f61e-4855-b810-6036c5167962","Type":"ContainerStarted","Data":"ab989de4e5ba2cda756210485f7873306eba81d2e96c1ec80e57cbd066b2b905"}
Jul 27 21:45:42.974958 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:45:42.974808    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-llwtf" event={"ID":"a2766c17-f61e-4855-b810-6036c5167962","Type":"ContainerStarted","Data":"c5f6d0af22ba3249ef9cdd771b7dd3525ac44a1d4b8792abee99160901b16360"}
Jul 27 21:45:45.987684 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:45:45.987651    2577 generic.go:356] "Generic (PLEG): container finished" podID="a2766c17-f61e-4855-b810-6036c5167962" containerID="ab989de4e5ba2cda756210485f7873306eba81d2e96c1ec80e57cbd066b2b905" exitCode=0
Jul 27 21:45:45.988094 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:45:45.987696    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-llwtf" event={"ID":"a2766c17-f61e-4855-b810-6036c5167962","Type":"ContainerDied","Data":"ab989de4e5ba2cda756210485f7873306eba81d2e96c1ec80e57cbd066b2b905"}
Jul 27 21:45:45.988913 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:45:45.988898    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 21:45:57.415497 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:45:57.415456    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:47:59.582141 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:47:59.582114    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-llwtf" event={"ID":"a2766c17-f61e-4855-b810-6036c5167962","Type":"ContainerStarted","Data":"b3ca708a598440bb851374f2b96f645ade5690a4a50c77b1bdf01dc272f4fd91"}
Jul 27 21:48:00.587845 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:00.587811    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-llwtf" event={"ID":"a2766c17-f61e-4855-b810-6036c5167962","Type":"ContainerStarted","Data":"b37422be5f20781aeff76466dab8a5f595cca4d51170429112d91b7d90617e94"}
Jul 27 21:48:00.588336 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:00.587983    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-llwtf"
Jul 27 21:48:00.614601 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:00.614548    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-llwtf" podStartSLOduration=3.529549718 podStartE2EDuration="2m20.614531904s" totalImagesPullingTime="2m13.389091488s" totalInitContainerRuntime="3.695890698s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:45:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:48:00.613683882 +0000 UTC m=+2603.980873752" watchObservedRunningTime="2026-07-27 21:48:00.614531904 +0000 UTC m=+2603.981721769"
Jul 27 21:48:01.592167 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:01.592119    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-llwtf"
Jul 27 21:48:07.601812 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:07.601780    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-llwtf"
Jul 27 21:48:37.606057 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:37.606024    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-llwtf"
Jul 27 21:48:41.183397 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:41.183364    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-llwtf"]
Jul 27 21:48:41.183845 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:41.183592    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-llwtf" podUID="a2766c17-f61e-4855-b810-6036c5167962" containerName="kserve-container" containerID="cri-o://b3ca708a598440bb851374f2b96f645ade5690a4a50c77b1bdf01dc272f4fd91" gracePeriod=30
Jul 27 21:48:41.183845 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:41.183632    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-llwtf" podUID="a2766c17-f61e-4855-b810-6036c5167962" containerName="kube-rbac-proxy" containerID="cri-o://b37422be5f20781aeff76466dab8a5f595cca4d51170429112d91b7d90617e94" gracePeriod=30
Jul 27 21:48:41.238082 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:41.238054    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:48:41.753558 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:41.753521    2577 generic.go:356] "Generic (PLEG): container finished" podID="a2766c17-f61e-4855-b810-6036c5167962" containerID="b37422be5f20781aeff76466dab8a5f595cca4d51170429112d91b7d90617e94" exitCode=2
Jul 27 21:48:41.753742 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:41.753593    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-llwtf" event={"ID":"a2766c17-f61e-4855-b810-6036c5167962","Type":"ContainerDied","Data":"b37422be5f20781aeff76466dab8a5f595cca4d51170429112d91b7d90617e94"}
Jul 27 21:48:42.133868 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:42.133847    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-llwtf"
Jul 27 21:48:42.265219 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:42.265164    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ms9c4\" (UniqueName: \"kubernetes.io/projected/a2766c17-f61e-4855-b810-6036c5167962-kube-api-access-ms9c4\") pod \"a2766c17-f61e-4855-b810-6036c5167962\" (UID: \"a2766c17-f61e-4855-b810-6036c5167962\") "
Jul 27 21:48:42.265219 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:42.265221    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2766c17-f61e-4855-b810-6036c5167962-kserve-provision-location\") pod \"a2766c17-f61e-4855-b810-6036c5167962\" (UID: \"a2766c17-f61e-4855-b810-6036c5167962\") "
Jul 27 21:48:42.265719 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:42.265273    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2766c17-f61e-4855-b810-6036c5167962-kube-rbac-proxy-sar-config\") pod \"a2766c17-f61e-4855-b810-6036c5167962\" (UID: \"a2766c17-f61e-4855-b810-6036c5167962\") "
Jul 27 21:48:42.265719 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:42.265360    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2766c17-f61e-4855-b810-6036c5167962-proxy-tls\") pod \"a2766c17-f61e-4855-b810-6036c5167962\" (UID: \"a2766c17-f61e-4855-b810-6036c5167962\") "
Jul 27 21:48:42.265719 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:42.265385    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2766c17-f61e-4855-b810-6036c5167962-cabundle-cert\") pod \"a2766c17-f61e-4855-b810-6036c5167962\" (UID: \"a2766c17-f61e-4855-b810-6036c5167962\") "
Jul 27 21:48:42.265719 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:42.265586    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2766c17-f61e-4855-b810-6036c5167962-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a2766c17-f61e-4855-b810-6036c5167962" (UID: "a2766c17-f61e-4855-b810-6036c5167962"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:48:42.265719 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:42.265690    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2766c17-f61e-4855-b810-6036c5167962-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a2766c17-f61e-4855-b810-6036c5167962" (UID: "a2766c17-f61e-4855-b810-6036c5167962"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:48:42.265905 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:42.265799    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2766c17-f61e-4855-b810-6036c5167962-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a2766c17-f61e-4855-b810-6036c5167962" (UID: "a2766c17-f61e-4855-b810-6036c5167962"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:48:42.267277 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:42.267253    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2766c17-f61e-4855-b810-6036c5167962-kube-api-access-ms9c4" (OuterVolumeSpecName: "kube-api-access-ms9c4") pod "a2766c17-f61e-4855-b810-6036c5167962" (UID: "a2766c17-f61e-4855-b810-6036c5167962"). InnerVolumeSpecName "kube-api-access-ms9c4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:48:42.267384 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:42.267316    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2766c17-f61e-4855-b810-6036c5167962-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a2766c17-f61e-4855-b810-6036c5167962" (UID: "a2766c17-f61e-4855-b810-6036c5167962"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:48:42.366812 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:42.366748    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2766c17-f61e-4855-b810-6036c5167962-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:48:42.366812 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:42.366785    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2766c17-f61e-4855-b810-6036c5167962-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:48:42.366812 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:42.366796    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms9c4\" (UniqueName: \"kubernetes.io/projected/a2766c17-f61e-4855-b810-6036c5167962-kube-api-access-ms9c4\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:48:42.366812 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:42.366807    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2766c17-f61e-4855-b810-6036c5167962-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:48:42.366812 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:42.366815    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2766c17-f61e-4855-b810-6036c5167962-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:48:42.759253 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:42.759218    2577 generic.go:356] "Generic (PLEG): container finished" podID="a2766c17-f61e-4855-b810-6036c5167962" containerID="b3ca708a598440bb851374f2b96f645ade5690a4a50c77b1bdf01dc272f4fd91" exitCode=0
Jul 27 21:48:42.759446 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:42.759276    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-llwtf"
Jul 27 21:48:42.759446 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:42.759301    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-llwtf" event={"ID":"a2766c17-f61e-4855-b810-6036c5167962","Type":"ContainerDied","Data":"b3ca708a598440bb851374f2b96f645ade5690a4a50c77b1bdf01dc272f4fd91"}
Jul 27 21:48:42.759446 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:42.759362    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-llwtf" event={"ID":"a2766c17-f61e-4855-b810-6036c5167962","Type":"ContainerDied","Data":"c5f6d0af22ba3249ef9cdd771b7dd3525ac44a1d4b8792abee99160901b16360"}
Jul 27 21:48:42.759446 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:42.759403    2577 scope.go:117] "RemoveContainer" containerID="b37422be5f20781aeff76466dab8a5f595cca4d51170429112d91b7d90617e94"
Jul 27 21:48:42.768279 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:42.768264    2577 scope.go:117] "RemoveContainer" containerID="b3ca708a598440bb851374f2b96f645ade5690a4a50c77b1bdf01dc272f4fd91"
Jul 27 21:48:42.775617 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:42.775602    2577 scope.go:117] "RemoveContainer" containerID="ab989de4e5ba2cda756210485f7873306eba81d2e96c1ec80e57cbd066b2b905"
Jul 27 21:48:42.781774 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:42.781742    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-llwtf"]
Jul 27 21:48:42.782876 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:42.782862    2577 scope.go:117] "RemoveContainer" containerID="b37422be5f20781aeff76466dab8a5f595cca4d51170429112d91b7d90617e94"
Jul 27 21:48:42.783126 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:48:42.783101    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b37422be5f20781aeff76466dab8a5f595cca4d51170429112d91b7d90617e94\": container with ID starting with b37422be5f20781aeff76466dab8a5f595cca4d51170429112d91b7d90617e94 not found: ID does not exist" containerID="b37422be5f20781aeff76466dab8a5f595cca4d51170429112d91b7d90617e94"
Jul 27 21:48:42.783219 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:42.783132    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b37422be5f20781aeff76466dab8a5f595cca4d51170429112d91b7d90617e94"} err="failed to get container status \"b37422be5f20781aeff76466dab8a5f595cca4d51170429112d91b7d90617e94\": rpc error: code = NotFound desc = could not find container \"b37422be5f20781aeff76466dab8a5f595cca4d51170429112d91b7d90617e94\": container with ID starting with b37422be5f20781aeff76466dab8a5f595cca4d51170429112d91b7d90617e94 not found: ID does not exist"
Jul 27 21:48:42.783219 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:42.783154    2577 scope.go:117] "RemoveContainer" containerID="b3ca708a598440bb851374f2b96f645ade5690a4a50c77b1bdf01dc272f4fd91"
Jul 27 21:48:42.783444 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:48:42.783418    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3ca708a598440bb851374f2b96f645ade5690a4a50c77b1bdf01dc272f4fd91\": container with ID starting with b3ca708a598440bb851374f2b96f645ade5690a4a50c77b1bdf01dc272f4fd91 not found: ID does not exist" containerID="b3ca708a598440bb851374f2b96f645ade5690a4a50c77b1bdf01dc272f4fd91"
Jul 27 21:48:42.783519 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:42.783452    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3ca708a598440bb851374f2b96f645ade5690a4a50c77b1bdf01dc272f4fd91"} err="failed to get container status \"b3ca708a598440bb851374f2b96f645ade5690a4a50c77b1bdf01dc272f4fd91\": rpc error: code = NotFound desc = could not find container \"b3ca708a598440bb851374f2b96f645ade5690a4a50c77b1bdf01dc272f4fd91\": container with ID starting with b3ca708a598440bb851374f2b96f645ade5690a4a50c77b1bdf01dc272f4fd91 not found: ID does not exist"
Jul 27 21:48:42.783519 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:42.783474    2577 scope.go:117] "RemoveContainer" containerID="ab989de4e5ba2cda756210485f7873306eba81d2e96c1ec80e57cbd066b2b905"
Jul 27 21:48:42.783931 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:48:42.783891    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab989de4e5ba2cda756210485f7873306eba81d2e96c1ec80e57cbd066b2b905\": container with ID starting with ab989de4e5ba2cda756210485f7873306eba81d2e96c1ec80e57cbd066b2b905 not found: ID does not exist" containerID="ab989de4e5ba2cda756210485f7873306eba81d2e96c1ec80e57cbd066b2b905"
Jul 27 21:48:42.784007 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:42.783927    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab989de4e5ba2cda756210485f7873306eba81d2e96c1ec80e57cbd066b2b905"} err="failed to get container status \"ab989de4e5ba2cda756210485f7873306eba81d2e96c1ec80e57cbd066b2b905\": rpc error: code = NotFound desc = could not find container \"ab989de4e5ba2cda756210485f7873306eba81d2e96c1ec80e57cbd066b2b905\": container with ID starting with ab989de4e5ba2cda756210485f7873306eba81d2e96c1ec80e57cbd066b2b905 not found: ID does not exist"
Jul 27 21:48:42.785271 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:42.785253    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-llwtf"]
Jul 27 21:48:43.134685 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:43.134617    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2766c17-f61e-4855-b810-6036c5167962" path="/var/lib/kubelet/pods/a2766c17-f61e-4855-b810-6036c5167962/volumes"
Jul 27 21:48:46.265484 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:46.265444    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:48:46.323322 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:46.323293    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lm9mf"]
Jul 27 21:48:46.323587 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:46.323575    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2766c17-f61e-4855-b810-6036c5167962" containerName="kserve-container"
Jul 27 21:48:46.323630 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:46.323588    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2766c17-f61e-4855-b810-6036c5167962" containerName="kserve-container"
Jul 27 21:48:46.323630 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:46.323599    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2766c17-f61e-4855-b810-6036c5167962" containerName="storage-initializer"
Jul 27 21:48:46.323630 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:46.323604    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2766c17-f61e-4855-b810-6036c5167962" containerName="storage-initializer"
Jul 27 21:48:46.323630 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:46.323622    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2766c17-f61e-4855-b810-6036c5167962" containerName="kube-rbac-proxy"
Jul 27 21:48:46.323630 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:46.323628    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2766c17-f61e-4855-b810-6036c5167962" containerName="kube-rbac-proxy"
Jul 27 21:48:46.323794 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:46.323667    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2766c17-f61e-4855-b810-6036c5167962" containerName="kube-rbac-proxy"
Jul 27 21:48:46.323794 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:46.323678    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2766c17-f61e-4855-b810-6036c5167962" containerName="kserve-container"
Jul 27 21:48:46.325530 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:46.325515    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lm9mf"
Jul 27 21:48:46.327895 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:46.327862    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
Jul 27 21:48:46.327895 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:46.327886    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
Jul 27 21:48:46.328078 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:46.327918    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v7c66\""
Jul 27 21:48:46.328078 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:46.327868    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 21:48:46.328078 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:46.327865    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 21:48:46.328561 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:46.328544    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 21:48:46.328561 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:46.328553    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 21:48:46.339710 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:46.339687    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lm9mf"]
Jul 27 21:48:46.497137 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:46.497105    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lm9mf\" (UID: \"3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lm9mf"
Jul 27 21:48:46.497137 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:46.497138    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzl4q\" (UniqueName: \"kubernetes.io/projected/3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c-kube-api-access-tzl4q\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lm9mf\" (UID: \"3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lm9mf"
Jul 27 21:48:46.497339 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:46.497176    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lm9mf\" (UID: \"3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lm9mf"
Jul 27 21:48:46.497339 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:46.497265    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lm9mf\" (UID: \"3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lm9mf"
Jul 27 21:48:46.497339 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:46.497295    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lm9mf\" (UID: \"3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lm9mf"
Jul 27 21:48:46.548485 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:46.548393    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:48:46.598661 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:46.598638    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lm9mf\" (UID: \"3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lm9mf"
Jul 27 21:48:46.598821 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:46.598667    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzl4q\" (UniqueName: \"kubernetes.io/projected/3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c-kube-api-access-tzl4q\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lm9mf\" (UID: \"3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lm9mf"
Jul 27 21:48:46.598821 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:46.598707    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lm9mf\" (UID: \"3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lm9mf"
Jul 27 21:48:46.598821 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:46.598757    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lm9mf\" (UID: \"3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lm9mf"
Jul 27 21:48:46.598821 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:46.598818    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lm9mf\" (UID: \"3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lm9mf"
Jul 27 21:48:46.599181 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:46.599158    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lm9mf\" (UID: \"3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lm9mf"
Jul 27 21:48:46.599721 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:46.599699    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lm9mf\" (UID: \"3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lm9mf"
Jul 27 21:48:46.599792 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:46.599707    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lm9mf\" (UID: \"3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lm9mf"
Jul 27 21:48:46.601095 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:46.601074    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lm9mf\" (UID: \"3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lm9mf"
Jul 27 21:48:46.606297 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:46.606276    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzl4q\" (UniqueName: \"kubernetes.io/projected/3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c-kube-api-access-tzl4q\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lm9mf\" (UID: \"3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lm9mf"
Jul 27 21:48:46.635179 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:46.635160    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lm9mf"
Jul 27 21:48:46.807181 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:46.807110    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lm9mf"]
Jul 27 21:48:46.809871 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:48:46.809843    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a35146e_c5e1_4fbb_a9bc_6bb87fc4c24c.slice/crio-5816d78e8298a3279ee780230e14ed082847a03e140b71bf924cb899e77f6be5 WatchSource:0}: Error finding container 5816d78e8298a3279ee780230e14ed082847a03e140b71bf924cb899e77f6be5: Status 404 returned error can't find the container with id 5816d78e8298a3279ee780230e14ed082847a03e140b71bf924cb899e77f6be5
Jul 27 21:48:47.783898 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:47.783868    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lm9mf" event={"ID":"3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c","Type":"ContainerStarted","Data":"141e3a04b851e20dc8e8b9fc7fd06358de16ea9555a2f960c53cc50e9000a87f"}
Jul 27 21:48:47.784276 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:47.783926    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lm9mf" event={"ID":"3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c","Type":"ContainerStarted","Data":"5816d78e8298a3279ee780230e14ed082847a03e140b71bf924cb899e77f6be5"}
Jul 27 21:48:50.796690 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:50.796655    2577 generic.go:356] "Generic (PLEG): container finished" podID="3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c" containerID="141e3a04b851e20dc8e8b9fc7fd06358de16ea9555a2f960c53cc50e9000a87f" exitCode=0
Jul 27 21:48:50.797145 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:50.796703    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lm9mf" event={"ID":"3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c","Type":"ContainerDied","Data":"141e3a04b851e20dc8e8b9fc7fd06358de16ea9555a2f960c53cc50e9000a87f"}
Jul 27 21:48:56.831962 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:56.831925    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lm9mf" event={"ID":"3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c","Type":"ContainerStarted","Data":"41fa042704cff03f6c9a066b64e5818471f48442d076bf7a6581e3f93fb63f1c"}
Jul 27 21:48:56.832342 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:56.831993    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lm9mf" event={"ID":"3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c","Type":"ContainerStarted","Data":"1e08511f09822263a16ee6ea34a5ed2298484d0b51dce4bca043eef05d9d0037"}
Jul 27 21:48:56.832342 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:56.832222    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lm9mf"
Jul 27 21:48:56.852183 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:56.852131    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lm9mf" podStartSLOduration=1.113657105 podStartE2EDuration="10.852114443s" totalImagesPullingTime="5.905734448s" totalInitContainerRuntime="3.83272289s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:48:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:48:56.851028879 +0000 UTC m=+2660.218218742" watchObservedRunningTime="2026-07-27 21:48:56.852114443 +0000 UTC m=+2660.219304309"
Jul 27 21:48:57.836594 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:57.836562    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lm9mf"
Jul 27 21:48:57.837702 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:57.837668    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lm9mf" podUID="3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 27 21:48:58.840472 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:48:58.840426    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lm9mf" podUID="3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 27 21:49:03.844750 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:03.844720    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lm9mf"
Jul 27 21:49:03.845951 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:03.845925    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lm9mf"
Jul 27 21:49:06.351000 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:06.350968    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lm9mf"]
Jul 27 21:49:06.351398 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:06.351205    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lm9mf" podUID="3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c" containerName="kserve-container" containerID="cri-o://1e08511f09822263a16ee6ea34a5ed2298484d0b51dce4bca043eef05d9d0037" gracePeriod=30
Jul 27 21:49:06.351398 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:06.351237    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lm9mf" podUID="3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c" containerName="kube-rbac-proxy" containerID="cri-o://41fa042704cff03f6c9a066b64e5818471f48442d076bf7a6581e3f93fb63f1c" gracePeriod=30
Jul 27 21:49:06.370364 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:06.370341    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:49:06.875652 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:06.875610    2577 generic.go:356] "Generic (PLEG): container finished" podID="3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c" containerID="41fa042704cff03f6c9a066b64e5818471f48442d076bf7a6581e3f93fb63f1c" exitCode=2
Jul 27 21:49:06.875782 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:06.875653    2577 generic.go:356] "Generic (PLEG): container finished" podID="3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c" containerID="1e08511f09822263a16ee6ea34a5ed2298484d0b51dce4bca043eef05d9d0037" exitCode=0
Jul 27 21:49:06.875782 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:06.875670    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lm9mf" event={"ID":"3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c","Type":"ContainerDied","Data":"41fa042704cff03f6c9a066b64e5818471f48442d076bf7a6581e3f93fb63f1c"}
Jul 27 21:49:06.875782 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:06.875724    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lm9mf" event={"ID":"3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c","Type":"ContainerDied","Data":"1e08511f09822263a16ee6ea34a5ed2298484d0b51dce4bca043eef05d9d0037"}
Jul 27 21:49:06.993786 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:06.993748    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lm9mf"
Jul 27 21:49:07.059083 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:07.059059    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c-cabundle-cert\") pod \"3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c\" (UID: \"3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c\") "
Jul 27 21:49:07.059214 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:07.059110    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzl4q\" (UniqueName: \"kubernetes.io/projected/3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c-kube-api-access-tzl4q\") pod \"3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c\" (UID: \"3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c\") "
Jul 27 21:49:07.059214 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:07.059136    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c-kube-rbac-proxy-sar-config\") pod \"3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c\" (UID: \"3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c\") "
Jul 27 21:49:07.059214 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:07.059168    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c-kserve-provision-location\") pod \"3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c\" (UID: \"3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c\") "
Jul 27 21:49:07.059214 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:07.059199    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c-proxy-tls\") pod \"3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c\" (UID: \"3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c\") "
Jul 27 21:49:07.059439 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:07.059411    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c" (UID: "3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:49:07.059520 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:07.059495    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c" (UID: "3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:49:07.059564 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:07.059506    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c" (UID: "3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:49:07.061150 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:07.061125    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c" (UID: "3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:49:07.061212 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:07.061170    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c-kube-api-access-tzl4q" (OuterVolumeSpecName: "kube-api-access-tzl4q") pod "3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c" (UID: "3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c"). InnerVolumeSpecName "kube-api-access-tzl4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:49:07.160522 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:07.160499    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:49:07.160522 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:07.160518    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:49:07.160653 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:07.160527    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzl4q\" (UniqueName: \"kubernetes.io/projected/3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c-kube-api-access-tzl4q\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:49:07.160653 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:07.160536    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:49:07.160653 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:07.160545    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:49:07.880858 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:07.880830    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lm9mf"
Jul 27 21:49:07.881225 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:07.880830    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lm9mf" event={"ID":"3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c","Type":"ContainerDied","Data":"5816d78e8298a3279ee780230e14ed082847a03e140b71bf924cb899e77f6be5"}
Jul 27 21:49:07.881225 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:07.880979    2577 scope.go:117] "RemoveContainer" containerID="41fa042704cff03f6c9a066b64e5818471f48442d076bf7a6581e3f93fb63f1c"
Jul 27 21:49:07.888712 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:07.888694    2577 scope.go:117] "RemoveContainer" containerID="1e08511f09822263a16ee6ea34a5ed2298484d0b51dce4bca043eef05d9d0037"
Jul 27 21:49:07.895496 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:07.895478    2577 scope.go:117] "RemoveContainer" containerID="141e3a04b851e20dc8e8b9fc7fd06358de16ea9555a2f960c53cc50e9000a87f"
Jul 27 21:49:07.901267 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:07.901249    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lm9mf"]
Jul 27 21:49:07.904980 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:07.904959    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lm9mf"]
Jul 27 21:49:09.135278 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:09.135248    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c" path="/var/lib/kubelet/pods/3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c/volumes"
Jul 27 21:49:11.494155 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:11.494123    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:49:11.521373 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:11.521345    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-7m74r"]
Jul 27 21:49:11.521675 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:11.521643    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c" containerName="kube-rbac-proxy"
Jul 27 21:49:11.521719 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:11.521679    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c" containerName="kube-rbac-proxy"
Jul 27 21:49:11.521719 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:11.521689    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c" containerName="storage-initializer"
Jul 27 21:49:11.521719 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:11.521695    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c" containerName="storage-initializer"
Jul 27 21:49:11.521719 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:11.521701    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c" containerName="kserve-container"
Jul 27 21:49:11.521719 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:11.521707    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c" containerName="kserve-container"
Jul 27 21:49:11.521902 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:11.521751    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c" containerName="kserve-container"
Jul 27 21:49:11.521902 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:11.521773    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a35146e-c5e1-4fbb-a9bc-6bb87fc4c24c" containerName="kube-rbac-proxy"
Jul 27 21:49:11.525182 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:11.525165    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-7m74r"
Jul 27 21:49:11.527664 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:11.527629    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 21:49:11.527664 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:11.527646    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
Jul 27 21:49:11.527860 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:11.527679    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 21:49:11.527860 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:11.527696    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v7c66\""
Jul 27 21:49:11.527860 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:11.527721    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 21:49:11.528457 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:11.528439    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 21:49:11.528516 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:11.528483    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 27 21:49:11.535962 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:11.535940    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-7m74r"]
Jul 27 21:49:11.592467 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:11.592439    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80ea68d6-f3c3-4e77-9ef9-59d9098131c5-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-7m74r\" (UID: \"80ea68d6-f3c3-4e77-9ef9-59d9098131c5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-7m74r"
Jul 27 21:49:11.592467 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:11.592465    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80ea68d6-f3c3-4e77-9ef9-59d9098131c5-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-7m74r\" (UID: \"80ea68d6-f3c3-4e77-9ef9-59d9098131c5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-7m74r"
Jul 27 21:49:11.592640 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:11.592506    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/80ea68d6-f3c3-4e77-9ef9-59d9098131c5-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-7m74r\" (UID: \"80ea68d6-f3c3-4e77-9ef9-59d9098131c5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-7m74r"
Jul 27 21:49:11.592640 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:11.592525    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80ea68d6-f3c3-4e77-9ef9-59d9098131c5-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-7m74r\" (UID: \"80ea68d6-f3c3-4e77-9ef9-59d9098131c5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-7m74r"
Jul 27 21:49:11.592640 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:11.592571    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfvx7\" (UniqueName: \"kubernetes.io/projected/80ea68d6-f3c3-4e77-9ef9-59d9098131c5-kube-api-access-hfvx7\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-7m74r\" (UID: \"80ea68d6-f3c3-4e77-9ef9-59d9098131c5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-7m74r"
Jul 27 21:49:11.643429 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:11.643401    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:49:11.692973 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:11.692945    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfvx7\" (UniqueName: \"kubernetes.io/projected/80ea68d6-f3c3-4e77-9ef9-59d9098131c5-kube-api-access-hfvx7\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-7m74r\" (UID: \"80ea68d6-f3c3-4e77-9ef9-59d9098131c5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-7m74r"
Jul 27 21:49:11.693101 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:11.693012    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80ea68d6-f3c3-4e77-9ef9-59d9098131c5-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-7m74r\" (UID: \"80ea68d6-f3c3-4e77-9ef9-59d9098131c5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-7m74r"
Jul 27 21:49:11.693101 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:11.693036    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80ea68d6-f3c3-4e77-9ef9-59d9098131c5-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-7m74r\" (UID: \"80ea68d6-f3c3-4e77-9ef9-59d9098131c5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-7m74r"
Jul 27 21:49:11.693101 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:11.693080    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/80ea68d6-f3c3-4e77-9ef9-59d9098131c5-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-7m74r\" (UID: \"80ea68d6-f3c3-4e77-9ef9-59d9098131c5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-7m74r"
Jul 27 21:49:11.693266 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:11.693102    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80ea68d6-f3c3-4e77-9ef9-59d9098131c5-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-7m74r\" (UID: \"80ea68d6-f3c3-4e77-9ef9-59d9098131c5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-7m74r"
Jul 27 21:49:11.693528 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:11.693505    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80ea68d6-f3c3-4e77-9ef9-59d9098131c5-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-7m74r\" (UID: \"80ea68d6-f3c3-4e77-9ef9-59d9098131c5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-7m74r"
Jul 27 21:49:11.693803 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:11.693785    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80ea68d6-f3c3-4e77-9ef9-59d9098131c5-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-7m74r\" (UID: \"80ea68d6-f3c3-4e77-9ef9-59d9098131c5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-7m74r"
Jul 27 21:49:11.693886 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:11.693815    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/80ea68d6-f3c3-4e77-9ef9-59d9098131c5-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-7m74r\" (UID: \"80ea68d6-f3c3-4e77-9ef9-59d9098131c5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-7m74r"
Jul 27 21:49:11.695611 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:11.695595    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80ea68d6-f3c3-4e77-9ef9-59d9098131c5-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-7m74r\" (UID: \"80ea68d6-f3c3-4e77-9ef9-59d9098131c5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-7m74r"
Jul 27 21:49:11.701500 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:11.701476    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfvx7\" (UniqueName: \"kubernetes.io/projected/80ea68d6-f3c3-4e77-9ef9-59d9098131c5-kube-api-access-hfvx7\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-7m74r\" (UID: \"80ea68d6-f3c3-4e77-9ef9-59d9098131c5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-7m74r"
Jul 27 21:49:11.836149 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:11.836068    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-7m74r"
Jul 27 21:49:12.001379 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:12.001356    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-7m74r"]
Jul 27 21:49:12.003556 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:49:12.003527    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80ea68d6_f3c3_4e77_9ef9_59d9098131c5.slice/crio-dd7248d076b4b5b3e1a98485b6eefb71149a98eeb78476a9c4bd965364fc0f6c WatchSource:0}: Error finding container dd7248d076b4b5b3e1a98485b6eefb71149a98eeb78476a9c4bd965364fc0f6c: Status 404 returned error can't find the container with id dd7248d076b4b5b3e1a98485b6eefb71149a98eeb78476a9c4bd965364fc0f6c
Jul 27 21:49:12.901304 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:12.901267    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-7m74r" event={"ID":"80ea68d6-f3c3-4e77-9ef9-59d9098131c5","Type":"ContainerStarted","Data":"eb9d61a9dd8c054e72058f2405399df11b4be7394b7c738f9436006de752aa06"}
Jul 27 21:49:12.901723 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:12.901336    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-7m74r" event={"ID":"80ea68d6-f3c3-4e77-9ef9-59d9098131c5","Type":"ContainerStarted","Data":"dd7248d076b4b5b3e1a98485b6eefb71149a98eeb78476a9c4bd965364fc0f6c"}
Jul 27 21:49:16.917985 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:16.917950    2577 generic.go:356] "Generic (PLEG): container finished" podID="80ea68d6-f3c3-4e77-9ef9-59d9098131c5" containerID="eb9d61a9dd8c054e72058f2405399df11b4be7394b7c738f9436006de752aa06" exitCode=0
Jul 27 21:49:16.918421 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:16.918001    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-7m74r" event={"ID":"80ea68d6-f3c3-4e77-9ef9-59d9098131c5","Type":"ContainerDied","Data":"eb9d61a9dd8c054e72058f2405399df11b4be7394b7c738f9436006de752aa06"}
Jul 27 21:49:17.923971 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:17.923935    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-7m74r" event={"ID":"80ea68d6-f3c3-4e77-9ef9-59d9098131c5","Type":"ContainerStarted","Data":"ec9c23f442084415a788ca8e1de28471779b39ac8b80412ee1f2317feee90ce5"}
Jul 27 21:49:17.924389 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:17.923997    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-7m74r" event={"ID":"80ea68d6-f3c3-4e77-9ef9-59d9098131c5","Type":"ContainerStarted","Data":"f0f68cfdee40fbe04ab7bfcce0728d7908ebbf4722702716a05fbafa95e15ed6"}
Jul 27 21:49:17.924389 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:17.924203    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-7m74r"
Jul 27 21:49:17.924389 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:17.924262    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-7m74r"
Jul 27 21:49:17.943299 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:17.943252    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-7m74r" podStartSLOduration=3.032391577 podStartE2EDuration="6.943237922s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.910846345s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:49:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:49:17.942389412 +0000 UTC m=+2681.309579315" watchObservedRunningTime="2026-07-27 21:49:17.943237922 +0000 UTC m=+2681.310427789"
Jul 27 21:49:23.934727 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:23.934698    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-7m74r"
Jul 27 21:49:39.885916 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:39.885871    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:49:53.938306 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:49:53.938275    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-7m74r"
Jul 27 21:50:01.610097 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:50:01.610058    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-7m74r"]
Jul 27 21:50:01.610485 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:50:01.610374    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-7m74r" podUID="80ea68d6-f3c3-4e77-9ef9-59d9098131c5" containerName="kserve-container" containerID="cri-o://f0f68cfdee40fbe04ab7bfcce0728d7908ebbf4722702716a05fbafa95e15ed6" gracePeriod=30
Jul 27 21:50:01.610485 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:50:01.610440    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-7m74r" podUID="80ea68d6-f3c3-4e77-9ef9-59d9098131c5" containerName="kube-rbac-proxy" containerID="cri-o://ec9c23f442084415a788ca8e1de28471779b39ac8b80412ee1f2317feee90ce5" gracePeriod=30
Jul 27 21:50:01.769666 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:50:01.769637    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:50:02.100807 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:50:02.100751    2577 generic.go:356] "Generic (PLEG): container finished" podID="80ea68d6-f3c3-4e77-9ef9-59d9098131c5" containerID="ec9c23f442084415a788ca8e1de28471779b39ac8b80412ee1f2317feee90ce5" exitCode=2
Jul 27 21:50:02.100967 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:50:02.100826    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-7m74r" event={"ID":"80ea68d6-f3c3-4e77-9ef9-59d9098131c5","Type":"ContainerDied","Data":"ec9c23f442084415a788ca8e1de28471779b39ac8b80412ee1f2317feee90ce5"}
Jul 27 21:50:02.750559 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:50:02.750535    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-7m74r"
Jul 27 21:50:02.800916 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:50:02.800858    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80ea68d6-f3c3-4e77-9ef9-59d9098131c5-kube-rbac-proxy-sar-config\") pod \"80ea68d6-f3c3-4e77-9ef9-59d9098131c5\" (UID: \"80ea68d6-f3c3-4e77-9ef9-59d9098131c5\") "
Jul 27 21:50:02.800916 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:50:02.800905    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hfvx7\" (UniqueName: \"kubernetes.io/projected/80ea68d6-f3c3-4e77-9ef9-59d9098131c5-kube-api-access-hfvx7\") pod \"80ea68d6-f3c3-4e77-9ef9-59d9098131c5\" (UID: \"80ea68d6-f3c3-4e77-9ef9-59d9098131c5\") "
Jul 27 21:50:02.801095 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:50:02.800924    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80ea68d6-f3c3-4e77-9ef9-59d9098131c5-kserve-provision-location\") pod \"80ea68d6-f3c3-4e77-9ef9-59d9098131c5\" (UID: \"80ea68d6-f3c3-4e77-9ef9-59d9098131c5\") "
Jul 27 21:50:02.801095 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:50:02.800991    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80ea68d6-f3c3-4e77-9ef9-59d9098131c5-proxy-tls\") pod \"80ea68d6-f3c3-4e77-9ef9-59d9098131c5\" (UID: \"80ea68d6-f3c3-4e77-9ef9-59d9098131c5\") "
Jul 27 21:50:02.801247 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:50:02.801226    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80ea68d6-f3c3-4e77-9ef9-59d9098131c5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "80ea68d6-f3c3-4e77-9ef9-59d9098131c5" (UID: "80ea68d6-f3c3-4e77-9ef9-59d9098131c5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:50:02.801314 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:50:02.801244    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80ea68d6-f3c3-4e77-9ef9-59d9098131c5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "80ea68d6-f3c3-4e77-9ef9-59d9098131c5" (UID: "80ea68d6-f3c3-4e77-9ef9-59d9098131c5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:50:02.802868 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:50:02.802847    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80ea68d6-f3c3-4e77-9ef9-59d9098131c5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "80ea68d6-f3c3-4e77-9ef9-59d9098131c5" (UID: "80ea68d6-f3c3-4e77-9ef9-59d9098131c5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:50:02.802987 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:50:02.802971    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80ea68d6-f3c3-4e77-9ef9-59d9098131c5-kube-api-access-hfvx7" (OuterVolumeSpecName: "kube-api-access-hfvx7") pod "80ea68d6-f3c3-4e77-9ef9-59d9098131c5" (UID: "80ea68d6-f3c3-4e77-9ef9-59d9098131c5"). InnerVolumeSpecName "kube-api-access-hfvx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:50:02.901542 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:50:02.901518    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/80ea68d6-f3c3-4e77-9ef9-59d9098131c5-cabundle-cert\") pod \"80ea68d6-f3c3-4e77-9ef9-59d9098131c5\" (UID: \"80ea68d6-f3c3-4e77-9ef9-59d9098131c5\") "
Jul 27 21:50:02.901731 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:50:02.901716    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfvx7\" (UniqueName: \"kubernetes.io/projected/80ea68d6-f3c3-4e77-9ef9-59d9098131c5-kube-api-access-hfvx7\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:50:02.901833 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:50:02.901737    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80ea68d6-f3c3-4e77-9ef9-59d9098131c5-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:50:02.901833 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:50:02.901752    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80ea68d6-f3c3-4e77-9ef9-59d9098131c5-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:50:02.901833 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:50:02.901790    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80ea68d6-f3c3-4e77-9ef9-59d9098131c5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:50:02.901937 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:50:02.901825    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80ea68d6-f3c3-4e77-9ef9-59d9098131c5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "80ea68d6-f3c3-4e77-9ef9-59d9098131c5" (UID: "80ea68d6-f3c3-4e77-9ef9-59d9098131c5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:50:03.002911 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:50:03.002888    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/80ea68d6-f3c3-4e77-9ef9-59d9098131c5-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:50:03.106661 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:50:03.106602    2577 generic.go:356] "Generic (PLEG): container finished" podID="80ea68d6-f3c3-4e77-9ef9-59d9098131c5" containerID="f0f68cfdee40fbe04ab7bfcce0728d7908ebbf4722702716a05fbafa95e15ed6" exitCode=0
Jul 27 21:50:03.106739 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:50:03.106660    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-7m74r" event={"ID":"80ea68d6-f3c3-4e77-9ef9-59d9098131c5","Type":"ContainerDied","Data":"f0f68cfdee40fbe04ab7bfcce0728d7908ebbf4722702716a05fbafa95e15ed6"}
Jul 27 21:50:03.106739 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:50:03.106666    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-7m74r"
Jul 27 21:50:03.106739 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:50:03.106717    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-7m74r" event={"ID":"80ea68d6-f3c3-4e77-9ef9-59d9098131c5","Type":"ContainerDied","Data":"dd7248d076b4b5b3e1a98485b6eefb71149a98eeb78476a9c4bd965364fc0f6c"}
Jul 27 21:50:03.106853 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:50:03.106755    2577 scope.go:117] "RemoveContainer" containerID="ec9c23f442084415a788ca8e1de28471779b39ac8b80412ee1f2317feee90ce5"
Jul 27 21:50:03.114831 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:50:03.114812    2577 scope.go:117] "RemoveContainer" containerID="f0f68cfdee40fbe04ab7bfcce0728d7908ebbf4722702716a05fbafa95e15ed6"
Jul 27 21:50:03.122294 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:50:03.122276    2577 scope.go:117] "RemoveContainer" containerID="eb9d61a9dd8c054e72058f2405399df11b4be7394b7c738f9436006de752aa06"
Jul 27 21:50:03.128052 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:50:03.128032    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-7m74r"]
Jul 27 21:50:03.129209 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:50:03.129194    2577 scope.go:117] "RemoveContainer" containerID="ec9c23f442084415a788ca8e1de28471779b39ac8b80412ee1f2317feee90ce5"
Jul 27 21:50:03.129410 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:50:03.129393    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec9c23f442084415a788ca8e1de28471779b39ac8b80412ee1f2317feee90ce5\": container with ID starting with ec9c23f442084415a788ca8e1de28471779b39ac8b80412ee1f2317feee90ce5 not found: ID does not exist" containerID="ec9c23f442084415a788ca8e1de28471779b39ac8b80412ee1f2317feee90ce5"
Jul 27 21:50:03.129495 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:50:03.129419    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec9c23f442084415a788ca8e1de28471779b39ac8b80412ee1f2317feee90ce5"} err="failed to get container status \"ec9c23f442084415a788ca8e1de28471779b39ac8b80412ee1f2317feee90ce5\": rpc error: code = NotFound desc = could not find container \"ec9c23f442084415a788ca8e1de28471779b39ac8b80412ee1f2317feee90ce5\": container with ID starting with ec9c23f442084415a788ca8e1de28471779b39ac8b80412ee1f2317feee90ce5 not found: ID does not exist"
Jul 27 21:50:03.129495 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:50:03.129441    2577 scope.go:117] "RemoveContainer" containerID="f0f68cfdee40fbe04ab7bfcce0728d7908ebbf4722702716a05fbafa95e15ed6"
Jul 27 21:50:03.129689 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:50:03.129670    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0f68cfdee40fbe04ab7bfcce0728d7908ebbf4722702716a05fbafa95e15ed6\": container with ID starting with f0f68cfdee40fbe04ab7bfcce0728d7908ebbf4722702716a05fbafa95e15ed6 not found: ID does not exist" containerID="f0f68cfdee40fbe04ab7bfcce0728d7908ebbf4722702716a05fbafa95e15ed6"
Jul 27 21:50:03.129733 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:50:03.129694    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0f68cfdee40fbe04ab7bfcce0728d7908ebbf4722702716a05fbafa95e15ed6"} err="failed to get container status \"f0f68cfdee40fbe04ab7bfcce0728d7908ebbf4722702716a05fbafa95e15ed6\": rpc error: code = NotFound desc = could not find container \"f0f68cfdee40fbe04ab7bfcce0728d7908ebbf4722702716a05fbafa95e15ed6\": container with ID starting with f0f68cfdee40fbe04ab7bfcce0728d7908ebbf4722702716a05fbafa95e15ed6 not found: ID does not exist"
Jul 27 21:50:03.129733 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:50:03.129712    2577 scope.go:117] "RemoveContainer" containerID="eb9d61a9dd8c054e72058f2405399df11b4be7394b7c738f9436006de752aa06"
Jul 27 21:50:03.129930 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:50:03.129912    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb9d61a9dd8c054e72058f2405399df11b4be7394b7c738f9436006de752aa06\": container with ID starting with eb9d61a9dd8c054e72058f2405399df11b4be7394b7c738f9436006de752aa06 not found: ID does not exist" containerID="eb9d61a9dd8c054e72058f2405399df11b4be7394b7c738f9436006de752aa06"
Jul 27 21:50:03.129996 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:50:03.129935    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb9d61a9dd8c054e72058f2405399df11b4be7394b7c738f9436006de752aa06"} err="failed to get container status \"eb9d61a9dd8c054e72058f2405399df11b4be7394b7c738f9436006de752aa06\": rpc error: code = NotFound desc = could not find container \"eb9d61a9dd8c054e72058f2405399df11b4be7394b7c738f9436006de752aa06\": container with ID starting with eb9d61a9dd8c054e72058f2405399df11b4be7394b7c738f9436006de752aa06 not found: ID does not exist"
Jul 27 21:50:03.135035 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:50:03.135018    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-7m74r"]
Jul 27 21:50:05.134495 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:50:05.134460    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80ea68d6-f3c3-4e77-9ef9-59d9098131c5" path="/var/lib/kubelet/pods/80ea68d6-f3c3-4e77-9ef9-59d9098131c5/volumes"
Jul 27 21:50:06.753928 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:50:06.753890    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:50:07.005914 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:50:07.005809    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:51:26.880514 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:51:26.880438    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:51:31.975066 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:51:31.975010    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:51:32.009389 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:51:32.009356    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q6v2s"]
Jul 27 21:51:32.009803 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:51:32.009788    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80ea68d6-f3c3-4e77-9ef9-59d9098131c5" containerName="storage-initializer"
Jul 27 21:51:32.009855 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:51:32.009806    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="80ea68d6-f3c3-4e77-9ef9-59d9098131c5" containerName="storage-initializer"
Jul 27 21:51:32.009855 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:51:32.009826    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80ea68d6-f3c3-4e77-9ef9-59d9098131c5" containerName="kube-rbac-proxy"
Jul 27 21:51:32.009855 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:51:32.009836    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="80ea68d6-f3c3-4e77-9ef9-59d9098131c5" containerName="kube-rbac-proxy"
Jul 27 21:51:32.009855 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:51:32.009853    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80ea68d6-f3c3-4e77-9ef9-59d9098131c5" containerName="kserve-container"
Jul 27 21:51:32.009977 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:51:32.009862    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="80ea68d6-f3c3-4e77-9ef9-59d9098131c5" containerName="kserve-container"
Jul 27 21:51:32.009977 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:51:32.009931    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="80ea68d6-f3c3-4e77-9ef9-59d9098131c5" containerName="kserve-container"
Jul 27 21:51:32.009977 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:51:32.009946    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="80ea68d6-f3c3-4e77-9ef9-59d9098131c5" containerName="kube-rbac-proxy"
Jul 27 21:51:32.013284 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:51:32.013265    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q6v2s"
Jul 27 21:51:32.015700 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:51:32.015660    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 21:51:32.015700 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:51:32.015691    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 21:51:32.015908 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:51:32.015721    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
Jul 27 21:51:32.015908 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:51:32.015724    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 21:51:32.015908 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:51:32.015691    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\""
Jul 27 21:51:32.015908 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:51:32.015814    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v7c66\""
Jul 27 21:51:32.015908 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:51:32.015823    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 21:51:32.022535 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:51:32.022515    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q6v2s"]
Jul 27 21:51:32.060993 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:51:32.060971    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a099162-9e26-49c0-9e37-0a6338a58caa-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-q6v2s\" (UID: \"4a099162-9e26-49c0-9e37-0a6338a58caa\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q6v2s"
Jul 27 21:51:32.061096 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:51:32.061005    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhvbk\" (UniqueName: \"kubernetes.io/projected/4a099162-9e26-49c0-9e37-0a6338a58caa-kube-api-access-dhvbk\") pod \"isvc-paddle-predictor-6d5c9fd4f6-q6v2s\" (UID: \"4a099162-9e26-49c0-9e37-0a6338a58caa\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q6v2s"
Jul 27 21:51:32.061096 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:51:32.061068    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a099162-9e26-49c0-9e37-0a6338a58caa-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-q6v2s\" (UID: \"4a099162-9e26-49c0-9e37-0a6338a58caa\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q6v2s"
Jul 27 21:51:32.061169 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:51:32.061114    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a099162-9e26-49c0-9e37-0a6338a58caa-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-q6v2s\" (UID: \"4a099162-9e26-49c0-9e37-0a6338a58caa\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q6v2s"
Jul 27 21:51:32.061169 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:51:32.061135    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a099162-9e26-49c0-9e37-0a6338a58caa-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-q6v2s\" (UID: \"4a099162-9e26-49c0-9e37-0a6338a58caa\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q6v2s"
Jul 27 21:51:32.162408 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:51:32.162359    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a099162-9e26-49c0-9e37-0a6338a58caa-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-q6v2s\" (UID: \"4a099162-9e26-49c0-9e37-0a6338a58caa\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q6v2s"
Jul 27 21:51:32.162408 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:51:32.162415    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhvbk\" (UniqueName: \"kubernetes.io/projected/4a099162-9e26-49c0-9e37-0a6338a58caa-kube-api-access-dhvbk\") pod \"isvc-paddle-predictor-6d5c9fd4f6-q6v2s\" (UID: \"4a099162-9e26-49c0-9e37-0a6338a58caa\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q6v2s"
Jul 27 21:51:32.162584 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:51:32.162457    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a099162-9e26-49c0-9e37-0a6338a58caa-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-q6v2s\" (UID: \"4a099162-9e26-49c0-9e37-0a6338a58caa\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q6v2s"
Jul 27 21:51:32.162584 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:51:32.162508    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-predictor-serving-cert: secret "isvc-paddle-predictor-serving-cert" not found
Jul 27 21:51:32.162584 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:51:32.162514    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a099162-9e26-49c0-9e37-0a6338a58caa-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-q6v2s\" (UID: \"4a099162-9e26-49c0-9e37-0a6338a58caa\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q6v2s"
Jul 27 21:51:32.162584 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:51:32.162546    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a099162-9e26-49c0-9e37-0a6338a58caa-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-q6v2s\" (UID: \"4a099162-9e26-49c0-9e37-0a6338a58caa\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q6v2s"
Jul 27 21:51:32.162584 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:51:32.162564    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a099162-9e26-49c0-9e37-0a6338a58caa-proxy-tls podName:4a099162-9e26-49c0-9e37-0a6338a58caa nodeName:}" failed. No retries permitted until 2026-07-27 21:51:32.662548601 +0000 UTC m=+2816.029738448 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4a099162-9e26-49c0-9e37-0a6338a58caa-proxy-tls") pod "isvc-paddle-predictor-6d5c9fd4f6-q6v2s" (UID: "4a099162-9e26-49c0-9e37-0a6338a58caa") : secret "isvc-paddle-predictor-serving-cert" not found
Jul 27 21:51:32.162947 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:51:32.162931    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a099162-9e26-49c0-9e37-0a6338a58caa-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-q6v2s\" (UID: \"4a099162-9e26-49c0-9e37-0a6338a58caa\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q6v2s"
Jul 27 21:51:32.163265 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:51:32.163244    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a099162-9e26-49c0-9e37-0a6338a58caa-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-q6v2s\" (UID: \"4a099162-9e26-49c0-9e37-0a6338a58caa\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q6v2s"
Jul 27 21:51:32.163337 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:51:32.163285    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a099162-9e26-49c0-9e37-0a6338a58caa-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-q6v2s\" (UID: \"4a099162-9e26-49c0-9e37-0a6338a58caa\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q6v2s"
Jul 27 21:51:32.173786 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:51:32.173749    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhvbk\" (UniqueName: \"kubernetes.io/projected/4a099162-9e26-49c0-9e37-0a6338a58caa-kube-api-access-dhvbk\") pod \"isvc-paddle-predictor-6d5c9fd4f6-q6v2s\" (UID: \"4a099162-9e26-49c0-9e37-0a6338a58caa\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q6v2s"
Jul 27 21:51:32.252996 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:51:32.252932    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:51:32.666871 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:51:32.666784    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a099162-9e26-49c0-9e37-0a6338a58caa-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-q6v2s\" (UID: \"4a099162-9e26-49c0-9e37-0a6338a58caa\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q6v2s"
Jul 27 21:51:32.669291 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:51:32.669269    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a099162-9e26-49c0-9e37-0a6338a58caa-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-q6v2s\" (UID: \"4a099162-9e26-49c0-9e37-0a6338a58caa\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q6v2s"
Jul 27 21:51:32.924554 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:51:32.924528    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q6v2s"
Jul 27 21:51:33.083677 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:51:33.083654    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q6v2s"]
Jul 27 21:51:33.085625 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:51:33.085600    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a099162_9e26_49c0_9e37_0a6338a58caa.slice/crio-e3104d2b241a97603db2c4575734b81d32eeac0de78025d1fb4ba5d6c05c0538 WatchSource:0}: Error finding container e3104d2b241a97603db2c4575734b81d32eeac0de78025d1fb4ba5d6c05c0538: Status 404 returned error can't find the container with id e3104d2b241a97603db2c4575734b81d32eeac0de78025d1fb4ba5d6c05c0538
Jul 27 21:51:33.439346 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:51:33.439303    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q6v2s" event={"ID":"4a099162-9e26-49c0-9e37-0a6338a58caa","Type":"ContainerStarted","Data":"e8038bb9ec6a4cbc354c6021df44c69dd4dff146bf49604350d66cff40d0ee55"}
Jul 27 21:51:33.439536 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:51:33.439361    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q6v2s" event={"ID":"4a099162-9e26-49c0-9e37-0a6338a58caa","Type":"ContainerStarted","Data":"e3104d2b241a97603db2c4575734b81d32eeac0de78025d1fb4ba5d6c05c0538"}
Jul 27 21:51:38.460491 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:51:38.460459    2577 generic.go:356] "Generic (PLEG): container finished" podID="4a099162-9e26-49c0-9e37-0a6338a58caa" containerID="e8038bb9ec6a4cbc354c6021df44c69dd4dff146bf49604350d66cff40d0ee55" exitCode=0
Jul 27 21:51:38.460883 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:51:38.460523    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q6v2s" event={"ID":"4a099162-9e26-49c0-9e37-0a6338a58caa","Type":"ContainerDied","Data":"e8038bb9ec6a4cbc354c6021df44c69dd4dff146bf49604350d66cff40d0ee55"}
Jul 27 21:51:38.461710 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:51:38.461675    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 21:51:50.514398 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:51:50.514364    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q6v2s" event={"ID":"4a099162-9e26-49c0-9e37-0a6338a58caa","Type":"ContainerStarted","Data":"96c817dd1d19abedf80957acee1caabb8e895fddc922652abded2d047b248e60"}
Jul 27 21:51:50.514832 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:51:50.514431    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q6v2s" event={"ID":"4a099162-9e26-49c0-9e37-0a6338a58caa","Type":"ContainerStarted","Data":"d527bb815d3b09043cdc4c43f1ef43443c37f9ae8ef0eb45d22f502732653316"}
Jul 27 21:51:50.514832 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:51:50.514637    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q6v2s"
Jul 27 21:51:50.536975 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:51:50.536930    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q6v2s" podStartSLOduration=4.227801447 podStartE2EDuration="19.536913744s" totalImagesPullingTime="10.997865149s" totalInitContainerRuntime="4.311247148s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:51:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:51:50.535250697 +0000 UTC m=+2833.902440563" watchObservedRunningTime="2026-07-27 21:51:50.536913744 +0000 UTC m=+2833.904103611"
Jul 27 21:51:51.518820 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:51:51.518782    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q6v2s"
Jul 27 21:51:51.520141 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:51:51.520111    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q6v2s" podUID="4a099162-9e26-49c0-9e37-0a6338a58caa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 27 21:51:52.523661 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:51:52.523620    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q6v2s" podUID="4a099162-9e26-49c0-9e37-0a6338a58caa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 27 21:51:57.528688 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:51:57.528662    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q6v2s"
Jul 27 21:51:57.529180 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:51:57.529153    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q6v2s" podUID="4a099162-9e26-49c0-9e37-0a6338a58caa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 27 21:52:07.529214 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:52:07.529174    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q6v2s" podUID="4a099162-9e26-49c0-9e37-0a6338a58caa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 27 21:52:17.530052 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:52:17.530006    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q6v2s" podUID="4a099162-9e26-49c0-9e37-0a6338a58caa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 27 21:52:27.530078 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:52:27.530041    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q6v2s" podUID="4a099162-9e26-49c0-9e37-0a6338a58caa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 27 21:52:37.529947 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:52:37.529873    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q6v2s"
Jul 27 21:52:43.654134 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:52:43.654100    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q6v2s"]
Jul 27 21:52:43.654554 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:52:43.654346    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q6v2s" podUID="4a099162-9e26-49c0-9e37-0a6338a58caa" containerName="kserve-container" containerID="cri-o://d527bb815d3b09043cdc4c43f1ef43443c37f9ae8ef0eb45d22f502732653316" gracePeriod=30
Jul 27 21:52:43.654554 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:52:43.654370    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q6v2s" podUID="4a099162-9e26-49c0-9e37-0a6338a58caa" containerName="kube-rbac-proxy" containerID="cri-o://96c817dd1d19abedf80957acee1caabb8e895fddc922652abded2d047b248e60" gracePeriod=30
Jul 27 21:52:43.675144 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:52:43.675119    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:52:44.728437 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:52:44.728397    2577 generic.go:356] "Generic (PLEG): container finished" podID="4a099162-9e26-49c0-9e37-0a6338a58caa" containerID="96c817dd1d19abedf80957acee1caabb8e895fddc922652abded2d047b248e60" exitCode=2
Jul 27 21:52:44.728437 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:52:44.728431    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q6v2s" event={"ID":"4a099162-9e26-49c0-9e37-0a6338a58caa","Type":"ContainerDied","Data":"96c817dd1d19abedf80957acee1caabb8e895fddc922652abded2d047b248e60"}
Jul 27 21:52:46.092322 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:52:46.092298    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q6v2s"
Jul 27 21:52:46.158260 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:52:46.158228    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a099162-9e26-49c0-9e37-0a6338a58caa-kube-rbac-proxy-sar-config\") pod \"4a099162-9e26-49c0-9e37-0a6338a58caa\" (UID: \"4a099162-9e26-49c0-9e37-0a6338a58caa\") "
Jul 27 21:52:46.158432 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:52:46.158289    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a099162-9e26-49c0-9e37-0a6338a58caa-proxy-tls\") pod \"4a099162-9e26-49c0-9e37-0a6338a58caa\" (UID: \"4a099162-9e26-49c0-9e37-0a6338a58caa\") "
Jul 27 21:52:46.158432 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:52:46.158333    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a099162-9e26-49c0-9e37-0a6338a58caa-kserve-provision-location\") pod \"4a099162-9e26-49c0-9e37-0a6338a58caa\" (UID: \"4a099162-9e26-49c0-9e37-0a6338a58caa\") "
Jul 27 21:52:46.158432 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:52:46.158357    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhvbk\" (UniqueName: \"kubernetes.io/projected/4a099162-9e26-49c0-9e37-0a6338a58caa-kube-api-access-dhvbk\") pod \"4a099162-9e26-49c0-9e37-0a6338a58caa\" (UID: \"4a099162-9e26-49c0-9e37-0a6338a58caa\") "
Jul 27 21:52:46.158432 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:52:46.158423    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a099162-9e26-49c0-9e37-0a6338a58caa-cabundle-cert\") pod \"4a099162-9e26-49c0-9e37-0a6338a58caa\" (UID: \"4a099162-9e26-49c0-9e37-0a6338a58caa\") "
Jul 27 21:52:46.158717 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:52:46.158659    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a099162-9e26-49c0-9e37-0a6338a58caa-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4a099162-9e26-49c0-9e37-0a6338a58caa" (UID: "4a099162-9e26-49c0-9e37-0a6338a58caa"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:52:46.158853 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:52:46.158830    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a099162-9e26-49c0-9e37-0a6338a58caa-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4a099162-9e26-49c0-9e37-0a6338a58caa" (UID: "4a099162-9e26-49c0-9e37-0a6338a58caa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:52:46.160455 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:52:46.160428    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a099162-9e26-49c0-9e37-0a6338a58caa-kube-api-access-dhvbk" (OuterVolumeSpecName: "kube-api-access-dhvbk") pod "4a099162-9e26-49c0-9e37-0a6338a58caa" (UID: "4a099162-9e26-49c0-9e37-0a6338a58caa"). InnerVolumeSpecName "kube-api-access-dhvbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:52:46.160642 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:52:46.160624    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a099162-9e26-49c0-9e37-0a6338a58caa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4a099162-9e26-49c0-9e37-0a6338a58caa" (UID: "4a099162-9e26-49c0-9e37-0a6338a58caa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:52:46.168893 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:52:46.168869    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a099162-9e26-49c0-9e37-0a6338a58caa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4a099162-9e26-49c0-9e37-0a6338a58caa" (UID: "4a099162-9e26-49c0-9e37-0a6338a58caa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:52:46.259452 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:52:46.259387    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a099162-9e26-49c0-9e37-0a6338a58caa-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:52:46.259452 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:52:46.259411    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhvbk\" (UniqueName: \"kubernetes.io/projected/4a099162-9e26-49c0-9e37-0a6338a58caa-kube-api-access-dhvbk\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:52:46.259452 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:52:46.259423    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a099162-9e26-49c0-9e37-0a6338a58caa-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:52:46.259452 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:52:46.259431    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a099162-9e26-49c0-9e37-0a6338a58caa-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:52:46.259452 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:52:46.259440    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a099162-9e26-49c0-9e37-0a6338a58caa-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:52:46.738446 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:52:46.738418    2577 generic.go:356] "Generic (PLEG): container finished" podID="4a099162-9e26-49c0-9e37-0a6338a58caa" containerID="d527bb815d3b09043cdc4c43f1ef43443c37f9ae8ef0eb45d22f502732653316" exitCode=0
Jul 27 21:52:46.738608 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:52:46.738468    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q6v2s" event={"ID":"4a099162-9e26-49c0-9e37-0a6338a58caa","Type":"ContainerDied","Data":"d527bb815d3b09043cdc4c43f1ef43443c37f9ae8ef0eb45d22f502732653316"}
Jul 27 21:52:46.738608 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:52:46.738477    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q6v2s"
Jul 27 21:52:46.738608 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:52:46.738514    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q6v2s" event={"ID":"4a099162-9e26-49c0-9e37-0a6338a58caa","Type":"ContainerDied","Data":"e3104d2b241a97603db2c4575734b81d32eeac0de78025d1fb4ba5d6c05c0538"}
Jul 27 21:52:46.738608 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:52:46.738553    2577 scope.go:117] "RemoveContainer" containerID="96c817dd1d19abedf80957acee1caabb8e895fddc922652abded2d047b248e60"
Jul 27 21:52:46.747503 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:52:46.747484    2577 scope.go:117] "RemoveContainer" containerID="d527bb815d3b09043cdc4c43f1ef43443c37f9ae8ef0eb45d22f502732653316"
Jul 27 21:52:46.754338 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:52:46.754321    2577 scope.go:117] "RemoveContainer" containerID="e8038bb9ec6a4cbc354c6021df44c69dd4dff146bf49604350d66cff40d0ee55"
Jul 27 21:52:46.760125 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:52:46.760105    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q6v2s"]
Jul 27 21:52:46.761271 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:52:46.761250    2577 scope.go:117] "RemoveContainer" containerID="96c817dd1d19abedf80957acee1caabb8e895fddc922652abded2d047b248e60"
Jul 27 21:52:46.761514 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:52:46.761496    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96c817dd1d19abedf80957acee1caabb8e895fddc922652abded2d047b248e60\": container with ID starting with 96c817dd1d19abedf80957acee1caabb8e895fddc922652abded2d047b248e60 not found: ID does not exist" containerID="96c817dd1d19abedf80957acee1caabb8e895fddc922652abded2d047b248e60"
Jul 27 21:52:46.761568 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:52:46.761523    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96c817dd1d19abedf80957acee1caabb8e895fddc922652abded2d047b248e60"} err="failed to get container status \"96c817dd1d19abedf80957acee1caabb8e895fddc922652abded2d047b248e60\": rpc error: code = NotFound desc = could not find container \"96c817dd1d19abedf80957acee1caabb8e895fddc922652abded2d047b248e60\": container with ID starting with 96c817dd1d19abedf80957acee1caabb8e895fddc922652abded2d047b248e60 not found: ID does not exist"
Jul 27 21:52:46.761568 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:52:46.761541    2577 scope.go:117] "RemoveContainer" containerID="d527bb815d3b09043cdc4c43f1ef43443c37f9ae8ef0eb45d22f502732653316"
Jul 27 21:52:46.761773 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:52:46.761744    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d527bb815d3b09043cdc4c43f1ef43443c37f9ae8ef0eb45d22f502732653316\": container with ID starting with d527bb815d3b09043cdc4c43f1ef43443c37f9ae8ef0eb45d22f502732653316 not found: ID does not exist" containerID="d527bb815d3b09043cdc4c43f1ef43443c37f9ae8ef0eb45d22f502732653316"
Jul 27 21:52:46.761821 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:52:46.761802    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d527bb815d3b09043cdc4c43f1ef43443c37f9ae8ef0eb45d22f502732653316"} err="failed to get container status \"d527bb815d3b09043cdc4c43f1ef43443c37f9ae8ef0eb45d22f502732653316\": rpc error: code = NotFound desc = could not find container \"d527bb815d3b09043cdc4c43f1ef43443c37f9ae8ef0eb45d22f502732653316\": container with ID starting with d527bb815d3b09043cdc4c43f1ef43443c37f9ae8ef0eb45d22f502732653316 not found: ID does not exist"
Jul 27 21:52:46.761821 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:52:46.761818    2577 scope.go:117] "RemoveContainer" containerID="e8038bb9ec6a4cbc354c6021df44c69dd4dff146bf49604350d66cff40d0ee55"
Jul 27 21:52:46.762097 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:52:46.762071    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8038bb9ec6a4cbc354c6021df44c69dd4dff146bf49604350d66cff40d0ee55\": container with ID starting with e8038bb9ec6a4cbc354c6021df44c69dd4dff146bf49604350d66cff40d0ee55 not found: ID does not exist" containerID="e8038bb9ec6a4cbc354c6021df44c69dd4dff146bf49604350d66cff40d0ee55"
Jul 27 21:52:46.762184 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:52:46.762104    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8038bb9ec6a4cbc354c6021df44c69dd4dff146bf49604350d66cff40d0ee55"} err="failed to get container status \"e8038bb9ec6a4cbc354c6021df44c69dd4dff146bf49604350d66cff40d0ee55\": rpc error: code = NotFound desc = could not find container \"e8038bb9ec6a4cbc354c6021df44c69dd4dff146bf49604350d66cff40d0ee55\": container with ID starting with e8038bb9ec6a4cbc354c6021df44c69dd4dff146bf49604350d66cff40d0ee55 not found: ID does not exist"
Jul 27 21:52:46.764280 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:52:46.764259    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-q6v2s"]
Jul 27 21:52:47.136324 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:52:47.136252    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a099162-9e26-49c0-9e37-0a6338a58caa" path="/var/lib/kubelet/pods/4a099162-9e26-49c0-9e37-0a6338a58caa/volumes"
Jul 27 21:52:48.792310 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:52:48.792271    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:52:49.128247 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:52:49.128154    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:54:00.440151 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:54:00.440067    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:54:05.540344 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:54:05.540292    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:54:05.572248 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:54:05.572217    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n"]
Jul 27 21:54:05.572515 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:54:05.572501    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a099162-9e26-49c0-9e37-0a6338a58caa" containerName="storage-initializer"
Jul 27 21:54:05.572515 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:54:05.572515    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a099162-9e26-49c0-9e37-0a6338a58caa" containerName="storage-initializer"
Jul 27 21:54:05.572621 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:54:05.572530    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a099162-9e26-49c0-9e37-0a6338a58caa" containerName="kserve-container"
Jul 27 21:54:05.572621 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:54:05.572536    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a099162-9e26-49c0-9e37-0a6338a58caa" containerName="kserve-container"
Jul 27 21:54:05.572621 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:54:05.572548    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a099162-9e26-49c0-9e37-0a6338a58caa" containerName="kube-rbac-proxy"
Jul 27 21:54:05.572621 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:54:05.572556    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a099162-9e26-49c0-9e37-0a6338a58caa" containerName="kube-rbac-proxy"
Jul 27 21:54:05.572621 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:54:05.572606    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a099162-9e26-49c0-9e37-0a6338a58caa" containerName="kube-rbac-proxy"
Jul 27 21:54:05.572621 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:54:05.572615    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a099162-9e26-49c0-9e37-0a6338a58caa" containerName="kserve-container"
Jul 27 21:54:05.575751 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:54:05.575737    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n"
Jul 27 21:54:05.578248 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:54:05.578221    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 21:54:05.578396 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:54:05.578301    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 21:54:05.578396 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:54:05.578302    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
Jul 27 21:54:05.578396 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:54:05.578353    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
Jul 27 21:54:05.578570 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:54:05.578310    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 21:54:05.578570 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:54:05.578418    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v7c66\""
Jul 27 21:54:05.578570 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:54:05.578423    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 21:54:05.584832 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:54:05.584812    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n"]
Jul 27 21:54:05.683138 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:54:05.683107    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/108c9f59-3150-47d1-9b23-f7e06259a617-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n\" (UID: \"108c9f59-3150-47d1-9b23-f7e06259a617\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n"
Jul 27 21:54:05.683138 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:54:05.683143    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/108c9f59-3150-47d1-9b23-f7e06259a617-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n\" (UID: \"108c9f59-3150-47d1-9b23-f7e06259a617\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n"
Jul 27 21:54:05.683333 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:54:05.683169    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/108c9f59-3150-47d1-9b23-f7e06259a617-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n\" (UID: \"108c9f59-3150-47d1-9b23-f7e06259a617\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n"
Jul 27 21:54:05.683333 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:54:05.683252    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsfzj\" (UniqueName: \"kubernetes.io/projected/108c9f59-3150-47d1-9b23-f7e06259a617-kube-api-access-hsfzj\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n\" (UID: \"108c9f59-3150-47d1-9b23-f7e06259a617\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n"
Jul 27 21:54:05.683333 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:54:05.683272    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/108c9f59-3150-47d1-9b23-f7e06259a617-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n\" (UID: \"108c9f59-3150-47d1-9b23-f7e06259a617\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n"
Jul 27 21:54:05.763804 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:54:05.763749    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:54:05.783857 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:54:05.783834    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsfzj\" (UniqueName: \"kubernetes.io/projected/108c9f59-3150-47d1-9b23-f7e06259a617-kube-api-access-hsfzj\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n\" (UID: \"108c9f59-3150-47d1-9b23-f7e06259a617\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n"
Jul 27 21:54:05.783969 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:54:05.783860    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/108c9f59-3150-47d1-9b23-f7e06259a617-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n\" (UID: \"108c9f59-3150-47d1-9b23-f7e06259a617\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n"
Jul 27 21:54:05.783969 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:54:05.783887    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/108c9f59-3150-47d1-9b23-f7e06259a617-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n\" (UID: \"108c9f59-3150-47d1-9b23-f7e06259a617\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n"
Jul 27 21:54:05.783969 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:54:05.783916    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/108c9f59-3150-47d1-9b23-f7e06259a617-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n\" (UID: \"108c9f59-3150-47d1-9b23-f7e06259a617\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n"
Jul 27 21:54:05.784121 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:54:05.784099    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/108c9f59-3150-47d1-9b23-f7e06259a617-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n\" (UID: \"108c9f59-3150-47d1-9b23-f7e06259a617\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n"
Jul 27 21:54:05.784276 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:54:05.784258    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/108c9f59-3150-47d1-9b23-f7e06259a617-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n\" (UID: \"108c9f59-3150-47d1-9b23-f7e06259a617\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n"
Jul 27 21:54:05.784601 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:54:05.784583    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/108c9f59-3150-47d1-9b23-f7e06259a617-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n\" (UID: \"108c9f59-3150-47d1-9b23-f7e06259a617\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n"
Jul 27 21:54:05.784653 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:54:05.784585    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/108c9f59-3150-47d1-9b23-f7e06259a617-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n\" (UID: \"108c9f59-3150-47d1-9b23-f7e06259a617\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n"
Jul 27 21:54:05.786349 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:54:05.786328    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/108c9f59-3150-47d1-9b23-f7e06259a617-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n\" (UID: \"108c9f59-3150-47d1-9b23-f7e06259a617\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n"
Jul 27 21:54:05.791986 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:54:05.791935    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsfzj\" (UniqueName: \"kubernetes.io/projected/108c9f59-3150-47d1-9b23-f7e06259a617-kube-api-access-hsfzj\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n\" (UID: \"108c9f59-3150-47d1-9b23-f7e06259a617\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n"
Jul 27 21:54:05.887231 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:54:05.887203    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n"
Jul 27 21:54:06.015360 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:54:06.015336    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n"]
Jul 27 21:54:06.017700 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:54:06.017669    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod108c9f59_3150_47d1_9b23_f7e06259a617.slice/crio-747fb7d16d91951ddbca920d364ad4019217d4b4a980f22c067b6e01c72ca32e WatchSource:0}: Error finding container 747fb7d16d91951ddbca920d364ad4019217d4b4a980f22c067b6e01c72ca32e: Status 404 returned error can't find the container with id 747fb7d16d91951ddbca920d364ad4019217d4b4a980f22c067b6e01c72ca32e
Jul 27 21:54:06.041150 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:54:06.041127    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n" event={"ID":"108c9f59-3150-47d1-9b23-f7e06259a617","Type":"ContainerStarted","Data":"747fb7d16d91951ddbca920d364ad4019217d4b4a980f22c067b6e01c72ca32e"}
Jul 27 21:54:07.047291 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:54:07.047258    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n" event={"ID":"108c9f59-3150-47d1-9b23-f7e06259a617","Type":"ContainerStarted","Data":"997f711058baf463059c90d9d6e93ca2d81465fe2921906b1ef93117bec7a549"}
Jul 27 21:54:11.063548 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:54:11.063513    2577 generic.go:356] "Generic (PLEG): container finished" podID="108c9f59-3150-47d1-9b23-f7e06259a617" containerID="997f711058baf463059c90d9d6e93ca2d81465fe2921906b1ef93117bec7a549" exitCode=0
Jul 27 21:54:11.063967 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:54:11.063585    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n" event={"ID":"108c9f59-3150-47d1-9b23-f7e06259a617","Type":"ContainerDied","Data":"997f711058baf463059c90d9d6e93ca2d81465fe2921906b1ef93117bec7a549"}
Jul 27 21:54:12.070167 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:54:12.070132    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n" event={"ID":"108c9f59-3150-47d1-9b23-f7e06259a617","Type":"ContainerStarted","Data":"667417d9cc05f00ae244b269bb391ffb13b1d2f4f8ed6dc0d5e9ad3152cce0ca"}
Jul 27 21:54:12.070532 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:54:12.070198    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n" event={"ID":"108c9f59-3150-47d1-9b23-f7e06259a617","Type":"ContainerStarted","Data":"90d136ec5786b64fcb499f6f00718143f56e5de839b37d7387d2fc2f5578ce48"}
Jul 27 21:54:12.070532 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:54:12.070387    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n"
Jul 27 21:54:12.089559 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:54:12.089504    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n" podStartSLOduration=2.634140243 podStartE2EDuration="7.089489177s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.455348934s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:54:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:54:12.087648201 +0000 UTC m=+2975.454838068" watchObservedRunningTime="2026-07-27 21:54:12.089489177 +0000 UTC m=+2975.456679042"
Jul 27 21:54:13.074386 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:54:13.074356    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n"
Jul 27 21:54:13.075411 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:54:13.075385    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n" podUID="108c9f59-3150-47d1-9b23-f7e06259a617" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 27 21:54:14.078501 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:54:14.078455    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n" podUID="108c9f59-3150-47d1-9b23-f7e06259a617" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 27 21:54:19.083664 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:54:19.083634    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n"
Jul 27 21:54:19.084187 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:54:19.084161    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n" podUID="108c9f59-3150-47d1-9b23-f7e06259a617" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 27 21:54:29.085085 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:54:29.085040    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n" podUID="108c9f59-3150-47d1-9b23-f7e06259a617" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 27 21:54:39.084638 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:54:39.084594    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n" podUID="108c9f59-3150-47d1-9b23-f7e06259a617" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 27 21:54:49.084872 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:54:49.084830    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n" podUID="108c9f59-3150-47d1-9b23-f7e06259a617" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 27 21:54:59.085830 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:54:59.085797    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n"
Jul 27 21:55:07.660751 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:07.660706    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n"]
Jul 27 21:55:07.661246 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:07.661063    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n" podUID="108c9f59-3150-47d1-9b23-f7e06259a617" containerName="kserve-container" containerID="cri-o://90d136ec5786b64fcb499f6f00718143f56e5de839b37d7387d2fc2f5578ce48" gracePeriod=30
Jul 27 21:55:07.661246 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:07.661103    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n" podUID="108c9f59-3150-47d1-9b23-f7e06259a617" containerName="kube-rbac-proxy" containerID="cri-o://667417d9cc05f00ae244b269bb391ffb13b1d2f4f8ed6dc0d5e9ad3152cce0ca" gracePeriod=30
Jul 27 21:55:07.681061 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:07.681033    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:55:08.287246 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:08.287214    2577 generic.go:356] "Generic (PLEG): container finished" podID="108c9f59-3150-47d1-9b23-f7e06259a617" containerID="667417d9cc05f00ae244b269bb391ffb13b1d2f4f8ed6dc0d5e9ad3152cce0ca" exitCode=2
Jul 27 21:55:08.287419 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:08.287281    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n" event={"ID":"108c9f59-3150-47d1-9b23-f7e06259a617","Type":"ContainerDied","Data":"667417d9cc05f00ae244b269bb391ffb13b1d2f4f8ed6dc0d5e9ad3152cce0ca"}
Jul 27 21:55:09.078776 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:09.078724    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n" podUID="108c9f59-3150-47d1-9b23-f7e06259a617" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.45:8643/healthz\": dial tcp 10.133.0.45:8643: connect: connection refused"
Jul 27 21:55:09.084873 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:09.084849    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n" podUID="108c9f59-3150-47d1-9b23-f7e06259a617" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 27 21:55:12.779233 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:12.779196    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:55:12.811654 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:12.811616    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rgmk6"]
Jul 27 21:55:12.815170 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:12.815145    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rgmk6"
Jul 27 21:55:12.817405 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:12.817382    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
Jul 27 21:55:12.817405 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:12.817398    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\""
Jul 27 21:55:12.827338 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:12.827298    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rgmk6"]
Jul 27 21:55:12.918910 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:12.918873    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5pvn\" (UniqueName: \"kubernetes.io/projected/ef39854c-4198-47ad-b6b2-760ebd9534f8-kube-api-access-j5pvn\") pod \"isvc-pmml-predictor-8674795f59-rgmk6\" (UID: \"ef39854c-4198-47ad-b6b2-760ebd9534f8\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rgmk6"
Jul 27 21:55:12.919073 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:12.918917    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef39854c-4198-47ad-b6b2-760ebd9534f8-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-rgmk6\" (UID: \"ef39854c-4198-47ad-b6b2-760ebd9534f8\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rgmk6"
Jul 27 21:55:12.919073 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:12.918941    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef39854c-4198-47ad-b6b2-760ebd9534f8-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-rgmk6\" (UID: \"ef39854c-4198-47ad-b6b2-760ebd9534f8\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rgmk6"
Jul 27 21:55:12.919073 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:12.919057    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef39854c-4198-47ad-b6b2-760ebd9534f8-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-rgmk6\" (UID: \"ef39854c-4198-47ad-b6b2-760ebd9534f8\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rgmk6"
Jul 27 21:55:12.919193 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:12.919120    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef39854c-4198-47ad-b6b2-760ebd9534f8-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-rgmk6\" (UID: \"ef39854c-4198-47ad-b6b2-760ebd9534f8\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rgmk6"
Jul 27 21:55:13.020525 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:13.020485    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef39854c-4198-47ad-b6b2-760ebd9534f8-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-rgmk6\" (UID: \"ef39854c-4198-47ad-b6b2-760ebd9534f8\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rgmk6"
Jul 27 21:55:13.020717 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:13.020558    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef39854c-4198-47ad-b6b2-760ebd9534f8-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-rgmk6\" (UID: \"ef39854c-4198-47ad-b6b2-760ebd9534f8\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rgmk6"
Jul 27 21:55:13.020717 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:13.020607    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5pvn\" (UniqueName: \"kubernetes.io/projected/ef39854c-4198-47ad-b6b2-760ebd9534f8-kube-api-access-j5pvn\") pod \"isvc-pmml-predictor-8674795f59-rgmk6\" (UID: \"ef39854c-4198-47ad-b6b2-760ebd9534f8\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rgmk6"
Jul 27 21:55:13.020717 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:13.020629    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef39854c-4198-47ad-b6b2-760ebd9534f8-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-rgmk6\" (UID: \"ef39854c-4198-47ad-b6b2-760ebd9534f8\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rgmk6"
Jul 27 21:55:13.020717 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:55:13.020641    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-pmml-predictor-serving-cert: secret "isvc-pmml-predictor-serving-cert" not found
Jul 27 21:55:13.020717 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:13.020652    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef39854c-4198-47ad-b6b2-760ebd9534f8-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-rgmk6\" (UID: \"ef39854c-4198-47ad-b6b2-760ebd9534f8\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rgmk6"
Jul 27 21:55:13.020717 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:55:13.020710    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef39854c-4198-47ad-b6b2-760ebd9534f8-proxy-tls podName:ef39854c-4198-47ad-b6b2-760ebd9534f8 nodeName:}" failed. No retries permitted until 2026-07-27 21:55:13.520693241 +0000 UTC m=+3036.887883084 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ef39854c-4198-47ad-b6b2-760ebd9534f8-proxy-tls") pod "isvc-pmml-predictor-8674795f59-rgmk6" (UID: "ef39854c-4198-47ad-b6b2-760ebd9534f8") : secret "isvc-pmml-predictor-serving-cert" not found
Jul 27 21:55:13.021140 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:13.021118    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef39854c-4198-47ad-b6b2-760ebd9534f8-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-rgmk6\" (UID: \"ef39854c-4198-47ad-b6b2-760ebd9534f8\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rgmk6"
Jul 27 21:55:13.021435 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:13.021414    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef39854c-4198-47ad-b6b2-760ebd9534f8-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-rgmk6\" (UID: \"ef39854c-4198-47ad-b6b2-760ebd9534f8\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rgmk6"
Jul 27 21:55:13.021509 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:13.021464    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef39854c-4198-47ad-b6b2-760ebd9534f8-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-rgmk6\" (UID: \"ef39854c-4198-47ad-b6b2-760ebd9534f8\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rgmk6"
Jul 27 21:55:13.039611 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:13.039558    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5pvn\" (UniqueName: \"kubernetes.io/projected/ef39854c-4198-47ad-b6b2-760ebd9534f8-kube-api-access-j5pvn\") pod \"isvc-pmml-predictor-8674795f59-rgmk6\" (UID: \"ef39854c-4198-47ad-b6b2-760ebd9534f8\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rgmk6"
Jul 27 21:55:13.426755 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:13.426728    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:55:13.526098 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:13.526062    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef39854c-4198-47ad-b6b2-760ebd9534f8-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-rgmk6\" (UID: \"ef39854c-4198-47ad-b6b2-760ebd9534f8\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rgmk6"
Jul 27 21:55:13.528626 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:13.528591    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef39854c-4198-47ad-b6b2-760ebd9534f8-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-rgmk6\" (UID: \"ef39854c-4198-47ad-b6b2-760ebd9534f8\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rgmk6"
Jul 27 21:55:13.725892 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:13.725795    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rgmk6"
Jul 27 21:55:13.854634 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:13.854602    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rgmk6"]
Jul 27 21:55:13.858391 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:55:13.858367    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef39854c_4198_47ad_b6b2_760ebd9534f8.slice/crio-069c57205d26789eb9161a8348ceef484ecada458f307e0923ad07e57dc5e54a WatchSource:0}: Error finding container 069c57205d26789eb9161a8348ceef484ecada458f307e0923ad07e57dc5e54a: Status 404 returned error can't find the container with id 069c57205d26789eb9161a8348ceef484ecada458f307e0923ad07e57dc5e54a
Jul 27 21:55:14.079128 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:14.079090    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n" podUID="108c9f59-3150-47d1-9b23-f7e06259a617" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.45:8643/healthz\": dial tcp 10.133.0.45:8643: connect: connection refused"
Jul 27 21:55:14.204382 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:14.204355    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n"
Jul 27 21:55:14.312425 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:14.312337    2577 generic.go:356] "Generic (PLEG): container finished" podID="108c9f59-3150-47d1-9b23-f7e06259a617" containerID="90d136ec5786b64fcb499f6f00718143f56e5de839b37d7387d2fc2f5578ce48" exitCode=0
Jul 27 21:55:14.312425 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:14.312399    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n"
Jul 27 21:55:14.312425 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:14.312418    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n" event={"ID":"108c9f59-3150-47d1-9b23-f7e06259a617","Type":"ContainerDied","Data":"90d136ec5786b64fcb499f6f00718143f56e5de839b37d7387d2fc2f5578ce48"}
Jul 27 21:55:14.312680 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:14.312484    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n" event={"ID":"108c9f59-3150-47d1-9b23-f7e06259a617","Type":"ContainerDied","Data":"747fb7d16d91951ddbca920d364ad4019217d4b4a980f22c067b6e01c72ca32e"}
Jul 27 21:55:14.312680 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:14.312523    2577 scope.go:117] "RemoveContainer" containerID="667417d9cc05f00ae244b269bb391ffb13b1d2f4f8ed6dc0d5e9ad3152cce0ca"
Jul 27 21:55:14.314063 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:14.314039    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rgmk6" event={"ID":"ef39854c-4198-47ad-b6b2-760ebd9534f8","Type":"ContainerStarted","Data":"7d4ef90573c5cb4e72c90e21ed38b6713c4eee942ea84eae077fd0dfb9eebc57"}
Jul 27 21:55:14.314197 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:14.314090    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rgmk6" event={"ID":"ef39854c-4198-47ad-b6b2-760ebd9534f8","Type":"ContainerStarted","Data":"069c57205d26789eb9161a8348ceef484ecada458f307e0923ad07e57dc5e54a"}
Jul 27 21:55:14.320250 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:14.320231    2577 scope.go:117] "RemoveContainer" containerID="90d136ec5786b64fcb499f6f00718143f56e5de839b37d7387d2fc2f5578ce48"
Jul 27 21:55:14.327540 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:14.327522    2577 scope.go:117] "RemoveContainer" containerID="997f711058baf463059c90d9d6e93ca2d81465fe2921906b1ef93117bec7a549"
Jul 27 21:55:14.333184 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:14.332984    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/108c9f59-3150-47d1-9b23-f7e06259a617-kube-rbac-proxy-sar-config\") pod \"108c9f59-3150-47d1-9b23-f7e06259a617\" (UID: \"108c9f59-3150-47d1-9b23-f7e06259a617\") "
Jul 27 21:55:14.333184 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:14.333019    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hsfzj\" (UniqueName: \"kubernetes.io/projected/108c9f59-3150-47d1-9b23-f7e06259a617-kube-api-access-hsfzj\") pod \"108c9f59-3150-47d1-9b23-f7e06259a617\" (UID: \"108c9f59-3150-47d1-9b23-f7e06259a617\") "
Jul 27 21:55:14.333184 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:14.333058    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/108c9f59-3150-47d1-9b23-f7e06259a617-kserve-provision-location\") pod \"108c9f59-3150-47d1-9b23-f7e06259a617\" (UID: \"108c9f59-3150-47d1-9b23-f7e06259a617\") "
Jul 27 21:55:14.333184 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:14.333133    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/108c9f59-3150-47d1-9b23-f7e06259a617-proxy-tls\") pod \"108c9f59-3150-47d1-9b23-f7e06259a617\" (UID: \"108c9f59-3150-47d1-9b23-f7e06259a617\") "
Jul 27 21:55:14.333184 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:14.333182    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/108c9f59-3150-47d1-9b23-f7e06259a617-cabundle-cert\") pod \"108c9f59-3150-47d1-9b23-f7e06259a617\" (UID: \"108c9f59-3150-47d1-9b23-f7e06259a617\") "
Jul 27 21:55:14.333646 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:14.333616    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/108c9f59-3150-47d1-9b23-f7e06259a617-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "108c9f59-3150-47d1-9b23-f7e06259a617" (UID: "108c9f59-3150-47d1-9b23-f7e06259a617"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:55:14.333775 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:14.333702    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/108c9f59-3150-47d1-9b23-f7e06259a617-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "108c9f59-3150-47d1-9b23-f7e06259a617" (UID: "108c9f59-3150-47d1-9b23-f7e06259a617"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:55:14.335692 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:14.335665    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/108c9f59-3150-47d1-9b23-f7e06259a617-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "108c9f59-3150-47d1-9b23-f7e06259a617" (UID: "108c9f59-3150-47d1-9b23-f7e06259a617"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:55:14.335809 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:14.335704    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/108c9f59-3150-47d1-9b23-f7e06259a617-kube-api-access-hsfzj" (OuterVolumeSpecName: "kube-api-access-hsfzj") pod "108c9f59-3150-47d1-9b23-f7e06259a617" (UID: "108c9f59-3150-47d1-9b23-f7e06259a617"). InnerVolumeSpecName "kube-api-access-hsfzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:55:14.336567 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:14.336552    2577 scope.go:117] "RemoveContainer" containerID="667417d9cc05f00ae244b269bb391ffb13b1d2f4f8ed6dc0d5e9ad3152cce0ca"
Jul 27 21:55:14.336876 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:55:14.336843    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"667417d9cc05f00ae244b269bb391ffb13b1d2f4f8ed6dc0d5e9ad3152cce0ca\": container with ID starting with 667417d9cc05f00ae244b269bb391ffb13b1d2f4f8ed6dc0d5e9ad3152cce0ca not found: ID does not exist" containerID="667417d9cc05f00ae244b269bb391ffb13b1d2f4f8ed6dc0d5e9ad3152cce0ca"
Jul 27 21:55:14.336940 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:14.336878    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"667417d9cc05f00ae244b269bb391ffb13b1d2f4f8ed6dc0d5e9ad3152cce0ca"} err="failed to get container status \"667417d9cc05f00ae244b269bb391ffb13b1d2f4f8ed6dc0d5e9ad3152cce0ca\": rpc error: code = NotFound desc = could not find container \"667417d9cc05f00ae244b269bb391ffb13b1d2f4f8ed6dc0d5e9ad3152cce0ca\": container with ID starting with 667417d9cc05f00ae244b269bb391ffb13b1d2f4f8ed6dc0d5e9ad3152cce0ca not found: ID does not exist"
Jul 27 21:55:14.336940 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:14.336903    2577 scope.go:117] "RemoveContainer" containerID="90d136ec5786b64fcb499f6f00718143f56e5de839b37d7387d2fc2f5578ce48"
Jul 27 21:55:14.337165 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:55:14.337145    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90d136ec5786b64fcb499f6f00718143f56e5de839b37d7387d2fc2f5578ce48\": container with ID starting with 90d136ec5786b64fcb499f6f00718143f56e5de839b37d7387d2fc2f5578ce48 not found: ID does not exist" containerID="90d136ec5786b64fcb499f6f00718143f56e5de839b37d7387d2fc2f5578ce48"
Jul 27 21:55:14.337225 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:14.337172    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90d136ec5786b64fcb499f6f00718143f56e5de839b37d7387d2fc2f5578ce48"} err="failed to get container status \"90d136ec5786b64fcb499f6f00718143f56e5de839b37d7387d2fc2f5578ce48\": rpc error: code = NotFound desc = could not find container \"90d136ec5786b64fcb499f6f00718143f56e5de839b37d7387d2fc2f5578ce48\": container with ID starting with 90d136ec5786b64fcb499f6f00718143f56e5de839b37d7387d2fc2f5578ce48 not found: ID does not exist"
Jul 27 21:55:14.337225 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:14.337191    2577 scope.go:117] "RemoveContainer" containerID="997f711058baf463059c90d9d6e93ca2d81465fe2921906b1ef93117bec7a549"
Jul 27 21:55:14.337717 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:55:14.337691    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"997f711058baf463059c90d9d6e93ca2d81465fe2921906b1ef93117bec7a549\": container with ID starting with 997f711058baf463059c90d9d6e93ca2d81465fe2921906b1ef93117bec7a549 not found: ID does not exist" containerID="997f711058baf463059c90d9d6e93ca2d81465fe2921906b1ef93117bec7a549"
Jul 27 21:55:14.337845 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:14.337723    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"997f711058baf463059c90d9d6e93ca2d81465fe2921906b1ef93117bec7a549"} err="failed to get container status \"997f711058baf463059c90d9d6e93ca2d81465fe2921906b1ef93117bec7a549\": rpc error: code = NotFound desc = could not find container \"997f711058baf463059c90d9d6e93ca2d81465fe2921906b1ef93117bec7a549\": container with ID starting with 997f711058baf463059c90d9d6e93ca2d81465fe2921906b1ef93117bec7a549 not found: ID does not exist"
Jul 27 21:55:14.349185 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:14.349159    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/108c9f59-3150-47d1-9b23-f7e06259a617-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "108c9f59-3150-47d1-9b23-f7e06259a617" (UID: "108c9f59-3150-47d1-9b23-f7e06259a617"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:55:14.434649 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:14.434614    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/108c9f59-3150-47d1-9b23-f7e06259a617-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:55:14.434649 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:14.434641    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsfzj\" (UniqueName: \"kubernetes.io/projected/108c9f59-3150-47d1-9b23-f7e06259a617-kube-api-access-hsfzj\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:55:14.434649 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:14.434652    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/108c9f59-3150-47d1-9b23-f7e06259a617-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:55:14.434649 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:14.434661    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/108c9f59-3150-47d1-9b23-f7e06259a617-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:55:14.435022 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:14.434670    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/108c9f59-3150-47d1-9b23-f7e06259a617-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:55:14.633752 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:14.633720    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n"]
Jul 27 21:55:14.637184 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:14.637159    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-7628n"]
Jul 27 21:55:15.138796 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:15.136648    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="108c9f59-3150-47d1-9b23-f7e06259a617" path="/var/lib/kubelet/pods/108c9f59-3150-47d1-9b23-f7e06259a617/volumes"
Jul 27 21:55:18.331275 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:18.331220    2577 generic.go:356] "Generic (PLEG): container finished" podID="ef39854c-4198-47ad-b6b2-760ebd9534f8" containerID="7d4ef90573c5cb4e72c90e21ed38b6713c4eee942ea84eae077fd0dfb9eebc57" exitCode=0
Jul 27 21:55:18.331675 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:18.331300    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rgmk6" event={"ID":"ef39854c-4198-47ad-b6b2-760ebd9534f8","Type":"ContainerDied","Data":"7d4ef90573c5cb4e72c90e21ed38b6713c4eee942ea84eae077fd0dfb9eebc57"}
Jul 27 21:55:25.373284 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:25.373255    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rgmk6" event={"ID":"ef39854c-4198-47ad-b6b2-760ebd9534f8","Type":"ContainerStarted","Data":"3c38a77ac373bc7d3b771c2a2d0775a7ecc04b741fa2bb909fb5ee74842e5a69"}
Jul 27 21:55:25.373632 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:25.373315    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rgmk6" event={"ID":"ef39854c-4198-47ad-b6b2-760ebd9534f8","Type":"ContainerStarted","Data":"bb09b7ce76cfe839bc4e635bdc77731f5288d5116a5b39eecfc1f37b95b93490"}
Jul 27 21:55:25.373632 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:25.373506    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rgmk6"
Jul 27 21:55:25.391717 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:25.391673    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rgmk6" podStartSLOduration=2.819992914 podStartE2EDuration="13.391655983s" totalImagesPullingTime="6.879429803s" totalInitContainerRuntime="3.692233266s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:55:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:55:25.389582857 +0000 UTC m=+3048.756772722" watchObservedRunningTime="2026-07-27 21:55:25.391655983 +0000 UTC m=+3048.758845848"
Jul 27 21:55:26.376902 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:26.376870    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rgmk6"
Jul 27 21:55:26.378096 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:26.378068    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rgmk6" podUID="ef39854c-4198-47ad-b6b2-760ebd9534f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 27 21:55:27.381065 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:27.381020    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rgmk6" podUID="ef39854c-4198-47ad-b6b2-760ebd9534f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 27 21:55:29.444589 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:29.444499    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:55:32.385754 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:32.385723    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rgmk6"
Jul 27 21:55:32.386323 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:32.386296    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rgmk6" podUID="ef39854c-4198-47ad-b6b2-760ebd9534f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 27 21:55:42.386482 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:42.386434    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rgmk6" podUID="ef39854c-4198-47ad-b6b2-760ebd9534f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 27 21:55:42.661557 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:42.661477    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:55:52.386964 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:55:52.386923    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rgmk6" podUID="ef39854c-4198-47ad-b6b2-760ebd9534f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 27 21:56:02.386855 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:02.386812    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rgmk6" podUID="ef39854c-4198-47ad-b6b2-760ebd9534f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 27 21:56:12.386490 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:12.386446    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rgmk6" podUID="ef39854c-4198-47ad-b6b2-760ebd9534f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 27 21:56:22.387271 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:22.387229    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rgmk6" podUID="ef39854c-4198-47ad-b6b2-760ebd9534f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 27 21:56:32.387083 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:32.387049    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rgmk6" podUID="ef39854c-4198-47ad-b6b2-760ebd9534f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 27 21:56:33.130832 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:33.130787    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rgmk6" podUID="ef39854c-4198-47ad-b6b2-760ebd9534f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 27 21:56:43.135348 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:43.135323    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rgmk6"
Jul 27 21:56:53.959127 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:53.959086    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rgmk6"]
Jul 27 21:56:53.959653 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:53.959362    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rgmk6" podUID="ef39854c-4198-47ad-b6b2-760ebd9534f8" containerName="kserve-container" containerID="cri-o://bb09b7ce76cfe839bc4e635bdc77731f5288d5116a5b39eecfc1f37b95b93490" gracePeriod=30
Jul 27 21:56:53.959789 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:53.959744    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rgmk6" podUID="ef39854c-4198-47ad-b6b2-760ebd9534f8" containerName="kube-rbac-proxy" containerID="cri-o://3c38a77ac373bc7d3b771c2a2d0775a7ecc04b741fa2bb909fb5ee74842e5a69" gracePeriod=30
Jul 27 21:56:53.990691 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:53.990649    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:56:54.722430 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:54.722401    2577 generic.go:356] "Generic (PLEG): container finished" podID="ef39854c-4198-47ad-b6b2-760ebd9534f8" containerID="3c38a77ac373bc7d3b771c2a2d0775a7ecc04b741fa2bb909fb5ee74842e5a69" exitCode=2
Jul 27 21:56:54.722614 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:54.722475    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rgmk6" event={"ID":"ef39854c-4198-47ad-b6b2-760ebd9534f8","Type":"ContainerDied","Data":"3c38a77ac373bc7d3b771c2a2d0775a7ecc04b741fa2bb909fb5ee74842e5a69"}
Jul 27 21:56:57.203417 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:57.203394    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rgmk6"
Jul 27 21:56:57.231422 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:57.231400    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef39854c-4198-47ad-b6b2-760ebd9534f8-kube-rbac-proxy-sar-config\") pod \"ef39854c-4198-47ad-b6b2-760ebd9534f8\" (UID: \"ef39854c-4198-47ad-b6b2-760ebd9534f8\") "
Jul 27 21:56:57.231548 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:57.231447    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef39854c-4198-47ad-b6b2-760ebd9534f8-kserve-provision-location\") pod \"ef39854c-4198-47ad-b6b2-760ebd9534f8\" (UID: \"ef39854c-4198-47ad-b6b2-760ebd9534f8\") "
Jul 27 21:56:57.231548 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:57.231483    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5pvn\" (UniqueName: \"kubernetes.io/projected/ef39854c-4198-47ad-b6b2-760ebd9534f8-kube-api-access-j5pvn\") pod \"ef39854c-4198-47ad-b6b2-760ebd9534f8\" (UID: \"ef39854c-4198-47ad-b6b2-760ebd9534f8\") "
Jul 27 21:56:57.231645 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:57.231610    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef39854c-4198-47ad-b6b2-760ebd9534f8-cabundle-cert\") pod \"ef39854c-4198-47ad-b6b2-760ebd9534f8\" (UID: \"ef39854c-4198-47ad-b6b2-760ebd9534f8\") "
Jul 27 21:56:57.231705 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:57.231691    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef39854c-4198-47ad-b6b2-760ebd9534f8-proxy-tls\") pod \"ef39854c-4198-47ad-b6b2-760ebd9534f8\" (UID: \"ef39854c-4198-47ad-b6b2-760ebd9534f8\") "
Jul 27 21:56:57.231804 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:57.231780    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef39854c-4198-47ad-b6b2-760ebd9534f8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ef39854c-4198-47ad-b6b2-760ebd9534f8" (UID: "ef39854c-4198-47ad-b6b2-760ebd9534f8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:56:57.231907 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:57.231879    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef39854c-4198-47ad-b6b2-760ebd9534f8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ef39854c-4198-47ad-b6b2-760ebd9534f8" (UID: "ef39854c-4198-47ad-b6b2-760ebd9534f8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:56:57.231998 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:57.231974    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef39854c-4198-47ad-b6b2-760ebd9534f8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ef39854c-4198-47ad-b6b2-760ebd9534f8" (UID: "ef39854c-4198-47ad-b6b2-760ebd9534f8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:56:57.232086 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:57.232067    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef39854c-4198-47ad-b6b2-760ebd9534f8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:56:57.232129 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:57.232092    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef39854c-4198-47ad-b6b2-760ebd9534f8-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:56:57.232129 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:57.232107    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef39854c-4198-47ad-b6b2-760ebd9534f8-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:56:57.233440 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:57.233414    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef39854c-4198-47ad-b6b2-760ebd9534f8-kube-api-access-j5pvn" (OuterVolumeSpecName: "kube-api-access-j5pvn") pod "ef39854c-4198-47ad-b6b2-760ebd9534f8" (UID: "ef39854c-4198-47ad-b6b2-760ebd9534f8"). InnerVolumeSpecName "kube-api-access-j5pvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:56:57.233584 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:57.233562    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef39854c-4198-47ad-b6b2-760ebd9534f8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ef39854c-4198-47ad-b6b2-760ebd9534f8" (UID: "ef39854c-4198-47ad-b6b2-760ebd9534f8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:56:57.332437 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:57.332387    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef39854c-4198-47ad-b6b2-760ebd9534f8-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:56:57.332437 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:57.332406    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5pvn\" (UniqueName: \"kubernetes.io/projected/ef39854c-4198-47ad-b6b2-760ebd9534f8-kube-api-access-j5pvn\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:56:57.736027 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:57.735995    2577 generic.go:356] "Generic (PLEG): container finished" podID="ef39854c-4198-47ad-b6b2-760ebd9534f8" containerID="bb09b7ce76cfe839bc4e635bdc77731f5288d5116a5b39eecfc1f37b95b93490" exitCode=0
Jul 27 21:56:57.736184 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:57.736058    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rgmk6"
Jul 27 21:56:57.736184 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:57.736080    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rgmk6" event={"ID":"ef39854c-4198-47ad-b6b2-760ebd9534f8","Type":"ContainerDied","Data":"bb09b7ce76cfe839bc4e635bdc77731f5288d5116a5b39eecfc1f37b95b93490"}
Jul 27 21:56:57.736184 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:57.736140    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rgmk6" event={"ID":"ef39854c-4198-47ad-b6b2-760ebd9534f8","Type":"ContainerDied","Data":"069c57205d26789eb9161a8348ceef484ecada458f307e0923ad07e57dc5e54a"}
Jul 27 21:56:57.736184 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:57.736166    2577 scope.go:117] "RemoveContainer" containerID="3c38a77ac373bc7d3b771c2a2d0775a7ecc04b741fa2bb909fb5ee74842e5a69"
Jul 27 21:56:57.745262 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:57.745246    2577 scope.go:117] "RemoveContainer" containerID="bb09b7ce76cfe839bc4e635bdc77731f5288d5116a5b39eecfc1f37b95b93490"
Jul 27 21:56:57.752372 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:57.752354    2577 scope.go:117] "RemoveContainer" containerID="7d4ef90573c5cb4e72c90e21ed38b6713c4eee942ea84eae077fd0dfb9eebc57"
Jul 27 21:56:57.759005 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:57.758986    2577 scope.go:117] "RemoveContainer" containerID="3c38a77ac373bc7d3b771c2a2d0775a7ecc04b741fa2bb909fb5ee74842e5a69"
Jul 27 21:56:57.759246 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:56:57.759228    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c38a77ac373bc7d3b771c2a2d0775a7ecc04b741fa2bb909fb5ee74842e5a69\": container with ID starting with 3c38a77ac373bc7d3b771c2a2d0775a7ecc04b741fa2bb909fb5ee74842e5a69 not found: ID does not exist" containerID="3c38a77ac373bc7d3b771c2a2d0775a7ecc04b741fa2bb909fb5ee74842e5a69"
Jul 27 21:56:57.759300 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:57.759252    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c38a77ac373bc7d3b771c2a2d0775a7ecc04b741fa2bb909fb5ee74842e5a69"} err="failed to get container status \"3c38a77ac373bc7d3b771c2a2d0775a7ecc04b741fa2bb909fb5ee74842e5a69\": rpc error: code = NotFound desc = could not find container \"3c38a77ac373bc7d3b771c2a2d0775a7ecc04b741fa2bb909fb5ee74842e5a69\": container with ID starting with 3c38a77ac373bc7d3b771c2a2d0775a7ecc04b741fa2bb909fb5ee74842e5a69 not found: ID does not exist"
Jul 27 21:56:57.759300 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:57.759270    2577 scope.go:117] "RemoveContainer" containerID="bb09b7ce76cfe839bc4e635bdc77731f5288d5116a5b39eecfc1f37b95b93490"
Jul 27 21:56:57.759534 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:56:57.759520    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb09b7ce76cfe839bc4e635bdc77731f5288d5116a5b39eecfc1f37b95b93490\": container with ID starting with bb09b7ce76cfe839bc4e635bdc77731f5288d5116a5b39eecfc1f37b95b93490 not found: ID does not exist" containerID="bb09b7ce76cfe839bc4e635bdc77731f5288d5116a5b39eecfc1f37b95b93490"
Jul 27 21:56:57.759586 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:57.759537    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb09b7ce76cfe839bc4e635bdc77731f5288d5116a5b39eecfc1f37b95b93490"} err="failed to get container status \"bb09b7ce76cfe839bc4e635bdc77731f5288d5116a5b39eecfc1f37b95b93490\": rpc error: code = NotFound desc = could not find container \"bb09b7ce76cfe839bc4e635bdc77731f5288d5116a5b39eecfc1f37b95b93490\": container with ID starting with bb09b7ce76cfe839bc4e635bdc77731f5288d5116a5b39eecfc1f37b95b93490 not found: ID does not exist"
Jul 27 21:56:57.759586 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:57.759548    2577 scope.go:117] "RemoveContainer" containerID="7d4ef90573c5cb4e72c90e21ed38b6713c4eee942ea84eae077fd0dfb9eebc57"
Jul 27 21:56:57.759755 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:56:57.759739    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d4ef90573c5cb4e72c90e21ed38b6713c4eee942ea84eae077fd0dfb9eebc57\": container with ID starting with 7d4ef90573c5cb4e72c90e21ed38b6713c4eee942ea84eae077fd0dfb9eebc57 not found: ID does not exist" containerID="7d4ef90573c5cb4e72c90e21ed38b6713c4eee942ea84eae077fd0dfb9eebc57"
Jul 27 21:56:57.759816 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:57.759774    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d4ef90573c5cb4e72c90e21ed38b6713c4eee942ea84eae077fd0dfb9eebc57"} err="failed to get container status \"7d4ef90573c5cb4e72c90e21ed38b6713c4eee942ea84eae077fd0dfb9eebc57\": rpc error: code = NotFound desc = could not find container \"7d4ef90573c5cb4e72c90e21ed38b6713c4eee942ea84eae077fd0dfb9eebc57\": container with ID starting with 7d4ef90573c5cb4e72c90e21ed38b6713c4eee942ea84eae077fd0dfb9eebc57 not found: ID does not exist"
Jul 27 21:56:57.766345 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:57.766325    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rgmk6"]
Jul 27 21:56:57.770344 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:57.770324    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-rgmk6"]
Jul 27 21:56:58.993808 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:58.993695    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:56:59.049479 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:59.049444    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ksxtn"]
Jul 27 21:56:59.049897 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:59.049879    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef39854c-4198-47ad-b6b2-760ebd9534f8" containerName="kube-rbac-proxy"
Jul 27 21:56:59.049953 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:59.049901    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef39854c-4198-47ad-b6b2-760ebd9534f8" containerName="kube-rbac-proxy"
Jul 27 21:56:59.049953 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:59.049915    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="108c9f59-3150-47d1-9b23-f7e06259a617" containerName="kserve-container"
Jul 27 21:56:59.049953 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:59.049924    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="108c9f59-3150-47d1-9b23-f7e06259a617" containerName="kserve-container"
Jul 27 21:56:59.049953 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:59.049935    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="108c9f59-3150-47d1-9b23-f7e06259a617" containerName="storage-initializer"
Jul 27 21:56:59.049953 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:59.049944    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="108c9f59-3150-47d1-9b23-f7e06259a617" containerName="storage-initializer"
Jul 27 21:56:59.050105 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:59.049965    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef39854c-4198-47ad-b6b2-760ebd9534f8" containerName="kserve-container"
Jul 27 21:56:59.050105 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:59.049974    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef39854c-4198-47ad-b6b2-760ebd9534f8" containerName="kserve-container"
Jul 27 21:56:59.050105 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:59.049985    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef39854c-4198-47ad-b6b2-760ebd9534f8" containerName="storage-initializer"
Jul 27 21:56:59.050105 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:59.049993    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef39854c-4198-47ad-b6b2-760ebd9534f8" containerName="storage-initializer"
Jul 27 21:56:59.050105 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:59.050012    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="108c9f59-3150-47d1-9b23-f7e06259a617" containerName="kube-rbac-proxy"
Jul 27 21:56:59.050105 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:59.050020    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="108c9f59-3150-47d1-9b23-f7e06259a617" containerName="kube-rbac-proxy"
Jul 27 21:56:59.050105 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:59.050078    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="108c9f59-3150-47d1-9b23-f7e06259a617" containerName="kube-rbac-proxy"
Jul 27 21:56:59.050105 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:59.050091    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="108c9f59-3150-47d1-9b23-f7e06259a617" containerName="kserve-container"
Jul 27 21:56:59.050105 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:59.050100    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef39854c-4198-47ad-b6b2-760ebd9534f8" containerName="kube-rbac-proxy"
Jul 27 21:56:59.050364 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:59.050110    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef39854c-4198-47ad-b6b2-760ebd9534f8" containerName="kserve-container"
Jul 27 21:56:59.054497 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:59.054483    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ksxtn"
Jul 27 21:56:59.058193 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:59.058172    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\""
Jul 27 21:56:59.058325 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:59.058195    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 21:56:59.058325 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:59.058226    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
Jul 27 21:56:59.058325 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:59.058228    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 21:56:59.058535 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:59.058518    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 21:56:59.058625 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:59.058610    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 21:56:59.058897 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:59.058883    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v7c66\""
Jul 27 21:56:59.064816 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:59.064783    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ksxtn"]
Jul 27 21:56:59.101686 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:59.101662    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:56:59.135941 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:59.135917    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef39854c-4198-47ad-b6b2-760ebd9534f8" path="/var/lib/kubelet/pods/ef39854c-4198-47ad-b6b2-760ebd9534f8/volumes"
Jul 27 21:56:59.143744 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:59.143723    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a567a0d8-1e44-4f25-b3c1-c8d334e3c894-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-ksxtn\" (UID: \"a567a0d8-1e44-4f25-b3c1-c8d334e3c894\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ksxtn"
Jul 27 21:56:59.143848 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:59.143751    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a567a0d8-1e44-4f25-b3c1-c8d334e3c894-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-ksxtn\" (UID: \"a567a0d8-1e44-4f25-b3c1-c8d334e3c894\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ksxtn"
Jul 27 21:56:59.143904 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:59.143865    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a567a0d8-1e44-4f25-b3c1-c8d334e3c894-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-ksxtn\" (UID: \"a567a0d8-1e44-4f25-b3c1-c8d334e3c894\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ksxtn"
Jul 27 21:56:59.143964 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:59.143943    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdx64\" (UniqueName: \"kubernetes.io/projected/a567a0d8-1e44-4f25-b3c1-c8d334e3c894-kube-api-access-xdx64\") pod \"isvc-pmml-runtime-predictor-885fc8777-ksxtn\" (UID: \"a567a0d8-1e44-4f25-b3c1-c8d334e3c894\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ksxtn"
Jul 27 21:56:59.144017 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:59.143986    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a567a0d8-1e44-4f25-b3c1-c8d334e3c894-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-ksxtn\" (UID: \"a567a0d8-1e44-4f25-b3c1-c8d334e3c894\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ksxtn"
Jul 27 21:56:59.244402 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:59.244319    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a567a0d8-1e44-4f25-b3c1-c8d334e3c894-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-ksxtn\" (UID: \"a567a0d8-1e44-4f25-b3c1-c8d334e3c894\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ksxtn"
Jul 27 21:56:59.244402 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:59.244365    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdx64\" (UniqueName: \"kubernetes.io/projected/a567a0d8-1e44-4f25-b3c1-c8d334e3c894-kube-api-access-xdx64\") pod \"isvc-pmml-runtime-predictor-885fc8777-ksxtn\" (UID: \"a567a0d8-1e44-4f25-b3c1-c8d334e3c894\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ksxtn"
Jul 27 21:56:59.244630 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:59.244495    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a567a0d8-1e44-4f25-b3c1-c8d334e3c894-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-ksxtn\" (UID: \"a567a0d8-1e44-4f25-b3c1-c8d334e3c894\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ksxtn"
Jul 27 21:56:59.244630 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:59.244544    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a567a0d8-1e44-4f25-b3c1-c8d334e3c894-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-ksxtn\" (UID: \"a567a0d8-1e44-4f25-b3c1-c8d334e3c894\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ksxtn"
Jul 27 21:56:59.244630 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:59.244569    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a567a0d8-1e44-4f25-b3c1-c8d334e3c894-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-ksxtn\" (UID: \"a567a0d8-1e44-4f25-b3c1-c8d334e3c894\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ksxtn"
Jul 27 21:56:59.244995 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:59.244976    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a567a0d8-1e44-4f25-b3c1-c8d334e3c894-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-ksxtn\" (UID: \"a567a0d8-1e44-4f25-b3c1-c8d334e3c894\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ksxtn"
Jul 27 21:56:59.244995 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:59.244987    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a567a0d8-1e44-4f25-b3c1-c8d334e3c894-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-ksxtn\" (UID: \"a567a0d8-1e44-4f25-b3c1-c8d334e3c894\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ksxtn"
Jul 27 21:56:59.245117 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:59.245076    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a567a0d8-1e44-4f25-b3c1-c8d334e3c894-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-ksxtn\" (UID: \"a567a0d8-1e44-4f25-b3c1-c8d334e3c894\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ksxtn"
Jul 27 21:56:59.246753 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:59.246737    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a567a0d8-1e44-4f25-b3c1-c8d334e3c894-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-ksxtn\" (UID: \"a567a0d8-1e44-4f25-b3c1-c8d334e3c894\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ksxtn"
Jul 27 21:56:59.252456 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:59.252435    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdx64\" (UniqueName: \"kubernetes.io/projected/a567a0d8-1e44-4f25-b3c1-c8d334e3c894-kube-api-access-xdx64\") pod \"isvc-pmml-runtime-predictor-885fc8777-ksxtn\" (UID: \"a567a0d8-1e44-4f25-b3c1-c8d334e3c894\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ksxtn"
Jul 27 21:56:59.364852 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:59.364817    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ksxtn"
Jul 27 21:56:59.487607 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:59.487489    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ksxtn"]
Jul 27 21:56:59.490130 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:56:59.490100    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda567a0d8_1e44_4f25_b3c1_c8d334e3c894.slice/crio-670a50d60ec78d585f05cb2360ae161f6135954bc1bdd191c555652850496228 WatchSource:0}: Error finding container 670a50d60ec78d585f05cb2360ae161f6135954bc1bdd191c555652850496228: Status 404 returned error can't find the container with id 670a50d60ec78d585f05cb2360ae161f6135954bc1bdd191c555652850496228
Jul 27 21:56:59.746613 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:59.746577    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ksxtn" event={"ID":"a567a0d8-1e44-4f25-b3c1-c8d334e3c894","Type":"ContainerStarted","Data":"6bc36200555fdc840c89aa446dcddda6184fed7599c29f1b3feae7dbe8faa103"}
Jul 27 21:56:59.746799 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:56:59.746647    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ksxtn" event={"ID":"a567a0d8-1e44-4f25-b3c1-c8d334e3c894","Type":"ContainerStarted","Data":"670a50d60ec78d585f05cb2360ae161f6135954bc1bdd191c555652850496228"}
Jul 27 21:57:03.763403 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:57:03.763369    2577 generic.go:356] "Generic (PLEG): container finished" podID="a567a0d8-1e44-4f25-b3c1-c8d334e3c894" containerID="6bc36200555fdc840c89aa446dcddda6184fed7599c29f1b3feae7dbe8faa103" exitCode=0
Jul 27 21:57:03.763916 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:57:03.763448    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ksxtn" event={"ID":"a567a0d8-1e44-4f25-b3c1-c8d334e3c894","Type":"ContainerDied","Data":"6bc36200555fdc840c89aa446dcddda6184fed7599c29f1b3feae7dbe8faa103"}
Jul 27 21:57:04.769139 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:57:04.769100    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ksxtn" event={"ID":"a567a0d8-1e44-4f25-b3c1-c8d334e3c894","Type":"ContainerStarted","Data":"debe1809450fe05140055876aaf2c8907aba63d3feebc841154afbad64641d82"}
Jul 27 21:57:04.769519 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:57:04.769162    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ksxtn" event={"ID":"a567a0d8-1e44-4f25-b3c1-c8d334e3c894","Type":"ContainerStarted","Data":"d31cfa543a5507c3553785ef61a4b390c5de29bd742b9371b5349bb735cb4f96"}
Jul 27 21:57:04.769519 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:57:04.769368    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ksxtn"
Jul 27 21:57:04.788054 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:57:04.788006    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ksxtn" podStartSLOduration=2.124855179 podStartE2EDuration="5.787993129s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.66313795s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:56:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:57:04.786662401 +0000 UTC m=+3148.153852268" watchObservedRunningTime="2026-07-27 21:57:04.787993129 +0000 UTC m=+3148.155183058"
Jul 27 21:57:05.773035 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:57:05.772999    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ksxtn"
Jul 27 21:57:05.774244 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:57:05.774217    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ksxtn" podUID="a567a0d8-1e44-4f25-b3c1-c8d334e3c894" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 27 21:57:06.776917 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:57:06.776878    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ksxtn" podUID="a567a0d8-1e44-4f25-b3c1-c8d334e3c894" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 27 21:57:11.782477 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:57:11.782452    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ksxtn"
Jul 27 21:57:11.783011 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:57:11.782983    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ksxtn" podUID="a567a0d8-1e44-4f25-b3c1-c8d334e3c894" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 27 21:57:21.783020 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:57:21.782978    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ksxtn" podUID="a567a0d8-1e44-4f25-b3c1-c8d334e3c894" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 27 21:57:31.783885 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:57:31.783845    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ksxtn" podUID="a567a0d8-1e44-4f25-b3c1-c8d334e3c894" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 27 21:57:41.783939 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:57:41.783898    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ksxtn" podUID="a567a0d8-1e44-4f25-b3c1-c8d334e3c894" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 27 21:57:51.783538 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:57:51.783501    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ksxtn" podUID="a567a0d8-1e44-4f25-b3c1-c8d334e3c894" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 27 21:58:01.783127 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:01.783084    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ksxtn" podUID="a567a0d8-1e44-4f25-b3c1-c8d334e3c894" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 27 21:58:11.783609 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:11.783570    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ksxtn" podUID="a567a0d8-1e44-4f25-b3c1-c8d334e3c894" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 27 21:58:21.783704 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:21.783677    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ksxtn"
Jul 27 21:58:30.045455 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:30.045373    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ksxtn"]
Jul 27 21:58:30.045875 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:30.045692    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ksxtn" podUID="a567a0d8-1e44-4f25-b3c1-c8d334e3c894" containerName="kserve-container" containerID="cri-o://d31cfa543a5507c3553785ef61a4b390c5de29bd742b9371b5349bb735cb4f96" gracePeriod=30
Jul 27 21:58:30.045875 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:30.045724    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ksxtn" podUID="a567a0d8-1e44-4f25-b3c1-c8d334e3c894" containerName="kube-rbac-proxy" containerID="cri-o://debe1809450fe05140055876aaf2c8907aba63d3feebc841154afbad64641d82" gracePeriod=30
Jul 27 21:58:30.136952 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:30.136915    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:58:31.104818 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:31.104787    2577 generic.go:356] "Generic (PLEG): container finished" podID="a567a0d8-1e44-4f25-b3c1-c8d334e3c894" containerID="debe1809450fe05140055876aaf2c8907aba63d3feebc841154afbad64641d82" exitCode=2
Jul 27 21:58:31.105214 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:31.104885    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ksxtn" event={"ID":"a567a0d8-1e44-4f25-b3c1-c8d334e3c894","Type":"ContainerDied","Data":"debe1809450fe05140055876aaf2c8907aba63d3feebc841154afbad64641d82"}
Jul 27 21:58:31.777400 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:31.777357    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ksxtn" podUID="a567a0d8-1e44-4f25-b3c1-c8d334e3c894" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.47:8643/healthz\": dial tcp 10.133.0.47:8643: connect: connection refused"
Jul 27 21:58:31.783558 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:31.783527    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ksxtn" podUID="a567a0d8-1e44-4f25-b3c1-c8d334e3c894" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 27 21:58:33.283091 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:33.283068    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ksxtn"
Jul 27 21:58:33.387199 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:33.387137    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a567a0d8-1e44-4f25-b3c1-c8d334e3c894-kserve-provision-location\") pod \"a567a0d8-1e44-4f25-b3c1-c8d334e3c894\" (UID: \"a567a0d8-1e44-4f25-b3c1-c8d334e3c894\") "
Jul 27 21:58:33.387199 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:33.387182    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a567a0d8-1e44-4f25-b3c1-c8d334e3c894-proxy-tls\") pod \"a567a0d8-1e44-4f25-b3c1-c8d334e3c894\" (UID: \"a567a0d8-1e44-4f25-b3c1-c8d334e3c894\") "
Jul 27 21:58:33.387385 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:33.387229    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a567a0d8-1e44-4f25-b3c1-c8d334e3c894-cabundle-cert\") pod \"a567a0d8-1e44-4f25-b3c1-c8d334e3c894\" (UID: \"a567a0d8-1e44-4f25-b3c1-c8d334e3c894\") "
Jul 27 21:58:33.387385 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:33.387265    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a567a0d8-1e44-4f25-b3c1-c8d334e3c894-kube-rbac-proxy-sar-config\") pod \"a567a0d8-1e44-4f25-b3c1-c8d334e3c894\" (UID: \"a567a0d8-1e44-4f25-b3c1-c8d334e3c894\") "
Jul 27 21:58:33.387385 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:33.387289    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xdx64\" (UniqueName: \"kubernetes.io/projected/a567a0d8-1e44-4f25-b3c1-c8d334e3c894-kube-api-access-xdx64\") pod \"a567a0d8-1e44-4f25-b3c1-c8d334e3c894\" (UID: \"a567a0d8-1e44-4f25-b3c1-c8d334e3c894\") "
Jul 27 21:58:33.387561 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:33.387409    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a567a0d8-1e44-4f25-b3c1-c8d334e3c894-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a567a0d8-1e44-4f25-b3c1-c8d334e3c894" (UID: "a567a0d8-1e44-4f25-b3c1-c8d334e3c894"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:58:33.387625 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:33.387581    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a567a0d8-1e44-4f25-b3c1-c8d334e3c894-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a567a0d8-1e44-4f25-b3c1-c8d334e3c894" (UID: "a567a0d8-1e44-4f25-b3c1-c8d334e3c894"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:58:33.387625 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:33.387585    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a567a0d8-1e44-4f25-b3c1-c8d334e3c894-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:58:33.387705 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:33.387634    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a567a0d8-1e44-4f25-b3c1-c8d334e3c894-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a567a0d8-1e44-4f25-b3c1-c8d334e3c894" (UID: "a567a0d8-1e44-4f25-b3c1-c8d334e3c894"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:58:33.389153 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:33.389129    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a567a0d8-1e44-4f25-b3c1-c8d334e3c894-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a567a0d8-1e44-4f25-b3c1-c8d334e3c894" (UID: "a567a0d8-1e44-4f25-b3c1-c8d334e3c894"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:58:33.389268 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:33.389249    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a567a0d8-1e44-4f25-b3c1-c8d334e3c894-kube-api-access-xdx64" (OuterVolumeSpecName: "kube-api-access-xdx64") pod "a567a0d8-1e44-4f25-b3c1-c8d334e3c894" (UID: "a567a0d8-1e44-4f25-b3c1-c8d334e3c894"). InnerVolumeSpecName "kube-api-access-xdx64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:58:33.488025 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:33.487992    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a567a0d8-1e44-4f25-b3c1-c8d334e3c894-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:58:33.488025 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:33.488022    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a567a0d8-1e44-4f25-b3c1-c8d334e3c894-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:58:33.488025 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:33.488033    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdx64\" (UniqueName: \"kubernetes.io/projected/a567a0d8-1e44-4f25-b3c1-c8d334e3c894-kube-api-access-xdx64\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:58:33.488197 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:33.488043    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a567a0d8-1e44-4f25-b3c1-c8d334e3c894-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 21:58:34.117884 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:34.117853    2577 generic.go:356] "Generic (PLEG): container finished" podID="a567a0d8-1e44-4f25-b3c1-c8d334e3c894" containerID="d31cfa543a5507c3553785ef61a4b390c5de29bd742b9371b5349bb735cb4f96" exitCode=0
Jul 27 21:58:34.118054 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:34.117902    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ksxtn" event={"ID":"a567a0d8-1e44-4f25-b3c1-c8d334e3c894","Type":"ContainerDied","Data":"d31cfa543a5507c3553785ef61a4b390c5de29bd742b9371b5349bb735cb4f96"}
Jul 27 21:58:34.118054 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:34.117910    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ksxtn"
Jul 27 21:58:34.118054 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:34.117973    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ksxtn" event={"ID":"a567a0d8-1e44-4f25-b3c1-c8d334e3c894","Type":"ContainerDied","Data":"670a50d60ec78d585f05cb2360ae161f6135954bc1bdd191c555652850496228"}
Jul 27 21:58:34.118054 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:34.117999    2577 scope.go:117] "RemoveContainer" containerID="debe1809450fe05140055876aaf2c8907aba63d3feebc841154afbad64641d82"
Jul 27 21:58:34.126441 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:34.126420    2577 scope.go:117] "RemoveContainer" containerID="d31cfa543a5507c3553785ef61a4b390c5de29bd742b9371b5349bb735cb4f96"
Jul 27 21:58:34.133301 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:34.133284    2577 scope.go:117] "RemoveContainer" containerID="6bc36200555fdc840c89aa446dcddda6184fed7599c29f1b3feae7dbe8faa103"
Jul 27 21:58:34.140023 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:34.140004    2577 scope.go:117] "RemoveContainer" containerID="debe1809450fe05140055876aaf2c8907aba63d3feebc841154afbad64641d82"
Jul 27 21:58:34.140281 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:58:34.140254    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"debe1809450fe05140055876aaf2c8907aba63d3feebc841154afbad64641d82\": container with ID starting with debe1809450fe05140055876aaf2c8907aba63d3feebc841154afbad64641d82 not found: ID does not exist" containerID="debe1809450fe05140055876aaf2c8907aba63d3feebc841154afbad64641d82"
Jul 27 21:58:34.140360 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:34.140285    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"debe1809450fe05140055876aaf2c8907aba63d3feebc841154afbad64641d82"} err="failed to get container status \"debe1809450fe05140055876aaf2c8907aba63d3feebc841154afbad64641d82\": rpc error: code = NotFound desc = could not find container \"debe1809450fe05140055876aaf2c8907aba63d3feebc841154afbad64641d82\": container with ID starting with debe1809450fe05140055876aaf2c8907aba63d3feebc841154afbad64641d82 not found: ID does not exist"
Jul 27 21:58:34.140360 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:34.140308    2577 scope.go:117] "RemoveContainer" containerID="d31cfa543a5507c3553785ef61a4b390c5de29bd742b9371b5349bb735cb4f96"
Jul 27 21:58:34.140535 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:58:34.140514    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d31cfa543a5507c3553785ef61a4b390c5de29bd742b9371b5349bb735cb4f96\": container with ID starting with d31cfa543a5507c3553785ef61a4b390c5de29bd742b9371b5349bb735cb4f96 not found: ID does not exist" containerID="d31cfa543a5507c3553785ef61a4b390c5de29bd742b9371b5349bb735cb4f96"
Jul 27 21:58:34.140620 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:34.140540    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d31cfa543a5507c3553785ef61a4b390c5de29bd742b9371b5349bb735cb4f96"} err="failed to get container status \"d31cfa543a5507c3553785ef61a4b390c5de29bd742b9371b5349bb735cb4f96\": rpc error: code = NotFound desc = could not find container \"d31cfa543a5507c3553785ef61a4b390c5de29bd742b9371b5349bb735cb4f96\": container with ID starting with d31cfa543a5507c3553785ef61a4b390c5de29bd742b9371b5349bb735cb4f96 not found: ID does not exist"
Jul 27 21:58:34.140620 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:34.140555    2577 scope.go:117] "RemoveContainer" containerID="6bc36200555fdc840c89aa446dcddda6184fed7599c29f1b3feae7dbe8faa103"
Jul 27 21:58:34.140741 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:58:34.140727    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bc36200555fdc840c89aa446dcddda6184fed7599c29f1b3feae7dbe8faa103\": container with ID starting with 6bc36200555fdc840c89aa446dcddda6184fed7599c29f1b3feae7dbe8faa103 not found: ID does not exist" containerID="6bc36200555fdc840c89aa446dcddda6184fed7599c29f1b3feae7dbe8faa103"
Jul 27 21:58:34.140936 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:34.140744    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bc36200555fdc840c89aa446dcddda6184fed7599c29f1b3feae7dbe8faa103"} err="failed to get container status \"6bc36200555fdc840c89aa446dcddda6184fed7599c29f1b3feae7dbe8faa103\": rpc error: code = NotFound desc = could not find container \"6bc36200555fdc840c89aa446dcddda6184fed7599c29f1b3feae7dbe8faa103\": container with ID starting with 6bc36200555fdc840c89aa446dcddda6184fed7599c29f1b3feae7dbe8faa103 not found: ID does not exist"
Jul 27 21:58:34.143362 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:34.143343    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ksxtn"]
Jul 27 21:58:34.147450 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:34.147430    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-ksxtn"]
Jul 27 21:58:35.096436 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:35.096396    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:58:35.134888 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:35.134850    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a567a0d8-1e44-4f25-b3c1-c8d334e3c894" path="/var/lib/kubelet/pods/a567a0d8-1e44-4f25-b3c1-c8d334e3c894/volumes"
Jul 27 21:58:35.162528 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:35.162502    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc"]
Jul 27 21:58:35.162856 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:35.162840    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a567a0d8-1e44-4f25-b3c1-c8d334e3c894" containerName="kube-rbac-proxy"
Jul 27 21:58:35.162946 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:35.162859    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a567a0d8-1e44-4f25-b3c1-c8d334e3c894" containerName="kube-rbac-proxy"
Jul 27 21:58:35.162946 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:35.162886    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a567a0d8-1e44-4f25-b3c1-c8d334e3c894" containerName="storage-initializer"
Jul 27 21:58:35.162946 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:35.162895    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a567a0d8-1e44-4f25-b3c1-c8d334e3c894" containerName="storage-initializer"
Jul 27 21:58:35.162946 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:35.162909    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a567a0d8-1e44-4f25-b3c1-c8d334e3c894" containerName="kserve-container"
Jul 27 21:58:35.162946 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:35.162917    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a567a0d8-1e44-4f25-b3c1-c8d334e3c894" containerName="kserve-container"
Jul 27 21:58:35.163213 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:35.162978    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a567a0d8-1e44-4f25-b3c1-c8d334e3c894" containerName="kserve-container"
Jul 27 21:58:35.163213 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:35.162989    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a567a0d8-1e44-4f25-b3c1-c8d334e3c894" containerName="kube-rbac-proxy"
Jul 27 21:58:35.167689 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:35.167668    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc"
Jul 27 21:58:35.171176 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:35.171156    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 21:58:35.171333 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:35.171155    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 21:58:35.171484 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:35.171444    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
Jul 27 21:58:35.172254 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:35.172225    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v7c66\""
Jul 27 21:58:35.172254 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:35.172239    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 21:58:35.172254 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:35.172230    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 21:58:35.172590 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:35.172553    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
Jul 27 21:58:35.174836 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:35.174814    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc"]
Jul 27 21:58:35.301556 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:35.301529    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc\" (UID: \"2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc"
Jul 27 21:58:35.301746 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:35.301563    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc\" (UID: \"2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc"
Jul 27 21:58:35.301746 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:35.301624    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc\" (UID: \"2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc"
Jul 27 21:58:35.301746 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:35.301659    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6g69\" (UniqueName: \"kubernetes.io/projected/2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0-kube-api-access-b6g69\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc\" (UID: \"2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc"
Jul 27 21:58:35.301746 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:35.301714    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc\" (UID: \"2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc"
Jul 27 21:58:35.402814 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:35.402729    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc\" (UID: \"2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc"
Jul 27 21:58:35.402941 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:35.402826    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc\" (UID: \"2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc"
Jul 27 21:58:35.402941 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:35.402855    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc\" (UID: \"2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc"
Jul 27 21:58:35.402941 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:35.402889    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc\" (UID: \"2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc"
Jul 27 21:58:35.403105 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:35.402945    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6g69\" (UniqueName: \"kubernetes.io/projected/2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0-kube-api-access-b6g69\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc\" (UID: \"2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc"
Jul 27 21:58:35.403105 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:58:35.402967    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-serving-cert: secret "isvc-pmml-v2-kserve-predictor-serving-cert" not found
Jul 27 21:58:35.403105 ip-10-0-142-219 kubenswrapper[2577]: E0727 21:58:35.403022    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0-proxy-tls podName:2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0 nodeName:}" failed. No retries permitted until 2026-07-27 21:58:35.903006187 +0000 UTC m=+3239.270196033 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0-proxy-tls") pod "isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc" (UID: "2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0") : secret "isvc-pmml-v2-kserve-predictor-serving-cert" not found
Jul 27 21:58:35.403105 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:35.403066    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc\" (UID: \"2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc"
Jul 27 21:58:35.403509 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:35.403489    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc\" (UID: \"2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc"
Jul 27 21:58:35.403576 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:35.403531    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc\" (UID: \"2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc"
Jul 27 21:58:35.413484 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:35.413460    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6g69\" (UniqueName: \"kubernetes.io/projected/2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0-kube-api-access-b6g69\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc\" (UID: \"2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc"
Jul 27 21:58:35.649420 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:35.649376    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 21:58:35.907046 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:35.907003    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc\" (UID: \"2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc"
Jul 27 21:58:35.909319 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:35.909301    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc\" (UID: \"2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc"
Jul 27 21:58:36.078780 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:36.078723    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc"
Jul 27 21:58:36.209680 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:36.209633    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc"]
Jul 27 21:58:36.211804 ip-10-0-142-219 kubenswrapper[2577]: W0727 21:58:36.211776    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c5fe8b5_6f1e_482e_bbeb_03ccc03ee6b0.slice/crio-21d87c83ff88044f877dc2378dc46606bb7b5a500e508389f3564c58df6f1b86 WatchSource:0}: Error finding container 21d87c83ff88044f877dc2378dc46606bb7b5a500e508389f3564c58df6f1b86: Status 404 returned error can't find the container with id 21d87c83ff88044f877dc2378dc46606bb7b5a500e508389f3564c58df6f1b86
Jul 27 21:58:37.140009 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:37.139965    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc" event={"ID":"2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0","Type":"ContainerStarted","Data":"7c2e0d23ae2155b6acbf2ee7a5edddbb123f683c5fdfae12bbd53e7c18b0876a"}
Jul 27 21:58:37.140172 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:37.140028    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc" event={"ID":"2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0","Type":"ContainerStarted","Data":"21d87c83ff88044f877dc2378dc46606bb7b5a500e508389f3564c58df6f1b86"}
Jul 27 21:58:40.153407 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:40.153374    2577 generic.go:356] "Generic (PLEG): container finished" podID="2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0" containerID="7c2e0d23ae2155b6acbf2ee7a5edddbb123f683c5fdfae12bbd53e7c18b0876a" exitCode=0
Jul 27 21:58:40.153819 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:40.153428    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc" event={"ID":"2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0","Type":"ContainerDied","Data":"7c2e0d23ae2155b6acbf2ee7a5edddbb123f683c5fdfae12bbd53e7c18b0876a"}
Jul 27 21:58:41.158943 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:41.158911    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc" event={"ID":"2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0","Type":"ContainerStarted","Data":"7986e3cd9992ec7f19f633ed2ce6a71f90140c64c43330dd973b6e265791cbc8"}
Jul 27 21:58:41.159314 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:41.158968    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc" event={"ID":"2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0","Type":"ContainerStarted","Data":"453bb04dea214eb087d867958dd40d8571f7f0aaf38547235e12ef008964b2a9"}
Jul 27 21:58:41.159314 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:41.159149    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc"
Jul 27 21:58:41.179251 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:41.179209    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc" podStartSLOduration=2.450721687 podStartE2EDuration="6.179196473s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.728474786s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:58:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:58:41.177572259 +0000 UTC m=+3244.544762126" watchObservedRunningTime="2026-07-27 21:58:41.179196473 +0000 UTC m=+3244.546386338"
Jul 27 21:58:42.163455 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:42.163422    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc"
Jul 27 21:58:42.164649 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:42.164621    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc" podUID="2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Jul 27 21:58:43.166859 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:43.166815    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc" podUID="2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Jul 27 21:58:48.171852 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:48.171824    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc"
Jul 27 21:58:48.172336 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:48.172309    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc" podUID="2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Jul 27 21:58:58.172744 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:58:58.172706    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc" podUID="2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Jul 27 21:59:08.172509 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:59:08.172467    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc" podUID="2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Jul 27 21:59:18.172385 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:59:18.172345    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc" podUID="2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Jul 27 21:59:28.172774 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:59:28.172726    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc" podUID="2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Jul 27 21:59:38.172724 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:59:38.172687    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc" podUID="2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Jul 27 21:59:48.172593 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:59:48.172557    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc" podUID="2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Jul 27 21:59:51.130561 ip-10-0-142-219 kubenswrapper[2577]: I0727 21:59:51.130523    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc" podUID="2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Jul 27 22:00:01.135240 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:01.135150    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc"
Jul 27 22:00:06.308631 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:06.308598    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc"]
Jul 27 22:00:06.309067 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:06.308845    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc" podUID="2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0" containerName="kserve-container" containerID="cri-o://453bb04dea214eb087d867958dd40d8571f7f0aaf38547235e12ef008964b2a9" gracePeriod=30
Jul 27 22:00:06.309067 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:06.308896    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc" podUID="2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0" containerName="kube-rbac-proxy" containerID="cri-o://7986e3cd9992ec7f19f633ed2ce6a71f90140c64c43330dd973b6e265791cbc8" gracePeriod=30
Jul 27 22:00:06.503252 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:06.503219    2577 generic.go:356] "Generic (PLEG): container finished" podID="2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0" containerID="7986e3cd9992ec7f19f633ed2ce6a71f90140c64c43330dd973b6e265791cbc8" exitCode=2
Jul 27 22:00:06.503426 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:06.503277    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc" event={"ID":"2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0","Type":"ContainerDied","Data":"7986e3cd9992ec7f19f633ed2ce6a71f90140c64c43330dd973b6e265791cbc8"}
Jul 27 22:00:08.167404 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:08.167357    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc" podUID="2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.48:8643/healthz\": dial tcp 10.133.0.48:8643: connect: connection refused"
Jul 27 22:00:08.336890 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:08.336856    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:00:10.151620 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:10.151593    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc"
Jul 27 22:00:10.267061 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:10.267023    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0-cabundle-cert\") pod \"2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0\" (UID: \"2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0\") "
Jul 27 22:00:10.267246 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:10.267115    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0-proxy-tls\") pod \"2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0\" (UID: \"2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0\") "
Jul 27 22:00:10.267246 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:10.267153    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6g69\" (UniqueName: \"kubernetes.io/projected/2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0-kube-api-access-b6g69\") pod \"2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0\" (UID: \"2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0\") "
Jul 27 22:00:10.267246 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:10.267178    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0-kserve-provision-location\") pod \"2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0\" (UID: \"2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0\") "
Jul 27 22:00:10.267246 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:10.267202    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0-kube-rbac-proxy-sar-config\") pod \"2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0\" (UID: \"2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0\") "
Jul 27 22:00:10.267531 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:10.267487    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0" (UID: "2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:00:10.267531 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:10.267515    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0" (UID: "2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:00:10.267702 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:10.267631    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0" (UID: "2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:00:10.269328 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:10.269306    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0" (UID: "2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 22:00:10.269414 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:10.269329    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0-kube-api-access-b6g69" (OuterVolumeSpecName: "kube-api-access-b6g69") pod "2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0" (UID: "2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0"). InnerVolumeSpecName "kube-api-access-b6g69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:00:10.368543 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:10.368506    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:00:10.368543 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:10.368539    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6g69\" (UniqueName: \"kubernetes.io/projected/2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0-kube-api-access-b6g69\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:00:10.368543 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:10.368552    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:00:10.368818 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:10.368566    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:00:10.368818 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:10.368577    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:00:10.526581 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:10.526496    2577 generic.go:356] "Generic (PLEG): container finished" podID="2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0" containerID="453bb04dea214eb087d867958dd40d8571f7f0aaf38547235e12ef008964b2a9" exitCode=0
Jul 27 22:00:10.526581 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:10.526544    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc"
Jul 27 22:00:10.526581 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:10.526567    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc" event={"ID":"2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0","Type":"ContainerDied","Data":"453bb04dea214eb087d867958dd40d8571f7f0aaf38547235e12ef008964b2a9"}
Jul 27 22:00:10.526845 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:10.526621    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc" event={"ID":"2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0","Type":"ContainerDied","Data":"21d87c83ff88044f877dc2378dc46606bb7b5a500e508389f3564c58df6f1b86"}
Jul 27 22:00:10.526845 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:10.526649    2577 scope.go:117] "RemoveContainer" containerID="7986e3cd9992ec7f19f633ed2ce6a71f90140c64c43330dd973b6e265791cbc8"
Jul 27 22:00:10.536135 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:10.535678    2577 scope.go:117] "RemoveContainer" containerID="453bb04dea214eb087d867958dd40d8571f7f0aaf38547235e12ef008964b2a9"
Jul 27 22:00:10.543491 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:10.543475    2577 scope.go:117] "RemoveContainer" containerID="7c2e0d23ae2155b6acbf2ee7a5edddbb123f683c5fdfae12bbd53e7c18b0876a"
Jul 27 22:00:10.549369 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:10.549341    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc"]
Jul 27 22:00:10.550745 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:10.550727    2577 scope.go:117] "RemoveContainer" containerID="7986e3cd9992ec7f19f633ed2ce6a71f90140c64c43330dd973b6e265791cbc8"
Jul 27 22:00:10.551049 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:00:10.551022    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7986e3cd9992ec7f19f633ed2ce6a71f90140c64c43330dd973b6e265791cbc8\": container with ID starting with 7986e3cd9992ec7f19f633ed2ce6a71f90140c64c43330dd973b6e265791cbc8 not found: ID does not exist" containerID="7986e3cd9992ec7f19f633ed2ce6a71f90140c64c43330dd973b6e265791cbc8"
Jul 27 22:00:10.551152 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:10.551058    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7986e3cd9992ec7f19f633ed2ce6a71f90140c64c43330dd973b6e265791cbc8"} err="failed to get container status \"7986e3cd9992ec7f19f633ed2ce6a71f90140c64c43330dd973b6e265791cbc8\": rpc error: code = NotFound desc = could not find container \"7986e3cd9992ec7f19f633ed2ce6a71f90140c64c43330dd973b6e265791cbc8\": container with ID starting with 7986e3cd9992ec7f19f633ed2ce6a71f90140c64c43330dd973b6e265791cbc8 not found: ID does not exist"
Jul 27 22:00:10.551152 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:10.551121    2577 scope.go:117] "RemoveContainer" containerID="453bb04dea214eb087d867958dd40d8571f7f0aaf38547235e12ef008964b2a9"
Jul 27 22:00:10.551548 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:00:10.551517    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"453bb04dea214eb087d867958dd40d8571f7f0aaf38547235e12ef008964b2a9\": container with ID starting with 453bb04dea214eb087d867958dd40d8571f7f0aaf38547235e12ef008964b2a9 not found: ID does not exist" containerID="453bb04dea214eb087d867958dd40d8571f7f0aaf38547235e12ef008964b2a9"
Jul 27 22:00:10.551636 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:10.551551    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"453bb04dea214eb087d867958dd40d8571f7f0aaf38547235e12ef008964b2a9"} err="failed to get container status \"453bb04dea214eb087d867958dd40d8571f7f0aaf38547235e12ef008964b2a9\": rpc error: code = NotFound desc = could not find container \"453bb04dea214eb087d867958dd40d8571f7f0aaf38547235e12ef008964b2a9\": container with ID starting with 453bb04dea214eb087d867958dd40d8571f7f0aaf38547235e12ef008964b2a9 not found: ID does not exist"
Jul 27 22:00:10.551636 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:10.551573    2577 scope.go:117] "RemoveContainer" containerID="7c2e0d23ae2155b6acbf2ee7a5edddbb123f683c5fdfae12bbd53e7c18b0876a"
Jul 27 22:00:10.552011 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:00:10.551988    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c2e0d23ae2155b6acbf2ee7a5edddbb123f683c5fdfae12bbd53e7c18b0876a\": container with ID starting with 7c2e0d23ae2155b6acbf2ee7a5edddbb123f683c5fdfae12bbd53e7c18b0876a not found: ID does not exist" containerID="7c2e0d23ae2155b6acbf2ee7a5edddbb123f683c5fdfae12bbd53e7c18b0876a"
Jul 27 22:00:10.552090 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:10.552017    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c2e0d23ae2155b6acbf2ee7a5edddbb123f683c5fdfae12bbd53e7c18b0876a"} err="failed to get container status \"7c2e0d23ae2155b6acbf2ee7a5edddbb123f683c5fdfae12bbd53e7c18b0876a\": rpc error: code = NotFound desc = could not find container \"7c2e0d23ae2155b6acbf2ee7a5edddbb123f683c5fdfae12bbd53e7c18b0876a\": container with ID starting with 7c2e0d23ae2155b6acbf2ee7a5edddbb123f683c5fdfae12bbd53e7c18b0876a not found: ID does not exist"
Jul 27 22:00:10.553118 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:10.553095    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-wtzgc"]
Jul 27 22:00:11.135597 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:11.135565    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0" path="/var/lib/kubelet/pods/2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0/volumes"
Jul 27 22:00:11.443861 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:11.443829    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-f368c2-predictor-7bc44788b5-m44b9"]
Jul 27 22:00:11.444323 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:11.444094    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0" containerName="kube-rbac-proxy"
Jul 27 22:00:11.444323 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:11.444104    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0" containerName="kube-rbac-proxy"
Jul 27 22:00:11.444323 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:11.444111    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0" containerName="kserve-container"
Jul 27 22:00:11.444323 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:11.444118    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0" containerName="kserve-container"
Jul 27 22:00:11.444323 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:11.444128    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0" containerName="storage-initializer"
Jul 27 22:00:11.444323 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:11.444134    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0" containerName="storage-initializer"
Jul 27 22:00:11.444323 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:11.444183    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0" containerName="kube-rbac-proxy"
Jul 27 22:00:11.444323 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:11.444191    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c5fe8b5-6f1e-482e-bbeb-03ccc03ee6b0" containerName="kserve-container"
Jul 27 22:00:11.448600 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:11.448582    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-f368c2-predictor-7bc44788b5-m44b9"
Jul 27 22:00:11.451076 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:11.450997    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 22:00:11.451076 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:11.451047    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 22:00:11.451076 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:11.451053    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 22:00:11.451076 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:11.451047    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-f368c2-predictor-serving-cert\""
Jul 27 22:00:11.451351 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:11.451058    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-f368c2-kube-rbac-proxy-sar-config\""
Jul 27 22:00:11.451351 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:11.451056    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v7c66\""
Jul 27 22:00:11.451785 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:11.451756    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 22:00:11.456998 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:11.456978    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-f368c2-predictor-7bc44788b5-m44b9"]
Jul 27 22:00:11.578814 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:11.578782    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3110f628-56f9-4146-b1e4-7f82fecc437a-kube-rbac-proxy-sar-config\") pod \"isvc-primary-f368c2-predictor-7bc44788b5-m44b9\" (UID: \"3110f628-56f9-4146-b1e4-7f82fecc437a\") " pod="kserve-ci-e2e-test/isvc-primary-f368c2-predictor-7bc44788b5-m44b9"
Jul 27 22:00:11.578814 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:11.578819    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3110f628-56f9-4146-b1e4-7f82fecc437a-kserve-provision-location\") pod \"isvc-primary-f368c2-predictor-7bc44788b5-m44b9\" (UID: \"3110f628-56f9-4146-b1e4-7f82fecc437a\") " pod="kserve-ci-e2e-test/isvc-primary-f368c2-predictor-7bc44788b5-m44b9"
Jul 27 22:00:11.579043 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:11.578909    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhh44\" (UniqueName: \"kubernetes.io/projected/3110f628-56f9-4146-b1e4-7f82fecc437a-kube-api-access-xhh44\") pod \"isvc-primary-f368c2-predictor-7bc44788b5-m44b9\" (UID: \"3110f628-56f9-4146-b1e4-7f82fecc437a\") " pod="kserve-ci-e2e-test/isvc-primary-f368c2-predictor-7bc44788b5-m44b9"
Jul 27 22:00:11.579043 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:11.578946    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3110f628-56f9-4146-b1e4-7f82fecc437a-proxy-tls\") pod \"isvc-primary-f368c2-predictor-7bc44788b5-m44b9\" (UID: \"3110f628-56f9-4146-b1e4-7f82fecc437a\") " pod="kserve-ci-e2e-test/isvc-primary-f368c2-predictor-7bc44788b5-m44b9"
Jul 27 22:00:11.579043 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:11.578968    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3110f628-56f9-4146-b1e4-7f82fecc437a-cabundle-cert\") pod \"isvc-primary-f368c2-predictor-7bc44788b5-m44b9\" (UID: \"3110f628-56f9-4146-b1e4-7f82fecc437a\") " pod="kserve-ci-e2e-test/isvc-primary-f368c2-predictor-7bc44788b5-m44b9"
Jul 27 22:00:11.680196 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:11.680160    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3110f628-56f9-4146-b1e4-7f82fecc437a-kserve-provision-location\") pod \"isvc-primary-f368c2-predictor-7bc44788b5-m44b9\" (UID: \"3110f628-56f9-4146-b1e4-7f82fecc437a\") " pod="kserve-ci-e2e-test/isvc-primary-f368c2-predictor-7bc44788b5-m44b9"
Jul 27 22:00:11.680389 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:11.680258    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhh44\" (UniqueName: \"kubernetes.io/projected/3110f628-56f9-4146-b1e4-7f82fecc437a-kube-api-access-xhh44\") pod \"isvc-primary-f368c2-predictor-7bc44788b5-m44b9\" (UID: \"3110f628-56f9-4146-b1e4-7f82fecc437a\") " pod="kserve-ci-e2e-test/isvc-primary-f368c2-predictor-7bc44788b5-m44b9"
Jul 27 22:00:11.680389 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:11.680292    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3110f628-56f9-4146-b1e4-7f82fecc437a-proxy-tls\") pod \"isvc-primary-f368c2-predictor-7bc44788b5-m44b9\" (UID: \"3110f628-56f9-4146-b1e4-7f82fecc437a\") " pod="kserve-ci-e2e-test/isvc-primary-f368c2-predictor-7bc44788b5-m44b9"
Jul 27 22:00:11.680389 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:11.680320    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3110f628-56f9-4146-b1e4-7f82fecc437a-cabundle-cert\") pod \"isvc-primary-f368c2-predictor-7bc44788b5-m44b9\" (UID: \"3110f628-56f9-4146-b1e4-7f82fecc437a\") " pod="kserve-ci-e2e-test/isvc-primary-f368c2-predictor-7bc44788b5-m44b9"
Jul 27 22:00:11.680572 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:11.680510    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3110f628-56f9-4146-b1e4-7f82fecc437a-kube-rbac-proxy-sar-config\") pod \"isvc-primary-f368c2-predictor-7bc44788b5-m44b9\" (UID: \"3110f628-56f9-4146-b1e4-7f82fecc437a\") " pod="kserve-ci-e2e-test/isvc-primary-f368c2-predictor-7bc44788b5-m44b9"
Jul 27 22:00:11.680645 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:11.680625    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3110f628-56f9-4146-b1e4-7f82fecc437a-kserve-provision-location\") pod \"isvc-primary-f368c2-predictor-7bc44788b5-m44b9\" (UID: \"3110f628-56f9-4146-b1e4-7f82fecc437a\") " pod="kserve-ci-e2e-test/isvc-primary-f368c2-predictor-7bc44788b5-m44b9"
Jul 27 22:00:11.680945 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:11.680926    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3110f628-56f9-4146-b1e4-7f82fecc437a-cabundle-cert\") pod \"isvc-primary-f368c2-predictor-7bc44788b5-m44b9\" (UID: \"3110f628-56f9-4146-b1e4-7f82fecc437a\") " pod="kserve-ci-e2e-test/isvc-primary-f368c2-predictor-7bc44788b5-m44b9"
Jul 27 22:00:11.681090 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:11.681069    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3110f628-56f9-4146-b1e4-7f82fecc437a-kube-rbac-proxy-sar-config\") pod \"isvc-primary-f368c2-predictor-7bc44788b5-m44b9\" (UID: \"3110f628-56f9-4146-b1e4-7f82fecc437a\") " pod="kserve-ci-e2e-test/isvc-primary-f368c2-predictor-7bc44788b5-m44b9"
Jul 27 22:00:11.682713 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:11.682696    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3110f628-56f9-4146-b1e4-7f82fecc437a-proxy-tls\") pod \"isvc-primary-f368c2-predictor-7bc44788b5-m44b9\" (UID: \"3110f628-56f9-4146-b1e4-7f82fecc437a\") " pod="kserve-ci-e2e-test/isvc-primary-f368c2-predictor-7bc44788b5-m44b9"
Jul 27 22:00:11.688440 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:11.688416    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhh44\" (UniqueName: \"kubernetes.io/projected/3110f628-56f9-4146-b1e4-7f82fecc437a-kube-api-access-xhh44\") pod \"isvc-primary-f368c2-predictor-7bc44788b5-m44b9\" (UID: \"3110f628-56f9-4146-b1e4-7f82fecc437a\") " pod="kserve-ci-e2e-test/isvc-primary-f368c2-predictor-7bc44788b5-m44b9"
Jul 27 22:00:11.759182 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:11.759082    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-f368c2-predictor-7bc44788b5-m44b9"
Jul 27 22:00:11.885023 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:11.884989    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-f368c2-predictor-7bc44788b5-m44b9"]
Jul 27 22:00:11.888222 ip-10-0-142-219 kubenswrapper[2577]: W0727 22:00:11.888197    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3110f628_56f9_4146_b1e4_7f82fecc437a.slice/crio-f8d7495ca4dd095e4f583f6a2d9cc5a5849f33d7846fa7994036e57d6e89f8b4 WatchSource:0}: Error finding container f8d7495ca4dd095e4f583f6a2d9cc5a5849f33d7846fa7994036e57d6e89f8b4: Status 404 returned error can't find the container with id f8d7495ca4dd095e4f583f6a2d9cc5a5849f33d7846fa7994036e57d6e89f8b4
Jul 27 22:00:12.345952 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:12.345914    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:00:12.537596 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:12.537561    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-f368c2-predictor-7bc44788b5-m44b9" event={"ID":"3110f628-56f9-4146-b1e4-7f82fecc437a","Type":"ContainerStarted","Data":"825da8c790ef41f80b6f9ffe3716eb5b4204703927f5d4454ddc159f866034e4"}
Jul 27 22:00:12.538017 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:12.537623    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-f368c2-predictor-7bc44788b5-m44b9" event={"ID":"3110f628-56f9-4146-b1e4-7f82fecc437a","Type":"ContainerStarted","Data":"f8d7495ca4dd095e4f583f6a2d9cc5a5849f33d7846fa7994036e57d6e89f8b4"}
Jul 27 22:00:16.559934 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:16.559899    2577 generic.go:356] "Generic (PLEG): container finished" podID="3110f628-56f9-4146-b1e4-7f82fecc437a" containerID="825da8c790ef41f80b6f9ffe3716eb5b4204703927f5d4454ddc159f866034e4" exitCode=0
Jul 27 22:00:16.560310 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:16.559973    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-f368c2-predictor-7bc44788b5-m44b9" event={"ID":"3110f628-56f9-4146-b1e4-7f82fecc437a","Type":"ContainerDied","Data":"825da8c790ef41f80b6f9ffe3716eb5b4204703927f5d4454ddc159f866034e4"}
Jul 27 22:00:17.566357 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:17.566323    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-f368c2-predictor-7bc44788b5-m44b9" event={"ID":"3110f628-56f9-4146-b1e4-7f82fecc437a","Type":"ContainerStarted","Data":"9c6c7d8fa105a69a607729ccfba102d031f6c7094afe66a3a3d2a4e519af286c"}
Jul 27 22:00:17.566746 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:17.566386    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-f368c2-predictor-7bc44788b5-m44b9" event={"ID":"3110f628-56f9-4146-b1e4-7f82fecc437a","Type":"ContainerStarted","Data":"94afb4e7199f57048b518c96a8ccb2ac0fca7e79678612b34d5bd36610bff675"}
Jul 27 22:00:17.566746 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:17.566624    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-f368c2-predictor-7bc44788b5-m44b9"
Jul 27 22:00:17.584674 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:17.584630    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-f368c2-predictor-7bc44788b5-m44b9" podStartSLOduration=2.895643615 podStartE2EDuration="6.584617042s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.688973427s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:00:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:00:17.582919721 +0000 UTC m=+3340.950109589" watchObservedRunningTime="2026-07-27 22:00:17.584617042 +0000 UTC m=+3340.951806907"
Jul 27 22:00:18.573170 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:18.573141    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-f368c2-predictor-7bc44788b5-m44b9"
Jul 27 22:00:18.574441 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:18.574413    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f368c2-predictor-7bc44788b5-m44b9" podUID="3110f628-56f9-4146-b1e4-7f82fecc437a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 27 22:00:19.577069 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:19.577019    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f368c2-predictor-7bc44788b5-m44b9" podUID="3110f628-56f9-4146-b1e4-7f82fecc437a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 27 22:00:24.581459 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:24.581431    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-f368c2-predictor-7bc44788b5-m44b9"
Jul 27 22:00:24.582040 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:24.581960    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f368c2-predictor-7bc44788b5-m44b9" podUID="3110f628-56f9-4146-b1e4-7f82fecc437a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 27 22:00:34.582078 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:34.582039    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f368c2-predictor-7bc44788b5-m44b9" podUID="3110f628-56f9-4146-b1e4-7f82fecc437a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 27 22:00:44.582887 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:44.582847    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f368c2-predictor-7bc44788b5-m44b9" podUID="3110f628-56f9-4146-b1e4-7f82fecc437a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 27 22:00:54.582297 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:00:54.582255    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f368c2-predictor-7bc44788b5-m44b9" podUID="3110f628-56f9-4146-b1e4-7f82fecc437a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 27 22:01:04.581980 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:01:04.581936    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f368c2-predictor-7bc44788b5-m44b9" podUID="3110f628-56f9-4146-b1e4-7f82fecc437a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 27 22:01:14.582055 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:01:14.582013    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f368c2-predictor-7bc44788b5-m44b9" podUID="3110f628-56f9-4146-b1e4-7f82fecc437a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 27 22:01:24.582597 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:01:24.582555    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f368c2-predictor-7bc44788b5-m44b9" podUID="3110f628-56f9-4146-b1e4-7f82fecc437a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 27 22:01:34.582519 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:01:34.582447    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-f368c2-predictor-7bc44788b5-m44b9"
Jul 27 22:01:56.626519 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:01:56.626467    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:01:56.660311 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:01:56.655215    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-f368c2-predictor-76ff67b47d-5xdbd"]
Jul 27 22:01:56.661477 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:01:56.661455    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-f368c2-predictor-76ff67b47d-5xdbd"
Jul 27 22:01:56.664310 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:01:56.664289    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-f368c2-predictor-serving-cert\""
Jul 27 22:01:56.664310 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:01:56.664301    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-f368c2-dockercfg-l9srg\""
Jul 27 22:01:56.664461 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:01:56.664297    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-f368c2-kube-rbac-proxy-sar-config\""
Jul 27 22:01:56.664461 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:01:56.664325    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-f368c2\""
Jul 27 22:01:56.669719 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:01:56.669697    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-f368c2-predictor-76ff67b47d-5xdbd"]
Jul 27 22:01:56.719724 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:01:56.719695    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36b50cda-ca41-47fc-8257-a401fee51f36-kserve-provision-location\") pod \"isvc-secondary-f368c2-predictor-76ff67b47d-5xdbd\" (UID: \"36b50cda-ca41-47fc-8257-a401fee51f36\") " pod="kserve-ci-e2e-test/isvc-secondary-f368c2-predictor-76ff67b47d-5xdbd"
Jul 27 22:01:56.719872 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:01:56.719741    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tgss\" (UniqueName: \"kubernetes.io/projected/36b50cda-ca41-47fc-8257-a401fee51f36-kube-api-access-7tgss\") pod \"isvc-secondary-f368c2-predictor-76ff67b47d-5xdbd\" (UID: \"36b50cda-ca41-47fc-8257-a401fee51f36\") " pod="kserve-ci-e2e-test/isvc-secondary-f368c2-predictor-76ff67b47d-5xdbd"
Jul 27 22:01:56.719872 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:01:56.719758    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/36b50cda-ca41-47fc-8257-a401fee51f36-cabundle-cert\") pod \"isvc-secondary-f368c2-predictor-76ff67b47d-5xdbd\" (UID: \"36b50cda-ca41-47fc-8257-a401fee51f36\") " pod="kserve-ci-e2e-test/isvc-secondary-f368c2-predictor-76ff67b47d-5xdbd"
Jul 27 22:01:56.719872 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:01:56.719811    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36b50cda-ca41-47fc-8257-a401fee51f36-proxy-tls\") pod \"isvc-secondary-f368c2-predictor-76ff67b47d-5xdbd\" (UID: \"36b50cda-ca41-47fc-8257-a401fee51f36\") " pod="kserve-ci-e2e-test/isvc-secondary-f368c2-predictor-76ff67b47d-5xdbd"
Jul 27 22:01:56.719872 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:01:56.719839    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/36b50cda-ca41-47fc-8257-a401fee51f36-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-f368c2-predictor-76ff67b47d-5xdbd\" (UID: \"36b50cda-ca41-47fc-8257-a401fee51f36\") " pod="kserve-ci-e2e-test/isvc-secondary-f368c2-predictor-76ff67b47d-5xdbd"
Jul 27 22:01:56.770407 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:01:56.770378    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:01:56.820542 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:01:56.820503    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/36b50cda-ca41-47fc-8257-a401fee51f36-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-f368c2-predictor-76ff67b47d-5xdbd\" (UID: \"36b50cda-ca41-47fc-8257-a401fee51f36\") " pod="kserve-ci-e2e-test/isvc-secondary-f368c2-predictor-76ff67b47d-5xdbd"
Jul 27 22:01:56.820739 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:01:56.820570    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36b50cda-ca41-47fc-8257-a401fee51f36-kserve-provision-location\") pod \"isvc-secondary-f368c2-predictor-76ff67b47d-5xdbd\" (UID: \"36b50cda-ca41-47fc-8257-a401fee51f36\") " pod="kserve-ci-e2e-test/isvc-secondary-f368c2-predictor-76ff67b47d-5xdbd"
Jul 27 22:01:56.820739 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:01:56.820621    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tgss\" (UniqueName: \"kubernetes.io/projected/36b50cda-ca41-47fc-8257-a401fee51f36-kube-api-access-7tgss\") pod \"isvc-secondary-f368c2-predictor-76ff67b47d-5xdbd\" (UID: \"36b50cda-ca41-47fc-8257-a401fee51f36\") " pod="kserve-ci-e2e-test/isvc-secondary-f368c2-predictor-76ff67b47d-5xdbd"
Jul 27 22:01:56.820739 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:01:56.820643    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/36b50cda-ca41-47fc-8257-a401fee51f36-cabundle-cert\") pod \"isvc-secondary-f368c2-predictor-76ff67b47d-5xdbd\" (UID: \"36b50cda-ca41-47fc-8257-a401fee51f36\") " pod="kserve-ci-e2e-test/isvc-secondary-f368c2-predictor-76ff67b47d-5xdbd"
Jul 27 22:01:56.820739 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:01:56.820677    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36b50cda-ca41-47fc-8257-a401fee51f36-proxy-tls\") pod \"isvc-secondary-f368c2-predictor-76ff67b47d-5xdbd\" (UID: \"36b50cda-ca41-47fc-8257-a401fee51f36\") " pod="kserve-ci-e2e-test/isvc-secondary-f368c2-predictor-76ff67b47d-5xdbd"
Jul 27 22:01:56.821099 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:01:56.821066    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36b50cda-ca41-47fc-8257-a401fee51f36-kserve-provision-location\") pod \"isvc-secondary-f368c2-predictor-76ff67b47d-5xdbd\" (UID: \"36b50cda-ca41-47fc-8257-a401fee51f36\") " pod="kserve-ci-e2e-test/isvc-secondary-f368c2-predictor-76ff67b47d-5xdbd"
Jul 27 22:01:56.821366 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:01:56.821345    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/36b50cda-ca41-47fc-8257-a401fee51f36-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-f368c2-predictor-76ff67b47d-5xdbd\" (UID: \"36b50cda-ca41-47fc-8257-a401fee51f36\") " pod="kserve-ci-e2e-test/isvc-secondary-f368c2-predictor-76ff67b47d-5xdbd"
Jul 27 22:01:56.821413 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:01:56.821345    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/36b50cda-ca41-47fc-8257-a401fee51f36-cabundle-cert\") pod \"isvc-secondary-f368c2-predictor-76ff67b47d-5xdbd\" (UID: \"36b50cda-ca41-47fc-8257-a401fee51f36\") " pod="kserve-ci-e2e-test/isvc-secondary-f368c2-predictor-76ff67b47d-5xdbd"
Jul 27 22:01:56.823142 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:01:56.823123    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36b50cda-ca41-47fc-8257-a401fee51f36-proxy-tls\") pod \"isvc-secondary-f368c2-predictor-76ff67b47d-5xdbd\" (UID: \"36b50cda-ca41-47fc-8257-a401fee51f36\") " pod="kserve-ci-e2e-test/isvc-secondary-f368c2-predictor-76ff67b47d-5xdbd"
Jul 27 22:01:56.828694 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:01:56.828667    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tgss\" (UniqueName: \"kubernetes.io/projected/36b50cda-ca41-47fc-8257-a401fee51f36-kube-api-access-7tgss\") pod \"isvc-secondary-f368c2-predictor-76ff67b47d-5xdbd\" (UID: \"36b50cda-ca41-47fc-8257-a401fee51f36\") " pod="kserve-ci-e2e-test/isvc-secondary-f368c2-predictor-76ff67b47d-5xdbd"
Jul 27 22:01:56.973146 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:01:56.973113    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-f368c2-predictor-76ff67b47d-5xdbd"
Jul 27 22:01:57.100704 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:01:57.100679    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-f368c2-predictor-76ff67b47d-5xdbd"]
Jul 27 22:01:57.102707 ip-10-0-142-219 kubenswrapper[2577]: W0727 22:01:57.102679    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36b50cda_ca41_47fc_8257_a401fee51f36.slice/crio-7835a4e52e4c8b60d9c09c069d16d6e71a1b870f7b151bf27ac68356d29d98fa WatchSource:0}: Error finding container 7835a4e52e4c8b60d9c09c069d16d6e71a1b870f7b151bf27ac68356d29d98fa: Status 404 returned error can't find the container with id 7835a4e52e4c8b60d9c09c069d16d6e71a1b870f7b151bf27ac68356d29d98fa
Jul 27 22:01:57.954442 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:01:57.954411    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-f368c2-predictor-76ff67b47d-5xdbd" event={"ID":"36b50cda-ca41-47fc-8257-a401fee51f36","Type":"ContainerStarted","Data":"243c8e5138ee2887b188e9136a0aec2c0082e644d0c6eda0ac131677d67a99d3"}
Jul 27 22:01:57.954854 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:01:57.954471    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-f368c2-predictor-76ff67b47d-5xdbd" event={"ID":"36b50cda-ca41-47fc-8257-a401fee51f36","Type":"ContainerStarted","Data":"7835a4e52e4c8b60d9c09c069d16d6e71a1b870f7b151bf27ac68356d29d98fa"}
Jul 27 22:02:03.421891 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:03.421853    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:02:03.980157 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:03.980127    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-f368c2-predictor-76ff67b47d-5xdbd_36b50cda-ca41-47fc-8257-a401fee51f36/storage-initializer/0.log"
Jul 27 22:02:03.980327 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:03.980166    2577 generic.go:356] "Generic (PLEG): container finished" podID="36b50cda-ca41-47fc-8257-a401fee51f36" containerID="243c8e5138ee2887b188e9136a0aec2c0082e644d0c6eda0ac131677d67a99d3" exitCode=1
Jul 27 22:02:03.980327 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:03.980247    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-f368c2-predictor-76ff67b47d-5xdbd" event={"ID":"36b50cda-ca41-47fc-8257-a401fee51f36","Type":"ContainerDied","Data":"243c8e5138ee2887b188e9136a0aec2c0082e644d0c6eda0ac131677d67a99d3"}
Jul 27 22:02:04.986107 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:04.986079    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-f368c2-predictor-76ff67b47d-5xdbd_36b50cda-ca41-47fc-8257-a401fee51f36/storage-initializer/0.log"
Jul 27 22:02:04.986492 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:04.986176    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-f368c2-predictor-76ff67b47d-5xdbd" event={"ID":"36b50cda-ca41-47fc-8257-a401fee51f36","Type":"ContainerStarted","Data":"1e93d12f0f4ed841a83dc914f549c63dd6199f863d320653d56e085ebfde2fee"}
Jul 27 22:02:09.004837 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:09.004809    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-f368c2-predictor-76ff67b47d-5xdbd_36b50cda-ca41-47fc-8257-a401fee51f36/storage-initializer/1.log"
Jul 27 22:02:09.005202 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:09.005148    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-f368c2-predictor-76ff67b47d-5xdbd_36b50cda-ca41-47fc-8257-a401fee51f36/storage-initializer/0.log"
Jul 27 22:02:09.005202 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:09.005177    2577 generic.go:356] "Generic (PLEG): container finished" podID="36b50cda-ca41-47fc-8257-a401fee51f36" containerID="1e93d12f0f4ed841a83dc914f549c63dd6199f863d320653d56e085ebfde2fee" exitCode=1
Jul 27 22:02:09.005287 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:09.005200    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-f368c2-predictor-76ff67b47d-5xdbd" event={"ID":"36b50cda-ca41-47fc-8257-a401fee51f36","Type":"ContainerDied","Data":"1e93d12f0f4ed841a83dc914f549c63dd6199f863d320653d56e085ebfde2fee"}
Jul 27 22:02:09.005287 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:09.005276    2577 scope.go:117] "RemoveContainer" containerID="243c8e5138ee2887b188e9136a0aec2c0082e644d0c6eda0ac131677d67a99d3"
Jul 27 22:02:09.005669 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:09.005635    2577 scope.go:117] "RemoveContainer" containerID="243c8e5138ee2887b188e9136a0aec2c0082e644d0c6eda0ac131677d67a99d3"
Jul 27 22:02:09.016154 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:02:09.016131    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-f368c2-predictor-76ff67b47d-5xdbd_kserve-ci-e2e-test(36b50cda-ca41-47fc-8257-a401fee51f36)\"" pod="kserve-ci-e2e-test/isvc-secondary-f368c2-predictor-76ff67b47d-5xdbd" podUID="36b50cda-ca41-47fc-8257-a401fee51f36"
Jul 27 22:02:09.016394 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:02:09.016371    2577 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-f368c2-predictor-76ff67b47d-5xdbd_kserve-ci-e2e-test_36b50cda-ca41-47fc-8257-a401fee51f36_0 in pod sandbox 7835a4e52e4c8b60d9c09c069d16d6e71a1b870f7b151bf27ac68356d29d98fa from index: no such id: '243c8e5138ee2887b188e9136a0aec2c0082e644d0c6eda0ac131677d67a99d3'" containerID="243c8e5138ee2887b188e9136a0aec2c0082e644d0c6eda0ac131677d67a99d3"
Jul 27 22:02:09.016454 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:09.016406    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"243c8e5138ee2887b188e9136a0aec2c0082e644d0c6eda0ac131677d67a99d3"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-f368c2-predictor-76ff67b47d-5xdbd_kserve-ci-e2e-test_36b50cda-ca41-47fc-8257-a401fee51f36_0 in pod sandbox 7835a4e52e4c8b60d9c09c069d16d6e71a1b870f7b151bf27ac68356d29d98fa from index: no such id: '243c8e5138ee2887b188e9136a0aec2c0082e644d0c6eda0ac131677d67a99d3'"
Jul 27 22:02:10.010908 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:10.010880    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-f368c2-predictor-76ff67b47d-5xdbd_36b50cda-ca41-47fc-8257-a401fee51f36/storage-initializer/1.log"
Jul 27 22:02:14.700402 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:14.700370    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-f368c2-predictor-76ff67b47d-5xdbd"]
Jul 27 22:02:14.728370 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:14.728334    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:02:14.753569 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:14.753541    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-f368c2-predictor-7bc44788b5-m44b9"]
Jul 27 22:02:14.753839 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:14.753811    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-f368c2-predictor-7bc44788b5-m44b9" podUID="3110f628-56f9-4146-b1e4-7f82fecc437a" containerName="kserve-container" containerID="cri-o://94afb4e7199f57048b518c96a8ccb2ac0fca7e79678612b34d5bd36610bff675" gracePeriod=30
Jul 27 22:02:14.754021 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:14.753853    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-f368c2-predictor-7bc44788b5-m44b9" podUID="3110f628-56f9-4146-b1e4-7f82fecc437a" containerName="kube-rbac-proxy" containerID="cri-o://9c6c7d8fa105a69a607729ccfba102d031f6c7094afe66a3a3d2a4e519af286c" gracePeriod=30
Jul 27 22:02:14.793793 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:14.793746    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:02:14.873062 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:14.873041    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-f368c2-predictor-76ff67b47d-5xdbd_36b50cda-ca41-47fc-8257-a401fee51f36/storage-initializer/1.log"
Jul 27 22:02:14.875206 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:14.875188    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-f368c2-predictor-76ff67b47d-5xdbd"
Jul 27 22:02:14.967276 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:14.967195    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36b50cda-ca41-47fc-8257-a401fee51f36-kserve-provision-location\") pod \"36b50cda-ca41-47fc-8257-a401fee51f36\" (UID: \"36b50cda-ca41-47fc-8257-a401fee51f36\") "
Jul 27 22:02:14.967276 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:14.967252    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/36b50cda-ca41-47fc-8257-a401fee51f36-cabundle-cert\") pod \"36b50cda-ca41-47fc-8257-a401fee51f36\" (UID: \"36b50cda-ca41-47fc-8257-a401fee51f36\") "
Jul 27 22:02:14.967472 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:14.967282    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36b50cda-ca41-47fc-8257-a401fee51f36-proxy-tls\") pod \"36b50cda-ca41-47fc-8257-a401fee51f36\" (UID: \"36b50cda-ca41-47fc-8257-a401fee51f36\") "
Jul 27 22:02:14.967472 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:14.967315    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7tgss\" (UniqueName: \"kubernetes.io/projected/36b50cda-ca41-47fc-8257-a401fee51f36-kube-api-access-7tgss\") pod \"36b50cda-ca41-47fc-8257-a401fee51f36\" (UID: \"36b50cda-ca41-47fc-8257-a401fee51f36\") "
Jul 27 22:02:14.967472 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:14.967339    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/36b50cda-ca41-47fc-8257-a401fee51f36-kube-rbac-proxy-sar-config\") pod \"36b50cda-ca41-47fc-8257-a401fee51f36\" (UID: \"36b50cda-ca41-47fc-8257-a401fee51f36\") "
Jul 27 22:02:14.967572 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:14.967514    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b50cda-ca41-47fc-8257-a401fee51f36-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "36b50cda-ca41-47fc-8257-a401fee51f36" (UID: "36b50cda-ca41-47fc-8257-a401fee51f36"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:02:14.967625 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:14.967588    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36b50cda-ca41-47fc-8257-a401fee51f36-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "36b50cda-ca41-47fc-8257-a401fee51f36" (UID: "36b50cda-ca41-47fc-8257-a401fee51f36"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:02:14.967730 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:14.967708    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36b50cda-ca41-47fc-8257-a401fee51f36-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "36b50cda-ca41-47fc-8257-a401fee51f36" (UID: "36b50cda-ca41-47fc-8257-a401fee51f36"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:02:14.969359 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:14.969336    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36b50cda-ca41-47fc-8257-a401fee51f36-kube-api-access-7tgss" (OuterVolumeSpecName: "kube-api-access-7tgss") pod "36b50cda-ca41-47fc-8257-a401fee51f36" (UID: "36b50cda-ca41-47fc-8257-a401fee51f36"). InnerVolumeSpecName "kube-api-access-7tgss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:02:14.969446 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:14.969428    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36b50cda-ca41-47fc-8257-a401fee51f36-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "36b50cda-ca41-47fc-8257-a401fee51f36" (UID: "36b50cda-ca41-47fc-8257-a401fee51f36"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 22:02:15.032580 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:15.032555    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-f368c2-predictor-76ff67b47d-5xdbd_36b50cda-ca41-47fc-8257-a401fee51f36/storage-initializer/1.log"
Jul 27 22:02:15.032728 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:15.032658    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-f368c2-predictor-76ff67b47d-5xdbd"
Jul 27 22:02:15.032728 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:15.032667    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-f368c2-predictor-76ff67b47d-5xdbd" event={"ID":"36b50cda-ca41-47fc-8257-a401fee51f36","Type":"ContainerDied","Data":"7835a4e52e4c8b60d9c09c069d16d6e71a1b870f7b151bf27ac68356d29d98fa"}
Jul 27 22:02:15.032857 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:15.032787    2577 scope.go:117] "RemoveContainer" containerID="1e93d12f0f4ed841a83dc914f549c63dd6199f863d320653d56e085ebfde2fee"
Jul 27 22:02:15.034916 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:15.034887    2577 generic.go:356] "Generic (PLEG): container finished" podID="3110f628-56f9-4146-b1e4-7f82fecc437a" containerID="9c6c7d8fa105a69a607729ccfba102d031f6c7094afe66a3a3d2a4e519af286c" exitCode=2
Jul 27 22:02:15.035052 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:15.034959    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-f368c2-predictor-7bc44788b5-m44b9" event={"ID":"3110f628-56f9-4146-b1e4-7f82fecc437a","Type":"ContainerDied","Data":"9c6c7d8fa105a69a607729ccfba102d031f6c7094afe66a3a3d2a4e519af286c"}
Jul 27 22:02:15.068806 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:15.068777    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36b50cda-ca41-47fc-8257-a401fee51f36-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:02:15.068947 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:15.068810    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/36b50cda-ca41-47fc-8257-a401fee51f36-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:02:15.068947 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:15.068827    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36b50cda-ca41-47fc-8257-a401fee51f36-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:02:15.068947 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:15.068840    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tgss\" (UniqueName: \"kubernetes.io/projected/36b50cda-ca41-47fc-8257-a401fee51f36-kube-api-access-7tgss\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:02:15.068947 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:15.068853    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/36b50cda-ca41-47fc-8257-a401fee51f36-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:02:15.070066 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:15.070045    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-f368c2-predictor-76ff67b47d-5xdbd"]
Jul 27 22:02:15.074062 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:15.074039    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-f368c2-predictor-76ff67b47d-5xdbd"]
Jul 27 22:02:15.135455 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:15.135428    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36b50cda-ca41-47fc-8257-a401fee51f36" path="/var/lib/kubelet/pods/36b50cda-ca41-47fc-8257-a401fee51f36/volumes"
Jul 27 22:02:19.306958 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:19.306929    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-f368c2-predictor-7bc44788b5-m44b9"
Jul 27 22:02:19.404521 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:19.404443    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3110f628-56f9-4146-b1e4-7f82fecc437a-kserve-provision-location\") pod \"3110f628-56f9-4146-b1e4-7f82fecc437a\" (UID: \"3110f628-56f9-4146-b1e4-7f82fecc437a\") "
Jul 27 22:02:19.404521 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:19.404497    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xhh44\" (UniqueName: \"kubernetes.io/projected/3110f628-56f9-4146-b1e4-7f82fecc437a-kube-api-access-xhh44\") pod \"3110f628-56f9-4146-b1e4-7f82fecc437a\" (UID: \"3110f628-56f9-4146-b1e4-7f82fecc437a\") "
Jul 27 22:02:19.404521 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:19.404518    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3110f628-56f9-4146-b1e4-7f82fecc437a-proxy-tls\") pod \"3110f628-56f9-4146-b1e4-7f82fecc437a\" (UID: \"3110f628-56f9-4146-b1e4-7f82fecc437a\") "
Jul 27 22:02:19.404801 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:19.404549    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3110f628-56f9-4146-b1e4-7f82fecc437a-kube-rbac-proxy-sar-config\") pod \"3110f628-56f9-4146-b1e4-7f82fecc437a\" (UID: \"3110f628-56f9-4146-b1e4-7f82fecc437a\") "
Jul 27 22:02:19.404801 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:19.404595    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3110f628-56f9-4146-b1e4-7f82fecc437a-cabundle-cert\") pod \"3110f628-56f9-4146-b1e4-7f82fecc437a\" (UID: \"3110f628-56f9-4146-b1e4-7f82fecc437a\") "
Jul 27 22:02:19.404918 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:19.404823    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3110f628-56f9-4146-b1e4-7f82fecc437a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3110f628-56f9-4146-b1e4-7f82fecc437a" (UID: "3110f628-56f9-4146-b1e4-7f82fecc437a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:02:19.404975 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:19.404932    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3110f628-56f9-4146-b1e4-7f82fecc437a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3110f628-56f9-4146-b1e4-7f82fecc437a" (UID: "3110f628-56f9-4146-b1e4-7f82fecc437a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:02:19.405034 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:19.405007    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3110f628-56f9-4146-b1e4-7f82fecc437a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3110f628-56f9-4146-b1e4-7f82fecc437a" (UID: "3110f628-56f9-4146-b1e4-7f82fecc437a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:02:19.406566 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:19.406538    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3110f628-56f9-4146-b1e4-7f82fecc437a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3110f628-56f9-4146-b1e4-7f82fecc437a" (UID: "3110f628-56f9-4146-b1e4-7f82fecc437a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 22:02:19.406660 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:19.406582    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3110f628-56f9-4146-b1e4-7f82fecc437a-kube-api-access-xhh44" (OuterVolumeSpecName: "kube-api-access-xhh44") pod "3110f628-56f9-4146-b1e4-7f82fecc437a" (UID: "3110f628-56f9-4146-b1e4-7f82fecc437a"). InnerVolumeSpecName "kube-api-access-xhh44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:02:19.505277 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:19.505243    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3110f628-56f9-4146-b1e4-7f82fecc437a-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:02:19.505277 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:19.505270    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhh44\" (UniqueName: \"kubernetes.io/projected/3110f628-56f9-4146-b1e4-7f82fecc437a-kube-api-access-xhh44\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:02:19.505277 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:19.505281    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3110f628-56f9-4146-b1e4-7f82fecc437a-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:02:19.505496 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:19.505291    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3110f628-56f9-4146-b1e4-7f82fecc437a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:02:19.505496 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:19.505300    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3110f628-56f9-4146-b1e4-7f82fecc437a-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:02:19.897879 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:19.897836    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:02:19.944063 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:19.944037    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-3483ea-predictor-66f8c699b4-4x94s"]
Jul 27 22:02:19.944331 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:19.944319    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3110f628-56f9-4146-b1e4-7f82fecc437a" containerName="kserve-container"
Jul 27 22:02:19.944371 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:19.944333    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3110f628-56f9-4146-b1e4-7f82fecc437a" containerName="kserve-container"
Jul 27 22:02:19.944371 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:19.944350    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36b50cda-ca41-47fc-8257-a401fee51f36" containerName="storage-initializer"
Jul 27 22:02:19.944371 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:19.944356    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="36b50cda-ca41-47fc-8257-a401fee51f36" containerName="storage-initializer"
Jul 27 22:02:19.944371 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:19.944367    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3110f628-56f9-4146-b1e4-7f82fecc437a" containerName="storage-initializer"
Jul 27 22:02:19.944371 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:19.944372    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3110f628-56f9-4146-b1e4-7f82fecc437a" containerName="storage-initializer"
Jul 27 22:02:19.944531 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:19.944378    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3110f628-56f9-4146-b1e4-7f82fecc437a" containerName="kube-rbac-proxy"
Jul 27 22:02:19.944531 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:19.944383    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3110f628-56f9-4146-b1e4-7f82fecc437a" containerName="kube-rbac-proxy"
Jul 27 22:02:19.944531 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:19.944433    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="36b50cda-ca41-47fc-8257-a401fee51f36" containerName="storage-initializer"
Jul 27 22:02:19.944531 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:19.944441    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3110f628-56f9-4146-b1e4-7f82fecc437a" containerName="kserve-container"
Jul 27 22:02:19.944531 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:19.944447    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3110f628-56f9-4146-b1e4-7f82fecc437a" containerName="kube-rbac-proxy"
Jul 27 22:02:19.944531 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:19.944452    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="36b50cda-ca41-47fc-8257-a401fee51f36" containerName="storage-initializer"
Jul 27 22:02:19.944531 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:19.944498    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36b50cda-ca41-47fc-8257-a401fee51f36" containerName="storage-initializer"
Jul 27 22:02:19.944531 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:19.944504    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="36b50cda-ca41-47fc-8257-a401fee51f36" containerName="storage-initializer"
Jul 27 22:02:19.947620 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:19.947603    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-3483ea-predictor-66f8c699b4-4x94s"
Jul 27 22:02:19.950209 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:19.950166    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-3483ea-predictor-serving-cert\""
Jul 27 22:02:19.950393 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:19.950172    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-3483ea-kube-rbac-proxy-sar-config\""
Jul 27 22:02:19.950548 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:19.950528    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-3483ea-dockercfg-kkdln\""
Jul 27 22:02:19.950614 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:19.950531    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-3483ea\""
Jul 27 22:02:19.957644 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:19.957623    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-3483ea-predictor-66f8c699b4-4x94s"]
Jul 27 22:02:20.008654 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:20.008627    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad2bf35f-a2b6-4eba-9a1a-86577603b7ef-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-3483ea-predictor-66f8c699b4-4x94s\" (UID: \"ad2bf35f-a2b6-4eba-9a1a-86577603b7ef\") " pod="kserve-ci-e2e-test/isvc-init-fail-3483ea-predictor-66f8c699b4-4x94s"
Jul 27 22:02:20.008787 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:20.008661    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad2bf35f-a2b6-4eba-9a1a-86577603b7ef-cabundle-cert\") pod \"isvc-init-fail-3483ea-predictor-66f8c699b4-4x94s\" (UID: \"ad2bf35f-a2b6-4eba-9a1a-86577603b7ef\") " pod="kserve-ci-e2e-test/isvc-init-fail-3483ea-predictor-66f8c699b4-4x94s"
Jul 27 22:02:20.008787 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:20.008687    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxwkg\" (UniqueName: \"kubernetes.io/projected/ad2bf35f-a2b6-4eba-9a1a-86577603b7ef-kube-api-access-cxwkg\") pod \"isvc-init-fail-3483ea-predictor-66f8c699b4-4x94s\" (UID: \"ad2bf35f-a2b6-4eba-9a1a-86577603b7ef\") " pod="kserve-ci-e2e-test/isvc-init-fail-3483ea-predictor-66f8c699b4-4x94s"
Jul 27 22:02:20.008875 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:20.008803    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad2bf35f-a2b6-4eba-9a1a-86577603b7ef-kserve-provision-location\") pod \"isvc-init-fail-3483ea-predictor-66f8c699b4-4x94s\" (UID: \"ad2bf35f-a2b6-4eba-9a1a-86577603b7ef\") " pod="kserve-ci-e2e-test/isvc-init-fail-3483ea-predictor-66f8c699b4-4x94s"
Jul 27 22:02:20.008875 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:20.008855    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad2bf35f-a2b6-4eba-9a1a-86577603b7ef-proxy-tls\") pod \"isvc-init-fail-3483ea-predictor-66f8c699b4-4x94s\" (UID: \"ad2bf35f-a2b6-4eba-9a1a-86577603b7ef\") " pod="kserve-ci-e2e-test/isvc-init-fail-3483ea-predictor-66f8c699b4-4x94s"
Jul 27 22:02:20.056019 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:20.055994    2577 generic.go:356] "Generic (PLEG): container finished" podID="3110f628-56f9-4146-b1e4-7f82fecc437a" containerID="94afb4e7199f57048b518c96a8ccb2ac0fca7e79678612b34d5bd36610bff675" exitCode=0
Jul 27 22:02:20.056132 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:20.056030    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-f368c2-predictor-7bc44788b5-m44b9" event={"ID":"3110f628-56f9-4146-b1e4-7f82fecc437a","Type":"ContainerDied","Data":"94afb4e7199f57048b518c96a8ccb2ac0fca7e79678612b34d5bd36610bff675"}
Jul 27 22:02:20.056132 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:20.056041    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-f368c2-predictor-7bc44788b5-m44b9"
Jul 27 22:02:20.056132 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:20.056088    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-f368c2-predictor-7bc44788b5-m44b9" event={"ID":"3110f628-56f9-4146-b1e4-7f82fecc437a","Type":"ContainerDied","Data":"f8d7495ca4dd095e4f583f6a2d9cc5a5849f33d7846fa7994036e57d6e89f8b4"}
Jul 27 22:02:20.056132 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:20.056125    2577 scope.go:117] "RemoveContainer" containerID="9c6c7d8fa105a69a607729ccfba102d031f6c7094afe66a3a3d2a4e519af286c"
Jul 27 22:02:20.064744 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:20.064728    2577 scope.go:117] "RemoveContainer" containerID="94afb4e7199f57048b518c96a8ccb2ac0fca7e79678612b34d5bd36610bff675"
Jul 27 22:02:20.072227 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:20.072209    2577 scope.go:117] "RemoveContainer" containerID="825da8c790ef41f80b6f9ffe3716eb5b4204703927f5d4454ddc159f866034e4"
Jul 27 22:02:20.079932 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:20.079915    2577 scope.go:117] "RemoveContainer" containerID="9c6c7d8fa105a69a607729ccfba102d031f6c7094afe66a3a3d2a4e519af286c"
Jul 27 22:02:20.080181 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:02:20.080163    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c6c7d8fa105a69a607729ccfba102d031f6c7094afe66a3a3d2a4e519af286c\": container with ID starting with 9c6c7d8fa105a69a607729ccfba102d031f6c7094afe66a3a3d2a4e519af286c not found: ID does not exist" containerID="9c6c7d8fa105a69a607729ccfba102d031f6c7094afe66a3a3d2a4e519af286c"
Jul 27 22:02:20.080228 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:20.080188    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c6c7d8fa105a69a607729ccfba102d031f6c7094afe66a3a3d2a4e519af286c"} err="failed to get container status \"9c6c7d8fa105a69a607729ccfba102d031f6c7094afe66a3a3d2a4e519af286c\": rpc error: code = NotFound desc = could not find container \"9c6c7d8fa105a69a607729ccfba102d031f6c7094afe66a3a3d2a4e519af286c\": container with ID starting with 9c6c7d8fa105a69a607729ccfba102d031f6c7094afe66a3a3d2a4e519af286c not found: ID does not exist"
Jul 27 22:02:20.080228 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:20.080205    2577 scope.go:117] "RemoveContainer" containerID="94afb4e7199f57048b518c96a8ccb2ac0fca7e79678612b34d5bd36610bff675"
Jul 27 22:02:20.080469 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:02:20.080429    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94afb4e7199f57048b518c96a8ccb2ac0fca7e79678612b34d5bd36610bff675\": container with ID starting with 94afb4e7199f57048b518c96a8ccb2ac0fca7e79678612b34d5bd36610bff675 not found: ID does not exist" containerID="94afb4e7199f57048b518c96a8ccb2ac0fca7e79678612b34d5bd36610bff675"
Jul 27 22:02:20.080469 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:20.080460    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94afb4e7199f57048b518c96a8ccb2ac0fca7e79678612b34d5bd36610bff675"} err="failed to get container status \"94afb4e7199f57048b518c96a8ccb2ac0fca7e79678612b34d5bd36610bff675\": rpc error: code = NotFound desc = could not find container \"94afb4e7199f57048b518c96a8ccb2ac0fca7e79678612b34d5bd36610bff675\": container with ID starting with 94afb4e7199f57048b518c96a8ccb2ac0fca7e79678612b34d5bd36610bff675 not found: ID does not exist"
Jul 27 22:02:20.080648 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:20.080477    2577 scope.go:117] "RemoveContainer" containerID="825da8c790ef41f80b6f9ffe3716eb5b4204703927f5d4454ddc159f866034e4"
Jul 27 22:02:20.081149 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:02:20.080756    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"825da8c790ef41f80b6f9ffe3716eb5b4204703927f5d4454ddc159f866034e4\": container with ID starting with 825da8c790ef41f80b6f9ffe3716eb5b4204703927f5d4454ddc159f866034e4 not found: ID does not exist" containerID="825da8c790ef41f80b6f9ffe3716eb5b4204703927f5d4454ddc159f866034e4"
Jul 27 22:02:20.081220 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:20.081157    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"825da8c790ef41f80b6f9ffe3716eb5b4204703927f5d4454ddc159f866034e4"} err="failed to get container status \"825da8c790ef41f80b6f9ffe3716eb5b4204703927f5d4454ddc159f866034e4\": rpc error: code = NotFound desc = could not find container \"825da8c790ef41f80b6f9ffe3716eb5b4204703927f5d4454ddc159f866034e4\": container with ID starting with 825da8c790ef41f80b6f9ffe3716eb5b4204703927f5d4454ddc159f866034e4 not found: ID does not exist"
Jul 27 22:02:20.084204 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:20.084182    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-f368c2-predictor-7bc44788b5-m44b9"]
Jul 27 22:02:20.091796 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:20.091776    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-f368c2-predictor-7bc44788b5-m44b9"]
Jul 27 22:02:20.109596 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:20.109578    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad2bf35f-a2b6-4eba-9a1a-86577603b7ef-kserve-provision-location\") pod \"isvc-init-fail-3483ea-predictor-66f8c699b4-4x94s\" (UID: \"ad2bf35f-a2b6-4eba-9a1a-86577603b7ef\") " pod="kserve-ci-e2e-test/isvc-init-fail-3483ea-predictor-66f8c699b4-4x94s"
Jul 27 22:02:20.109669 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:20.109628    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad2bf35f-a2b6-4eba-9a1a-86577603b7ef-proxy-tls\") pod \"isvc-init-fail-3483ea-predictor-66f8c699b4-4x94s\" (UID: \"ad2bf35f-a2b6-4eba-9a1a-86577603b7ef\") " pod="kserve-ci-e2e-test/isvc-init-fail-3483ea-predictor-66f8c699b4-4x94s"
Jul 27 22:02:20.109669 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:20.109655    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad2bf35f-a2b6-4eba-9a1a-86577603b7ef-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-3483ea-predictor-66f8c699b4-4x94s\" (UID: \"ad2bf35f-a2b6-4eba-9a1a-86577603b7ef\") " pod="kserve-ci-e2e-test/isvc-init-fail-3483ea-predictor-66f8c699b4-4x94s"
Jul 27 22:02:20.109782 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:20.109678    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad2bf35f-a2b6-4eba-9a1a-86577603b7ef-cabundle-cert\") pod \"isvc-init-fail-3483ea-predictor-66f8c699b4-4x94s\" (UID: \"ad2bf35f-a2b6-4eba-9a1a-86577603b7ef\") " pod="kserve-ci-e2e-test/isvc-init-fail-3483ea-predictor-66f8c699b4-4x94s"
Jul 27 22:02:20.109782 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:20.109750    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxwkg\" (UniqueName: \"kubernetes.io/projected/ad2bf35f-a2b6-4eba-9a1a-86577603b7ef-kube-api-access-cxwkg\") pod \"isvc-init-fail-3483ea-predictor-66f8c699b4-4x94s\" (UID: \"ad2bf35f-a2b6-4eba-9a1a-86577603b7ef\") " pod="kserve-ci-e2e-test/isvc-init-fail-3483ea-predictor-66f8c699b4-4x94s"
Jul 27 22:02:20.109896 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:02:20.109797    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-3483ea-predictor-serving-cert: secret "isvc-init-fail-3483ea-predictor-serving-cert" not found
Jul 27 22:02:20.109896 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:02:20.109872    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad2bf35f-a2b6-4eba-9a1a-86577603b7ef-proxy-tls podName:ad2bf35f-a2b6-4eba-9a1a-86577603b7ef nodeName:}" failed. No retries permitted until 2026-07-27 22:02:20.609850385 +0000 UTC m=+3463.977040233 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ad2bf35f-a2b6-4eba-9a1a-86577603b7ef-proxy-tls") pod "isvc-init-fail-3483ea-predictor-66f8c699b4-4x94s" (UID: "ad2bf35f-a2b6-4eba-9a1a-86577603b7ef") : secret "isvc-init-fail-3483ea-predictor-serving-cert" not found
Jul 27 22:02:20.110027 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:20.109969    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad2bf35f-a2b6-4eba-9a1a-86577603b7ef-kserve-provision-location\") pod \"isvc-init-fail-3483ea-predictor-66f8c699b4-4x94s\" (UID: \"ad2bf35f-a2b6-4eba-9a1a-86577603b7ef\") " pod="kserve-ci-e2e-test/isvc-init-fail-3483ea-predictor-66f8c699b4-4x94s"
Jul 27 22:02:20.110354 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:20.110333    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad2bf35f-a2b6-4eba-9a1a-86577603b7ef-cabundle-cert\") pod \"isvc-init-fail-3483ea-predictor-66f8c699b4-4x94s\" (UID: \"ad2bf35f-a2b6-4eba-9a1a-86577603b7ef\") " pod="kserve-ci-e2e-test/isvc-init-fail-3483ea-predictor-66f8c699b4-4x94s"
Jul 27 22:02:20.110466 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:20.110451    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad2bf35f-a2b6-4eba-9a1a-86577603b7ef-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-3483ea-predictor-66f8c699b4-4x94s\" (UID: \"ad2bf35f-a2b6-4eba-9a1a-86577603b7ef\") " pod="kserve-ci-e2e-test/isvc-init-fail-3483ea-predictor-66f8c699b4-4x94s"
Jul 27 22:02:20.120677 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:20.120658    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxwkg\" (UniqueName: \"kubernetes.io/projected/ad2bf35f-a2b6-4eba-9a1a-86577603b7ef-kube-api-access-cxwkg\") pod \"isvc-init-fail-3483ea-predictor-66f8c699b4-4x94s\" (UID: \"ad2bf35f-a2b6-4eba-9a1a-86577603b7ef\") " pod="kserve-ci-e2e-test/isvc-init-fail-3483ea-predictor-66f8c699b4-4x94s"
Jul 27 22:02:20.328116 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:20.328079    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:02:20.614318 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:20.614232    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad2bf35f-a2b6-4eba-9a1a-86577603b7ef-proxy-tls\") pod \"isvc-init-fail-3483ea-predictor-66f8c699b4-4x94s\" (UID: \"ad2bf35f-a2b6-4eba-9a1a-86577603b7ef\") " pod="kserve-ci-e2e-test/isvc-init-fail-3483ea-predictor-66f8c699b4-4x94s"
Jul 27 22:02:20.616569 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:20.616547    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad2bf35f-a2b6-4eba-9a1a-86577603b7ef-proxy-tls\") pod \"isvc-init-fail-3483ea-predictor-66f8c699b4-4x94s\" (UID: \"ad2bf35f-a2b6-4eba-9a1a-86577603b7ef\") " pod="kserve-ci-e2e-test/isvc-init-fail-3483ea-predictor-66f8c699b4-4x94s"
Jul 27 22:02:20.858643 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:20.858603    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-3483ea-predictor-66f8c699b4-4x94s"
Jul 27 22:02:20.990073 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:20.990051    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-3483ea-predictor-66f8c699b4-4x94s"]
Jul 27 22:02:20.992435 ip-10-0-142-219 kubenswrapper[2577]: W0727 22:02:20.992411    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad2bf35f_a2b6_4eba_9a1a_86577603b7ef.slice/crio-2d9f18a3004d3b40d97dd2fb9cb2ddf1a0b5db06fe68d4447877b745add1242a WatchSource:0}: Error finding container 2d9f18a3004d3b40d97dd2fb9cb2ddf1a0b5db06fe68d4447877b745add1242a: Status 404 returned error can't find the container with id 2d9f18a3004d3b40d97dd2fb9cb2ddf1a0b5db06fe68d4447877b745add1242a
Jul 27 22:02:21.061945 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:21.061917    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-3483ea-predictor-66f8c699b4-4x94s" event={"ID":"ad2bf35f-a2b6-4eba-9a1a-86577603b7ef","Type":"ContainerStarted","Data":"20224f8ee250930dc66e7ea06580765cc6feb0ae5c22039ce3ca1e74ef24b3e3"}
Jul 27 22:02:21.062083 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:21.061968    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-3483ea-predictor-66f8c699b4-4x94s" event={"ID":"ad2bf35f-a2b6-4eba-9a1a-86577603b7ef","Type":"ContainerStarted","Data":"2d9f18a3004d3b40d97dd2fb9cb2ddf1a0b5db06fe68d4447877b745add1242a"}
Jul 27 22:02:21.135180 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:21.135087    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3110f628-56f9-4146-b1e4-7f82fecc437a" path="/var/lib/kubelet/pods/3110f628-56f9-4146-b1e4-7f82fecc437a/volumes"
Jul 27 22:02:27.085704 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:27.085676    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-3483ea-predictor-66f8c699b4-4x94s_ad2bf35f-a2b6-4eba-9a1a-86577603b7ef/storage-initializer/0.log"
Jul 27 22:02:27.086119 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:27.085715    2577 generic.go:356] "Generic (PLEG): container finished" podID="ad2bf35f-a2b6-4eba-9a1a-86577603b7ef" containerID="20224f8ee250930dc66e7ea06580765cc6feb0ae5c22039ce3ca1e74ef24b3e3" exitCode=1
Jul 27 22:02:27.086119 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:27.085748    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-3483ea-predictor-66f8c699b4-4x94s" event={"ID":"ad2bf35f-a2b6-4eba-9a1a-86577603b7ef","Type":"ContainerDied","Data":"20224f8ee250930dc66e7ea06580765cc6feb0ae5c22039ce3ca1e74ef24b3e3"}
Jul 27 22:02:28.091804 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:28.091779    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-3483ea-predictor-66f8c699b4-4x94s_ad2bf35f-a2b6-4eba-9a1a-86577603b7ef/storage-initializer/0.log"
Jul 27 22:02:28.092383 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:28.091837    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-3483ea-predictor-66f8c699b4-4x94s" event={"ID":"ad2bf35f-a2b6-4eba-9a1a-86577603b7ef","Type":"ContainerStarted","Data":"96bcf8f27f42d85dce6ae7d32081f6de25cf009566887f442cf9a026f0aab1c8"}
Jul 27 22:02:29.877960 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:29.877928    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-3483ea-predictor-66f8c699b4-4x94s"]
Jul 27 22:02:29.878368 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:29.878134    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-3483ea-predictor-66f8c699b4-4x94s" podUID="ad2bf35f-a2b6-4eba-9a1a-86577603b7ef" containerName="storage-initializer" containerID="cri-o://96bcf8f27f42d85dce6ae7d32081f6de25cf009566887f442cf9a026f0aab1c8" gracePeriod=30
Jul 27 22:02:29.900110 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:29.900080    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:02:31.618603 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:31.618579    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-3483ea-predictor-66f8c699b4-4x94s_ad2bf35f-a2b6-4eba-9a1a-86577603b7ef/storage-initializer/1.log"
Jul 27 22:02:31.619014 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:31.618999    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-3483ea-predictor-66f8c699b4-4x94s_ad2bf35f-a2b6-4eba-9a1a-86577603b7ef/storage-initializer/0.log"
Jul 27 22:02:31.621479 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:31.621465    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-3483ea-predictor-66f8c699b4-4x94s"
Jul 27 22:02:31.707001 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:31.706918    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad2bf35f-a2b6-4eba-9a1a-86577603b7ef-cabundle-cert\") pod \"ad2bf35f-a2b6-4eba-9a1a-86577603b7ef\" (UID: \"ad2bf35f-a2b6-4eba-9a1a-86577603b7ef\") "
Jul 27 22:02:31.707001 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:31.706969    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad2bf35f-a2b6-4eba-9a1a-86577603b7ef-kserve-provision-location\") pod \"ad2bf35f-a2b6-4eba-9a1a-86577603b7ef\" (UID: \"ad2bf35f-a2b6-4eba-9a1a-86577603b7ef\") "
Jul 27 22:02:31.707206 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:31.707027    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxwkg\" (UniqueName: \"kubernetes.io/projected/ad2bf35f-a2b6-4eba-9a1a-86577603b7ef-kube-api-access-cxwkg\") pod \"ad2bf35f-a2b6-4eba-9a1a-86577603b7ef\" (UID: \"ad2bf35f-a2b6-4eba-9a1a-86577603b7ef\") "
Jul 27 22:02:31.707206 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:31.707049    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad2bf35f-a2b6-4eba-9a1a-86577603b7ef-kube-rbac-proxy-sar-config\") pod \"ad2bf35f-a2b6-4eba-9a1a-86577603b7ef\" (UID: \"ad2bf35f-a2b6-4eba-9a1a-86577603b7ef\") "
Jul 27 22:02:31.707206 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:31.707092    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad2bf35f-a2b6-4eba-9a1a-86577603b7ef-proxy-tls\") pod \"ad2bf35f-a2b6-4eba-9a1a-86577603b7ef\" (UID: \"ad2bf35f-a2b6-4eba-9a1a-86577603b7ef\") "
Jul 27 22:02:31.707368 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:31.707306    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad2bf35f-a2b6-4eba-9a1a-86577603b7ef-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ad2bf35f-a2b6-4eba-9a1a-86577603b7ef" (UID: "ad2bf35f-a2b6-4eba-9a1a-86577603b7ef"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:02:31.707431 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:31.707364    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad2bf35f-a2b6-4eba-9a1a-86577603b7ef-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ad2bf35f-a2b6-4eba-9a1a-86577603b7ef" (UID: "ad2bf35f-a2b6-4eba-9a1a-86577603b7ef"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:02:31.707479 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:31.707437    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad2bf35f-a2b6-4eba-9a1a-86577603b7ef-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ad2bf35f-a2b6-4eba-9a1a-86577603b7ef" (UID: "ad2bf35f-a2b6-4eba-9a1a-86577603b7ef"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:02:31.709109 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:31.709082    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad2bf35f-a2b6-4eba-9a1a-86577603b7ef-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ad2bf35f-a2b6-4eba-9a1a-86577603b7ef" (UID: "ad2bf35f-a2b6-4eba-9a1a-86577603b7ef"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 22:02:31.709251 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:31.709230    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad2bf35f-a2b6-4eba-9a1a-86577603b7ef-kube-api-access-cxwkg" (OuterVolumeSpecName: "kube-api-access-cxwkg") pod "ad2bf35f-a2b6-4eba-9a1a-86577603b7ef" (UID: "ad2bf35f-a2b6-4eba-9a1a-86577603b7ef"). InnerVolumeSpecName "kube-api-access-cxwkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:02:31.808667 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:31.808623    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad2bf35f-a2b6-4eba-9a1a-86577603b7ef-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:02:31.808667 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:31.808655    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad2bf35f-a2b6-4eba-9a1a-86577603b7ef-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:02:31.808667 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:31.808669    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad2bf35f-a2b6-4eba-9a1a-86577603b7ef-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:02:31.808949 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:31.808682    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxwkg\" (UniqueName: \"kubernetes.io/projected/ad2bf35f-a2b6-4eba-9a1a-86577603b7ef-kube-api-access-cxwkg\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:02:31.808949 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:31.808696    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad2bf35f-a2b6-4eba-9a1a-86577603b7ef-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:02:32.108812 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:32.108713    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-3483ea-predictor-66f8c699b4-4x94s_ad2bf35f-a2b6-4eba-9a1a-86577603b7ef/storage-initializer/1.log"
Jul 27 22:02:32.109182 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:32.109163    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-3483ea-predictor-66f8c699b4-4x94s_ad2bf35f-a2b6-4eba-9a1a-86577603b7ef/storage-initializer/0.log"
Jul 27 22:02:32.109240 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:32.109205    2577 generic.go:356] "Generic (PLEG): container finished" podID="ad2bf35f-a2b6-4eba-9a1a-86577603b7ef" containerID="96bcf8f27f42d85dce6ae7d32081f6de25cf009566887f442cf9a026f0aab1c8" exitCode=1
Jul 27 22:02:32.109281 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:32.109257    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-3483ea-predictor-66f8c699b4-4x94s" event={"ID":"ad2bf35f-a2b6-4eba-9a1a-86577603b7ef","Type":"ContainerDied","Data":"96bcf8f27f42d85dce6ae7d32081f6de25cf009566887f442cf9a026f0aab1c8"}
Jul 27 22:02:32.109281 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:32.109272    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-3483ea-predictor-66f8c699b4-4x94s"
Jul 27 22:02:32.109364 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:32.109309    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-3483ea-predictor-66f8c699b4-4x94s" event={"ID":"ad2bf35f-a2b6-4eba-9a1a-86577603b7ef","Type":"ContainerDied","Data":"2d9f18a3004d3b40d97dd2fb9cb2ddf1a0b5db06fe68d4447877b745add1242a"}
Jul 27 22:02:32.109364 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:32.109338    2577 scope.go:117] "RemoveContainer" containerID="96bcf8f27f42d85dce6ae7d32081f6de25cf009566887f442cf9a026f0aab1c8"
Jul 27 22:02:32.118053 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:32.118035    2577 scope.go:117] "RemoveContainer" containerID="20224f8ee250930dc66e7ea06580765cc6feb0ae5c22039ce3ca1e74ef24b3e3"
Jul 27 22:02:32.125473 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:32.125450    2577 scope.go:117] "RemoveContainer" containerID="96bcf8f27f42d85dce6ae7d32081f6de25cf009566887f442cf9a026f0aab1c8"
Jul 27 22:02:32.125739 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:02:32.125718    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96bcf8f27f42d85dce6ae7d32081f6de25cf009566887f442cf9a026f0aab1c8\": container with ID starting with 96bcf8f27f42d85dce6ae7d32081f6de25cf009566887f442cf9a026f0aab1c8 not found: ID does not exist" containerID="96bcf8f27f42d85dce6ae7d32081f6de25cf009566887f442cf9a026f0aab1c8"
Jul 27 22:02:32.125798 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:32.125749    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96bcf8f27f42d85dce6ae7d32081f6de25cf009566887f442cf9a026f0aab1c8"} err="failed to get container status \"96bcf8f27f42d85dce6ae7d32081f6de25cf009566887f442cf9a026f0aab1c8\": rpc error: code = NotFound desc = could not find container \"96bcf8f27f42d85dce6ae7d32081f6de25cf009566887f442cf9a026f0aab1c8\": container with ID starting with 96bcf8f27f42d85dce6ae7d32081f6de25cf009566887f442cf9a026f0aab1c8 not found: ID does not exist"
Jul 27 22:02:32.125798 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:32.125788    2577 scope.go:117] "RemoveContainer" containerID="20224f8ee250930dc66e7ea06580765cc6feb0ae5c22039ce3ca1e74ef24b3e3"
Jul 27 22:02:32.126017 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:02:32.126000    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20224f8ee250930dc66e7ea06580765cc6feb0ae5c22039ce3ca1e74ef24b3e3\": container with ID starting with 20224f8ee250930dc66e7ea06580765cc6feb0ae5c22039ce3ca1e74ef24b3e3 not found: ID does not exist" containerID="20224f8ee250930dc66e7ea06580765cc6feb0ae5c22039ce3ca1e74ef24b3e3"
Jul 27 22:02:32.126057 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:32.126022    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20224f8ee250930dc66e7ea06580765cc6feb0ae5c22039ce3ca1e74ef24b3e3"} err="failed to get container status \"20224f8ee250930dc66e7ea06580765cc6feb0ae5c22039ce3ca1e74ef24b3e3\": rpc error: code = NotFound desc = could not find container \"20224f8ee250930dc66e7ea06580765cc6feb0ae5c22039ce3ca1e74ef24b3e3\": container with ID starting with 20224f8ee250930dc66e7ea06580765cc6feb0ae5c22039ce3ca1e74ef24b3e3 not found: ID does not exist"
Jul 27 22:02:32.148054 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:32.148029    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-3483ea-predictor-66f8c699b4-4x94s"]
Jul 27 22:02:32.153017 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:32.152996    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-3483ea-predictor-66f8c699b4-4x94s"]
Jul 27 22:02:33.134510 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:33.134474    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad2bf35f-a2b6-4eba-9a1a-86577603b7ef" path="/var/lib/kubelet/pods/ad2bf35f-a2b6-4eba-9a1a-86577603b7ef/volumes"
Jul 27 22:02:35.053628 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:35.053588    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:02:35.080029 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:35.079998    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-qdfqd"]
Jul 27 22:02:35.080405 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:35.080391    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad2bf35f-a2b6-4eba-9a1a-86577603b7ef" containerName="storage-initializer"
Jul 27 22:02:35.080451 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:35.080408    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad2bf35f-a2b6-4eba-9a1a-86577603b7ef" containerName="storage-initializer"
Jul 27 22:02:35.080503 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:35.080492    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad2bf35f-a2b6-4eba-9a1a-86577603b7ef" containerName="storage-initializer"
Jul 27 22:02:35.080535 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:35.080508    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad2bf35f-a2b6-4eba-9a1a-86577603b7ef" containerName="storage-initializer"
Jul 27 22:02:35.080589 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:35.080578    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad2bf35f-a2b6-4eba-9a1a-86577603b7ef" containerName="storage-initializer"
Jul 27 22:02:35.080624 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:35.080591    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad2bf35f-a2b6-4eba-9a1a-86577603b7ef" containerName="storage-initializer"
Jul 27 22:02:35.085453 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:35.085435    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-qdfqd"
Jul 27 22:02:35.087553 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:35.087533    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
Jul 27 22:02:35.087806 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:35.087780    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 22:02:35.087928 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:35.087849    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
Jul 27 22:02:35.087992 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:35.087951    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 22:02:35.087992 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:35.087956    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v7c66\""
Jul 27 22:02:35.088678 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:35.088664    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 22:02:35.088750 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:35.088698    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 22:02:35.092147 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:35.092125    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-qdfqd"]
Jul 27 22:02:35.237019 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:35.236986    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f498710-867d-4519-8399-fd46146e1332-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-qdfqd\" (UID: \"9f498710-867d-4519-8399-fd46146e1332\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-qdfqd"
Jul 27 22:02:35.237234 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:35.237029    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f498710-867d-4519-8399-fd46146e1332-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-qdfqd\" (UID: \"9f498710-867d-4519-8399-fd46146e1332\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-qdfqd"
Jul 27 22:02:35.237234 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:35.237086    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkvjr\" (UniqueName: \"kubernetes.io/projected/9f498710-867d-4519-8399-fd46146e1332-kube-api-access-wkvjr\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-qdfqd\" (UID: \"9f498710-867d-4519-8399-fd46146e1332\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-qdfqd"
Jul 27 22:02:35.237234 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:35.237125    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f498710-867d-4519-8399-fd46146e1332-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-qdfqd\" (UID: \"9f498710-867d-4519-8399-fd46146e1332\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-qdfqd"
Jul 27 22:02:35.237234 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:35.237164    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f498710-867d-4519-8399-fd46146e1332-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-qdfqd\" (UID: \"9f498710-867d-4519-8399-fd46146e1332\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-qdfqd"
Jul 27 22:02:35.338385 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:35.338287    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f498710-867d-4519-8399-fd46146e1332-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-qdfqd\" (UID: \"9f498710-867d-4519-8399-fd46146e1332\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-qdfqd"
Jul 27 22:02:35.338385 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:35.338340    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkvjr\" (UniqueName: \"kubernetes.io/projected/9f498710-867d-4519-8399-fd46146e1332-kube-api-access-wkvjr\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-qdfqd\" (UID: \"9f498710-867d-4519-8399-fd46146e1332\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-qdfqd"
Jul 27 22:02:35.338385 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:35.338376    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f498710-867d-4519-8399-fd46146e1332-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-qdfqd\" (UID: \"9f498710-867d-4519-8399-fd46146e1332\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-qdfqd"
Jul 27 22:02:35.338677 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:35.338393    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f498710-867d-4519-8399-fd46146e1332-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-qdfqd\" (UID: \"9f498710-867d-4519-8399-fd46146e1332\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-qdfqd"
Jul 27 22:02:35.338677 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:35.338423    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f498710-867d-4519-8399-fd46146e1332-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-qdfqd\" (UID: \"9f498710-867d-4519-8399-fd46146e1332\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-qdfqd"
Jul 27 22:02:35.338677 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:02:35.338518    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-serving-cert: secret "isvc-predictive-sklearn-predictor-serving-cert" not found
Jul 27 22:02:35.338677 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:02:35.338593    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f498710-867d-4519-8399-fd46146e1332-proxy-tls podName:9f498710-867d-4519-8399-fd46146e1332 nodeName:}" failed. No retries permitted until 2026-07-27 22:02:35.838571614 +0000 UTC m=+3479.205761458 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9f498710-867d-4519-8399-fd46146e1332-proxy-tls") pod "isvc-predictive-sklearn-predictor-5f849d69df-qdfqd" (UID: "9f498710-867d-4519-8399-fd46146e1332") : secret "isvc-predictive-sklearn-predictor-serving-cert" not found
Jul 27 22:02:35.338926 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:35.338730    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f498710-867d-4519-8399-fd46146e1332-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-qdfqd\" (UID: \"9f498710-867d-4519-8399-fd46146e1332\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-qdfqd"
Jul 27 22:02:35.339176 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:35.339155    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f498710-867d-4519-8399-fd46146e1332-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-qdfqd\" (UID: \"9f498710-867d-4519-8399-fd46146e1332\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-qdfqd"
Jul 27 22:02:35.339229 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:35.339155    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f498710-867d-4519-8399-fd46146e1332-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-qdfqd\" (UID: \"9f498710-867d-4519-8399-fd46146e1332\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-qdfqd"
Jul 27 22:02:35.346644 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:35.346622    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkvjr\" (UniqueName: \"kubernetes.io/projected/9f498710-867d-4519-8399-fd46146e1332-kube-api-access-wkvjr\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-qdfqd\" (UID: \"9f498710-867d-4519-8399-fd46146e1332\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-qdfqd"
Jul 27 22:02:35.565792 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:35.565723    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:02:35.842561 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:35.842525    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f498710-867d-4519-8399-fd46146e1332-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-qdfqd\" (UID: \"9f498710-867d-4519-8399-fd46146e1332\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-qdfqd"
Jul 27 22:02:35.844854 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:35.844832    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f498710-867d-4519-8399-fd46146e1332-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-qdfqd\" (UID: \"9f498710-867d-4519-8399-fd46146e1332\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-qdfqd"
Jul 27 22:02:35.997140 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:35.997082    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-qdfqd"
Jul 27 22:02:36.123465 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:36.123374    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-qdfqd"]
Jul 27 22:02:36.126085 ip-10-0-142-219 kubenswrapper[2577]: W0727 22:02:36.126056    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f498710_867d_4519_8399_fd46146e1332.slice/crio-ccc529978963ce9ab059a412d7ef7b002b78340d7ba9d0433c9d9840b5c59e65 WatchSource:0}: Error finding container ccc529978963ce9ab059a412d7ef7b002b78340d7ba9d0433c9d9840b5c59e65: Status 404 returned error can't find the container with id ccc529978963ce9ab059a412d7ef7b002b78340d7ba9d0433c9d9840b5c59e65
Jul 27 22:02:37.134211 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:37.134180    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-qdfqd" event={"ID":"9f498710-867d-4519-8399-fd46146e1332","Type":"ContainerStarted","Data":"9976dcd698c406c62344bb467a319980c041a9c9a608023cbb870dc651c4a3b3"}
Jul 27 22:02:37.134567 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:37.134234    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-qdfqd" event={"ID":"9f498710-867d-4519-8399-fd46146e1332","Type":"ContainerStarted","Data":"ccc529978963ce9ab059a412d7ef7b002b78340d7ba9d0433c9d9840b5c59e65"}
Jul 27 22:02:40.143158 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:40.143122    2577 generic.go:356] "Generic (PLEG): container finished" podID="9f498710-867d-4519-8399-fd46146e1332" containerID="9976dcd698c406c62344bb467a319980c041a9c9a608023cbb870dc651c4a3b3" exitCode=0
Jul 27 22:02:40.143520 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:40.143198    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-qdfqd" event={"ID":"9f498710-867d-4519-8399-fd46146e1332","Type":"ContainerDied","Data":"9976dcd698c406c62344bb467a319980c041a9c9a608023cbb870dc651c4a3b3"}
Jul 27 22:02:40.144442 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:02:40.144428    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 22:03:01.254207 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:03:01.254174    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-qdfqd" event={"ID":"9f498710-867d-4519-8399-fd46146e1332","Type":"ContainerStarted","Data":"1001b09d3da9d2a8d3170a2a2e2098cfc6025b45accf50f6e3436de731bb47c5"}
Jul 27 22:03:01.254615 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:03:01.254235    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-qdfqd" event={"ID":"9f498710-867d-4519-8399-fd46146e1332","Type":"ContainerStarted","Data":"a1c76f7882ef4f90ffcb31d3edcfd252d5dac960471e750c1ce4c224e8ac4502"}
Jul 27 22:03:01.254615 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:03:01.254545    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-qdfqd"
Jul 27 22:03:01.254696 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:03:01.254671    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-qdfqd"
Jul 27 22:03:01.255934 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:03:01.255908    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-qdfqd" podUID="9f498710-867d-4519-8399-fd46146e1332" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Jul 27 22:03:01.278869 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:03:01.278815    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-qdfqd" podStartSLOduration=1.747010484 podStartE2EDuration="26.278800746s" totalImagesPullingTime="20.731057699s" totalInitContainerRuntime="3.800732563s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:02:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:03:01.277487271 +0000 UTC m=+3504.644677134" watchObservedRunningTime="2026-07-27 22:03:01.278800746 +0000 UTC m=+3504.645990612"
Jul 27 22:03:02.259312 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:03:02.259271    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-qdfqd" podUID="9f498710-867d-4519-8399-fd46146e1332" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Jul 27 22:03:07.264388 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:03:07.264355    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-qdfqd"
Jul 27 22:03:07.264898 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:03:07.264870    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-qdfqd" podUID="9f498710-867d-4519-8399-fd46146e1332" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Jul 27 22:03:17.265819 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:03:17.265754    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-qdfqd" podUID="9f498710-867d-4519-8399-fd46146e1332" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Jul 27 22:03:18.032569 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:03:18.032536    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:03:27.265462 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:03:27.265417    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-qdfqd" podUID="9f498710-867d-4519-8399-fd46146e1332" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Jul 27 22:03:37.264905 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:03:37.264865    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-qdfqd" podUID="9f498710-867d-4519-8399-fd46146e1332" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Jul 27 22:03:47.265731 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:03:47.265687    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-qdfqd" podUID="9f498710-867d-4519-8399-fd46146e1332" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Jul 27 22:03:57.265649 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:03:57.265604    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-qdfqd" podUID="9f498710-867d-4519-8399-fd46146e1332" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Jul 27 22:04:07.264823 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:07.264781    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-qdfqd" podUID="9f498710-867d-4519-8399-fd46146e1332" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Jul 27 22:04:17.265825 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:17.265795    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-qdfqd"
Jul 27 22:04:25.246472 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:25.246440    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-qdfqd"]
Jul 27 22:04:25.246883 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:25.246692    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-qdfqd" podUID="9f498710-867d-4519-8399-fd46146e1332" containerName="kserve-container" containerID="cri-o://a1c76f7882ef4f90ffcb31d3edcfd252d5dac960471e750c1ce4c224e8ac4502" gracePeriod=30
Jul 27 22:04:25.246883 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:25.246792    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-qdfqd" podUID="9f498710-867d-4519-8399-fd46146e1332" containerName="kube-rbac-proxy" containerID="cri-o://1001b09d3da9d2a8d3170a2a2e2098cfc6025b45accf50f6e3436de731bb47c5" gracePeriod=30
Jul 27 22:04:25.265011 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:25.264983    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:04:25.582225 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:25.582139    2577 generic.go:356] "Generic (PLEG): container finished" podID="9f498710-867d-4519-8399-fd46146e1332" containerID="1001b09d3da9d2a8d3170a2a2e2098cfc6025b45accf50f6e3436de731bb47c5" exitCode=2
Jul 27 22:04:25.582225 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:25.582198    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-qdfqd" event={"ID":"9f498710-867d-4519-8399-fd46146e1332","Type":"ContainerDied","Data":"1001b09d3da9d2a8d3170a2a2e2098cfc6025b45accf50f6e3436de731bb47c5"}
Jul 27 22:04:27.260555 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:27.260515    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-qdfqd" podUID="9f498710-867d-4519-8399-fd46146e1332" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.52:8643/healthz\": dial tcp 10.133.0.52:8643: connect: connection refused"
Jul 27 22:04:27.264795 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:27.264750    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-qdfqd" podUID="9f498710-867d-4519-8399-fd46146e1332" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Jul 27 22:04:29.829993 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:29.829964    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-qdfqd"
Jul 27 22:04:29.906294 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:29.906219    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f498710-867d-4519-8399-fd46146e1332-proxy-tls\") pod \"9f498710-867d-4519-8399-fd46146e1332\" (UID: \"9f498710-867d-4519-8399-fd46146e1332\") "
Jul 27 22:04:29.906294 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:29.906252    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f498710-867d-4519-8399-fd46146e1332-kube-rbac-proxy-sar-config\") pod \"9f498710-867d-4519-8399-fd46146e1332\" (UID: \"9f498710-867d-4519-8399-fd46146e1332\") "
Jul 27 22:04:29.906294 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:29.906280    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f498710-867d-4519-8399-fd46146e1332-kserve-provision-location\") pod \"9f498710-867d-4519-8399-fd46146e1332\" (UID: \"9f498710-867d-4519-8399-fd46146e1332\") "
Jul 27 22:04:29.906546 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:29.906298    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wkvjr\" (UniqueName: \"kubernetes.io/projected/9f498710-867d-4519-8399-fd46146e1332-kube-api-access-wkvjr\") pod \"9f498710-867d-4519-8399-fd46146e1332\" (UID: \"9f498710-867d-4519-8399-fd46146e1332\") "
Jul 27 22:04:29.906546 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:29.906312    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f498710-867d-4519-8399-fd46146e1332-cabundle-cert\") pod \"9f498710-867d-4519-8399-fd46146e1332\" (UID: \"9f498710-867d-4519-8399-fd46146e1332\") "
Jul 27 22:04:29.906664 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:29.906628    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f498710-867d-4519-8399-fd46146e1332-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9f498710-867d-4519-8399-fd46146e1332" (UID: "9f498710-867d-4519-8399-fd46146e1332"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:04:29.906664 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:29.906642    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f498710-867d-4519-8399-fd46146e1332-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9f498710-867d-4519-8399-fd46146e1332" (UID: "9f498710-867d-4519-8399-fd46146e1332"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:04:29.906805 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:29.906739    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f498710-867d-4519-8399-fd46146e1332-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9f498710-867d-4519-8399-fd46146e1332" (UID: "9f498710-867d-4519-8399-fd46146e1332"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:04:29.908360 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:29.908323    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f498710-867d-4519-8399-fd46146e1332-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9f498710-867d-4519-8399-fd46146e1332" (UID: "9f498710-867d-4519-8399-fd46146e1332"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 22:04:29.908457 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:29.908363    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f498710-867d-4519-8399-fd46146e1332-kube-api-access-wkvjr" (OuterVolumeSpecName: "kube-api-access-wkvjr") pod "9f498710-867d-4519-8399-fd46146e1332" (UID: "9f498710-867d-4519-8399-fd46146e1332"). InnerVolumeSpecName "kube-api-access-wkvjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:04:30.007062 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:30.007034    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f498710-867d-4519-8399-fd46146e1332-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:04:30.007062 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:30.007059    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f498710-867d-4519-8399-fd46146e1332-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:04:30.007221 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:30.007069    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f498710-867d-4519-8399-fd46146e1332-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:04:30.007221 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:30.007078    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkvjr\" (UniqueName: \"kubernetes.io/projected/9f498710-867d-4519-8399-fd46146e1332-kube-api-access-wkvjr\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:04:30.007221 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:30.007089    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f498710-867d-4519-8399-fd46146e1332-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:04:30.328726 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:30.328691    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:04:30.383466 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:30.383432    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bp5lk"]
Jul 27 22:04:30.383864 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:30.383849    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f498710-867d-4519-8399-fd46146e1332" containerName="storage-initializer"
Jul 27 22:04:30.383920 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:30.383867    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f498710-867d-4519-8399-fd46146e1332" containerName="storage-initializer"
Jul 27 22:04:30.383920 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:30.383881    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f498710-867d-4519-8399-fd46146e1332" containerName="kserve-container"
Jul 27 22:04:30.383920 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:30.383890    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f498710-867d-4519-8399-fd46146e1332" containerName="kserve-container"
Jul 27 22:04:30.383920 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:30.383905    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f498710-867d-4519-8399-fd46146e1332" containerName="kube-rbac-proxy"
Jul 27 22:04:30.383920 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:30.383915    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f498710-867d-4519-8399-fd46146e1332" containerName="kube-rbac-proxy"
Jul 27 22:04:30.384119 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:30.384013    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f498710-867d-4519-8399-fd46146e1332" containerName="kserve-container"
Jul 27 22:04:30.384119 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:30.384023    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f498710-867d-4519-8399-fd46146e1332" containerName="kube-rbac-proxy"
Jul 27 22:04:30.386597 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:30.386581    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bp5lk"
Jul 27 22:04:30.388695 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:30.388669    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
Jul 27 22:04:30.388948 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:30.388933    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
Jul 27 22:04:30.396872 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:30.396849    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bp5lk"]
Jul 27 22:04:30.409133 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:30.409108    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6485b22a-80db-4f40-b10a-e8a12b76f6cf-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-bp5lk\" (UID: \"6485b22a-80db-4f40-b10a-e8a12b76f6cf\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bp5lk"
Jul 27 22:04:30.409228 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:30.409166    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmpdb\" (UniqueName: \"kubernetes.io/projected/6485b22a-80db-4f40-b10a-e8a12b76f6cf-kube-api-access-kmpdb\") pod \"isvc-predictive-xgboost-predictor-695f978978-bp5lk\" (UID: \"6485b22a-80db-4f40-b10a-e8a12b76f6cf\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bp5lk"
Jul 27 22:04:30.409228 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:30.409205    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6485b22a-80db-4f40-b10a-e8a12b76f6cf-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-bp5lk\" (UID: \"6485b22a-80db-4f40-b10a-e8a12b76f6cf\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bp5lk"
Jul 27 22:04:30.409317 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:30.409231    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6485b22a-80db-4f40-b10a-e8a12b76f6cf-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-bp5lk\" (UID: \"6485b22a-80db-4f40-b10a-e8a12b76f6cf\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bp5lk"
Jul 27 22:04:30.409354 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:30.409327    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6485b22a-80db-4f40-b10a-e8a12b76f6cf-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-bp5lk\" (UID: \"6485b22a-80db-4f40-b10a-e8a12b76f6cf\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bp5lk"
Jul 27 22:04:30.510131 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:30.510099    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6485b22a-80db-4f40-b10a-e8a12b76f6cf-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-bp5lk\" (UID: \"6485b22a-80db-4f40-b10a-e8a12b76f6cf\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bp5lk"
Jul 27 22:04:30.510275 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:30.510155    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmpdb\" (UniqueName: \"kubernetes.io/projected/6485b22a-80db-4f40-b10a-e8a12b76f6cf-kube-api-access-kmpdb\") pod \"isvc-predictive-xgboost-predictor-695f978978-bp5lk\" (UID: \"6485b22a-80db-4f40-b10a-e8a12b76f6cf\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bp5lk"
Jul 27 22:04:30.510275 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:30.510198    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6485b22a-80db-4f40-b10a-e8a12b76f6cf-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-bp5lk\" (UID: \"6485b22a-80db-4f40-b10a-e8a12b76f6cf\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bp5lk"
Jul 27 22:04:30.510275 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:30.510229    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6485b22a-80db-4f40-b10a-e8a12b76f6cf-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-bp5lk\" (UID: \"6485b22a-80db-4f40-b10a-e8a12b76f6cf\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bp5lk"
Jul 27 22:04:30.510438 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:30.510273    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6485b22a-80db-4f40-b10a-e8a12b76f6cf-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-bp5lk\" (UID: \"6485b22a-80db-4f40-b10a-e8a12b76f6cf\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bp5lk"
Jul 27 22:04:30.510438 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:04:30.510351    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-serving-cert: secret "isvc-predictive-xgboost-predictor-serving-cert" not found
Jul 27 22:04:30.510438 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:04:30.510432    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6485b22a-80db-4f40-b10a-e8a12b76f6cf-proxy-tls podName:6485b22a-80db-4f40-b10a-e8a12b76f6cf nodeName:}" failed. No retries permitted until 2026-07-27 22:04:31.010399809 +0000 UTC m=+3594.377589655 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6485b22a-80db-4f40-b10a-e8a12b76f6cf-proxy-tls") pod "isvc-predictive-xgboost-predictor-695f978978-bp5lk" (UID: "6485b22a-80db-4f40-b10a-e8a12b76f6cf") : secret "isvc-predictive-xgboost-predictor-serving-cert" not found
Jul 27 22:04:30.510611 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:30.510508    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6485b22a-80db-4f40-b10a-e8a12b76f6cf-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-bp5lk\" (UID: \"6485b22a-80db-4f40-b10a-e8a12b76f6cf\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bp5lk"
Jul 27 22:04:30.511227 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:30.511203    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6485b22a-80db-4f40-b10a-e8a12b76f6cf-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-bp5lk\" (UID: \"6485b22a-80db-4f40-b10a-e8a12b76f6cf\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bp5lk"
Jul 27 22:04:30.511448 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:30.511424    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6485b22a-80db-4f40-b10a-e8a12b76f6cf-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-bp5lk\" (UID: \"6485b22a-80db-4f40-b10a-e8a12b76f6cf\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bp5lk"
Jul 27 22:04:30.520846 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:30.520828    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmpdb\" (UniqueName: \"kubernetes.io/projected/6485b22a-80db-4f40-b10a-e8a12b76f6cf-kube-api-access-kmpdb\") pod \"isvc-predictive-xgboost-predictor-695f978978-bp5lk\" (UID: \"6485b22a-80db-4f40-b10a-e8a12b76f6cf\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bp5lk"
Jul 27 22:04:30.602604 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:30.602530    2577 generic.go:356] "Generic (PLEG): container finished" podID="9f498710-867d-4519-8399-fd46146e1332" containerID="a1c76f7882ef4f90ffcb31d3edcfd252d5dac960471e750c1ce4c224e8ac4502" exitCode=0
Jul 27 22:04:30.602696 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:30.602604    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-qdfqd"
Jul 27 22:04:30.602696 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:30.602618    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-qdfqd" event={"ID":"9f498710-867d-4519-8399-fd46146e1332","Type":"ContainerDied","Data":"a1c76f7882ef4f90ffcb31d3edcfd252d5dac960471e750c1ce4c224e8ac4502"}
Jul 27 22:04:30.602796 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:30.602702    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-qdfqd" event={"ID":"9f498710-867d-4519-8399-fd46146e1332","Type":"ContainerDied","Data":"ccc529978963ce9ab059a412d7ef7b002b78340d7ba9d0433c9d9840b5c59e65"}
Jul 27 22:04:30.602796 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:30.602729    2577 scope.go:117] "RemoveContainer" containerID="1001b09d3da9d2a8d3170a2a2e2098cfc6025b45accf50f6e3436de731bb47c5"
Jul 27 22:04:30.611206 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:30.611187    2577 scope.go:117] "RemoveContainer" containerID="a1c76f7882ef4f90ffcb31d3edcfd252d5dac960471e750c1ce4c224e8ac4502"
Jul 27 22:04:30.618494 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:30.618473    2577 scope.go:117] "RemoveContainer" containerID="9976dcd698c406c62344bb467a319980c041a9c9a608023cbb870dc651c4a3b3"
Jul 27 22:04:30.625918 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:30.625894    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-qdfqd"]
Jul 27 22:04:30.627194 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:30.627174    2577 scope.go:117] "RemoveContainer" containerID="1001b09d3da9d2a8d3170a2a2e2098cfc6025b45accf50f6e3436de731bb47c5"
Jul 27 22:04:30.627477 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:04:30.627456    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1001b09d3da9d2a8d3170a2a2e2098cfc6025b45accf50f6e3436de731bb47c5\": container with ID starting with 1001b09d3da9d2a8d3170a2a2e2098cfc6025b45accf50f6e3436de731bb47c5 not found: ID does not exist" containerID="1001b09d3da9d2a8d3170a2a2e2098cfc6025b45accf50f6e3436de731bb47c5"
Jul 27 22:04:30.627562 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:30.627483    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1001b09d3da9d2a8d3170a2a2e2098cfc6025b45accf50f6e3436de731bb47c5"} err="failed to get container status \"1001b09d3da9d2a8d3170a2a2e2098cfc6025b45accf50f6e3436de731bb47c5\": rpc error: code = NotFound desc = could not find container \"1001b09d3da9d2a8d3170a2a2e2098cfc6025b45accf50f6e3436de731bb47c5\": container with ID starting with 1001b09d3da9d2a8d3170a2a2e2098cfc6025b45accf50f6e3436de731bb47c5 not found: ID does not exist"
Jul 27 22:04:30.627562 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:30.627501    2577 scope.go:117] "RemoveContainer" containerID="a1c76f7882ef4f90ffcb31d3edcfd252d5dac960471e750c1ce4c224e8ac4502"
Jul 27 22:04:30.627856 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:04:30.627742    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1c76f7882ef4f90ffcb31d3edcfd252d5dac960471e750c1ce4c224e8ac4502\": container with ID starting with a1c76f7882ef4f90ffcb31d3edcfd252d5dac960471e750c1ce4c224e8ac4502 not found: ID does not exist" containerID="a1c76f7882ef4f90ffcb31d3edcfd252d5dac960471e750c1ce4c224e8ac4502"
Jul 27 22:04:30.627856 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:30.627787    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1c76f7882ef4f90ffcb31d3edcfd252d5dac960471e750c1ce4c224e8ac4502"} err="failed to get container status \"a1c76f7882ef4f90ffcb31d3edcfd252d5dac960471e750c1ce4c224e8ac4502\": rpc error: code = NotFound desc = could not find container \"a1c76f7882ef4f90ffcb31d3edcfd252d5dac960471e750c1ce4c224e8ac4502\": container with ID starting with a1c76f7882ef4f90ffcb31d3edcfd252d5dac960471e750c1ce4c224e8ac4502 not found: ID does not exist"
Jul 27 22:04:30.627856 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:30.627803    2577 scope.go:117] "RemoveContainer" containerID="9976dcd698c406c62344bb467a319980c041a9c9a608023cbb870dc651c4a3b3"
Jul 27 22:04:30.628086 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:04:30.628062    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9976dcd698c406c62344bb467a319980c041a9c9a608023cbb870dc651c4a3b3\": container with ID starting with 9976dcd698c406c62344bb467a319980c041a9c9a608023cbb870dc651c4a3b3 not found: ID does not exist" containerID="9976dcd698c406c62344bb467a319980c041a9c9a608023cbb870dc651c4a3b3"
Jul 27 22:04:30.628131 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:30.628091    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9976dcd698c406c62344bb467a319980c041a9c9a608023cbb870dc651c4a3b3"} err="failed to get container status \"9976dcd698c406c62344bb467a319980c041a9c9a608023cbb870dc651c4a3b3\": rpc error: code = NotFound desc = could not find container \"9976dcd698c406c62344bb467a319980c041a9c9a608023cbb870dc651c4a3b3\": container with ID starting with 9976dcd698c406c62344bb467a319980c041a9c9a608023cbb870dc651c4a3b3 not found: ID does not exist"
Jul 27 22:04:30.628806 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:30.628787    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-qdfqd"]
Jul 27 22:04:30.804071 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:30.804037    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:04:31.014415 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:31.014363    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6485b22a-80db-4f40-b10a-e8a12b76f6cf-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-bp5lk\" (UID: \"6485b22a-80db-4f40-b10a-e8a12b76f6cf\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bp5lk"
Jul 27 22:04:31.014862 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:04:31.014513    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-serving-cert: secret "isvc-predictive-xgboost-predictor-serving-cert" not found
Jul 27 22:04:31.014862 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:04:31.014603    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6485b22a-80db-4f40-b10a-e8a12b76f6cf-proxy-tls podName:6485b22a-80db-4f40-b10a-e8a12b76f6cf nodeName:}" failed. No retries permitted until 2026-07-27 22:04:32.014587389 +0000 UTC m=+3595.381777233 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6485b22a-80db-4f40-b10a-e8a12b76f6cf-proxy-tls") pod "isvc-predictive-xgboost-predictor-695f978978-bp5lk" (UID: "6485b22a-80db-4f40-b10a-e8a12b76f6cf") : secret "isvc-predictive-xgboost-predictor-serving-cert" not found
Jul 27 22:04:31.127871 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:31.127840    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:04:31.135381 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:31.135354    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f498710-867d-4519-8399-fd46146e1332" path="/var/lib/kubelet/pods/9f498710-867d-4519-8399-fd46146e1332/volumes"
Jul 27 22:04:32.023715 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:32.023687    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6485b22a-80db-4f40-b10a-e8a12b76f6cf-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-bp5lk\" (UID: \"6485b22a-80db-4f40-b10a-e8a12b76f6cf\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bp5lk"
Jul 27 22:04:32.026021 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:32.025993    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6485b22a-80db-4f40-b10a-e8a12b76f6cf-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-bp5lk\" (UID: \"6485b22a-80db-4f40-b10a-e8a12b76f6cf\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bp5lk"
Jul 27 22:04:32.196777 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:32.196732    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bp5lk"
Jul 27 22:04:32.329717 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:32.329684    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bp5lk"]
Jul 27 22:04:32.333046 ip-10-0-142-219 kubenswrapper[2577]: W0727 22:04:32.333015    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6485b22a_80db_4f40_b10a_e8a12b76f6cf.slice/crio-69aad5789c07c58db8301187bab25bf1f927e0dd18c1a682ada56206b62397ae WatchSource:0}: Error finding container 69aad5789c07c58db8301187bab25bf1f927e0dd18c1a682ada56206b62397ae: Status 404 returned error can't find the container with id 69aad5789c07c58db8301187bab25bf1f927e0dd18c1a682ada56206b62397ae
Jul 27 22:04:32.616147 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:32.616051    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bp5lk" event={"ID":"6485b22a-80db-4f40-b10a-e8a12b76f6cf","Type":"ContainerStarted","Data":"cd55c0036038dc285902d383df1acc00b88b8d66099024ac973e4515719c8b63"}
Jul 27 22:04:32.616147 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:32.616122    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bp5lk" event={"ID":"6485b22a-80db-4f40-b10a-e8a12b76f6cf","Type":"ContainerStarted","Data":"69aad5789c07c58db8301187bab25bf1f927e0dd18c1a682ada56206b62397ae"}
Jul 27 22:04:36.637853 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:36.637819    2577 generic.go:356] "Generic (PLEG): container finished" podID="6485b22a-80db-4f40-b10a-e8a12b76f6cf" containerID="cd55c0036038dc285902d383df1acc00b88b8d66099024ac973e4515719c8b63" exitCode=0
Jul 27 22:04:36.638236 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:36.637901    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bp5lk" event={"ID":"6485b22a-80db-4f40-b10a-e8a12b76f6cf","Type":"ContainerDied","Data":"cd55c0036038dc285902d383df1acc00b88b8d66099024ac973e4515719c8b63"}
Jul 27 22:04:37.644111 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:37.644077    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bp5lk" event={"ID":"6485b22a-80db-4f40-b10a-e8a12b76f6cf","Type":"ContainerStarted","Data":"5df5422beb51cf35ecacd35bb143e1cabb5bac96507d980e379aa2060dd3af7d"}
Jul 27 22:04:37.644507 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:37.644134    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bp5lk" event={"ID":"6485b22a-80db-4f40-b10a-e8a12b76f6cf","Type":"ContainerStarted","Data":"a85598bbafa02b5c25f421c3769e15237d507b71238dfdcbe033837c08aee357"}
Jul 27 22:04:37.644507 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:37.644429    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bp5lk"
Jul 27 22:04:37.644589 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:37.644567    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bp5lk"
Jul 27 22:04:37.645911 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:37.645885    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bp5lk" podUID="6485b22a-80db-4f40-b10a-e8a12b76f6cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Jul 27 22:04:37.663867 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:37.663829    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bp5lk" podStartSLOduration=4.157466686 podStartE2EDuration="7.663815099s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.506348413s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:04:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:04:37.662283459 +0000 UTC m=+3601.029473326" watchObservedRunningTime="2026-07-27 22:04:37.663815099 +0000 UTC m=+3601.031004965"
Jul 27 22:04:38.648554 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:38.648511    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bp5lk" podUID="6485b22a-80db-4f40-b10a-e8a12b76f6cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Jul 27 22:04:43.652857 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:43.652828    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bp5lk"
Jul 27 22:04:43.653408 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:43.653379    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bp5lk" podUID="6485b22a-80db-4f40-b10a-e8a12b76f6cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Jul 27 22:04:53.653678 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:04:53.653636    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bp5lk" podUID="6485b22a-80db-4f40-b10a-e8a12b76f6cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Jul 27 22:05:03.654012 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:05:03.653971    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bp5lk" podUID="6485b22a-80db-4f40-b10a-e8a12b76f6cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Jul 27 22:05:13.653466 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:05:13.653429    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bp5lk" podUID="6485b22a-80db-4f40-b10a-e8a12b76f6cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Jul 27 22:05:23.654264 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:05:23.654221    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bp5lk" podUID="6485b22a-80db-4f40-b10a-e8a12b76f6cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Jul 27 22:05:33.654080 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:05:33.654040    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bp5lk" podUID="6485b22a-80db-4f40-b10a-e8a12b76f6cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Jul 27 22:05:43.653441 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:05:43.653402    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bp5lk" podUID="6485b22a-80db-4f40-b10a-e8a12b76f6cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Jul 27 22:05:53.653913 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:05:53.653885    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bp5lk"
Jul 27 22:06:00.537430 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:00.537345    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bp5lk"]
Jul 27 22:06:00.537906 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:00.537646    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bp5lk" podUID="6485b22a-80db-4f40-b10a-e8a12b76f6cf" containerName="kserve-container" containerID="cri-o://a85598bbafa02b5c25f421c3769e15237d507b71238dfdcbe033837c08aee357" gracePeriod=30
Jul 27 22:06:00.537906 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:00.537691    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bp5lk" podUID="6485b22a-80db-4f40-b10a-e8a12b76f6cf" containerName="kube-rbac-proxy" containerID="cri-o://5df5422beb51cf35ecacd35bb143e1cabb5bac96507d980e379aa2060dd3af7d" gracePeriod=30
Jul 27 22:06:00.571973 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:00.571946    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:06:00.965625 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:00.965587    2577 generic.go:356] "Generic (PLEG): container finished" podID="6485b22a-80db-4f40-b10a-e8a12b76f6cf" containerID="5df5422beb51cf35ecacd35bb143e1cabb5bac96507d980e379aa2060dd3af7d" exitCode=2
Jul 27 22:06:00.965625 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:00.965625    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bp5lk" event={"ID":"6485b22a-80db-4f40-b10a-e8a12b76f6cf","Type":"ContainerDied","Data":"5df5422beb51cf35ecacd35bb143e1cabb5bac96507d980e379aa2060dd3af7d"}
Jul 27 22:06:03.649258 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:03.649216    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bp5lk" podUID="6485b22a-80db-4f40-b10a-e8a12b76f6cf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.53:8643/healthz\": dial tcp 10.133.0.53:8643: connect: connection refused"
Jul 27 22:06:03.653630 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:03.653601    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bp5lk" podUID="6485b22a-80db-4f40-b10a-e8a12b76f6cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Jul 27 22:06:04.983485 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:04.983445    2577 generic.go:356] "Generic (PLEG): container finished" podID="6485b22a-80db-4f40-b10a-e8a12b76f6cf" containerID="a85598bbafa02b5c25f421c3769e15237d507b71238dfdcbe033837c08aee357" exitCode=0
Jul 27 22:06:04.983862 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:04.983522    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bp5lk" event={"ID":"6485b22a-80db-4f40-b10a-e8a12b76f6cf","Type":"ContainerDied","Data":"a85598bbafa02b5c25f421c3769e15237d507b71238dfdcbe033837c08aee357"}
Jul 27 22:06:05.077311 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:05.077289    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bp5lk"
Jul 27 22:06:05.171575 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:05.171556    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmpdb\" (UniqueName: \"kubernetes.io/projected/6485b22a-80db-4f40-b10a-e8a12b76f6cf-kube-api-access-kmpdb\") pod \"6485b22a-80db-4f40-b10a-e8a12b76f6cf\" (UID: \"6485b22a-80db-4f40-b10a-e8a12b76f6cf\") "
Jul 27 22:06:05.171723 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:05.171602    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6485b22a-80db-4f40-b10a-e8a12b76f6cf-cabundle-cert\") pod \"6485b22a-80db-4f40-b10a-e8a12b76f6cf\" (UID: \"6485b22a-80db-4f40-b10a-e8a12b76f6cf\") "
Jul 27 22:06:05.171723 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:05.171637    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6485b22a-80db-4f40-b10a-e8a12b76f6cf-proxy-tls\") pod \"6485b22a-80db-4f40-b10a-e8a12b76f6cf\" (UID: \"6485b22a-80db-4f40-b10a-e8a12b76f6cf\") "
Jul 27 22:06:05.171839 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:05.171743    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6485b22a-80db-4f40-b10a-e8a12b76f6cf-kserve-provision-location\") pod \"6485b22a-80db-4f40-b10a-e8a12b76f6cf\" (UID: \"6485b22a-80db-4f40-b10a-e8a12b76f6cf\") "
Jul 27 22:06:05.171839 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:05.171824    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6485b22a-80db-4f40-b10a-e8a12b76f6cf-kube-rbac-proxy-sar-config\") pod \"6485b22a-80db-4f40-b10a-e8a12b76f6cf\" (UID: \"6485b22a-80db-4f40-b10a-e8a12b76f6cf\") "
Jul 27 22:06:05.172117 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:05.172026    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6485b22a-80db-4f40-b10a-e8a12b76f6cf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6485b22a-80db-4f40-b10a-e8a12b76f6cf" (UID: "6485b22a-80db-4f40-b10a-e8a12b76f6cf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:06:05.172187 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:05.172120    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6485b22a-80db-4f40-b10a-e8a12b76f6cf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6485b22a-80db-4f40-b10a-e8a12b76f6cf" (UID: "6485b22a-80db-4f40-b10a-e8a12b76f6cf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:06:05.172229 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:05.172206    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6485b22a-80db-4f40-b10a-e8a12b76f6cf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6485b22a-80db-4f40-b10a-e8a12b76f6cf" (UID: "6485b22a-80db-4f40-b10a-e8a12b76f6cf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:06:05.173705 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:05.173684    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6485b22a-80db-4f40-b10a-e8a12b76f6cf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6485b22a-80db-4f40-b10a-e8a12b76f6cf" (UID: "6485b22a-80db-4f40-b10a-e8a12b76f6cf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 22:06:05.173819 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:05.173721    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6485b22a-80db-4f40-b10a-e8a12b76f6cf-kube-api-access-kmpdb" (OuterVolumeSpecName: "kube-api-access-kmpdb") pod "6485b22a-80db-4f40-b10a-e8a12b76f6cf" (UID: "6485b22a-80db-4f40-b10a-e8a12b76f6cf"). InnerVolumeSpecName "kube-api-access-kmpdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:06:05.272641 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:05.272609    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6485b22a-80db-4f40-b10a-e8a12b76f6cf-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:06:05.272641 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:05.272634    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6485b22a-80db-4f40-b10a-e8a12b76f6cf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:06:05.272641 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:05.272644    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmpdb\" (UniqueName: \"kubernetes.io/projected/6485b22a-80db-4f40-b10a-e8a12b76f6cf-kube-api-access-kmpdb\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:06:05.272886 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:05.272654    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6485b22a-80db-4f40-b10a-e8a12b76f6cf-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:06:05.272886 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:05.272664    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6485b22a-80db-4f40-b10a-e8a12b76f6cf-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:06:05.656791 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:05.656684    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:06:05.723487 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:05.723451    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z"]
Jul 27 22:06:05.723731 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:05.723720    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6485b22a-80db-4f40-b10a-e8a12b76f6cf" containerName="kserve-container"
Jul 27 22:06:05.723816 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:05.723735    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6485b22a-80db-4f40-b10a-e8a12b76f6cf" containerName="kserve-container"
Jul 27 22:06:05.723816 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:05.723754    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6485b22a-80db-4f40-b10a-e8a12b76f6cf" containerName="storage-initializer"
Jul 27 22:06:05.723816 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:05.723771    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6485b22a-80db-4f40-b10a-e8a12b76f6cf" containerName="storage-initializer"
Jul 27 22:06:05.723816 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:05.723783    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6485b22a-80db-4f40-b10a-e8a12b76f6cf" containerName="kube-rbac-proxy"
Jul 27 22:06:05.723816 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:05.723792    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6485b22a-80db-4f40-b10a-e8a12b76f6cf" containerName="kube-rbac-proxy"
Jul 27 22:06:05.723976 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:05.723844    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6485b22a-80db-4f40-b10a-e8a12b76f6cf" containerName="kube-rbac-proxy"
Jul 27 22:06:05.723976 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:05.723852    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6485b22a-80db-4f40-b10a-e8a12b76f6cf" containerName="kserve-container"
Jul 27 22:06:05.727068 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:05.727052    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z"
Jul 27 22:06:05.731203 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:05.731173    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
Jul 27 22:06:05.731337 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:05.731193    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
Jul 27 22:06:05.741175 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:05.741153    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z"]
Jul 27 22:06:05.878091 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:05.878059    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fff81b2c-5ba5-411d-9674-d8f00a824498-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z\" (UID: \"fff81b2c-5ba5-411d-9674-d8f00a824498\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z"
Jul 27 22:06:05.878296 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:05.878098    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fff81b2c-5ba5-411d-9674-d8f00a824498-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z\" (UID: \"fff81b2c-5ba5-411d-9674-d8f00a824498\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z"
Jul 27 22:06:05.878296 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:05.878196    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fff81b2c-5ba5-411d-9674-d8f00a824498-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z\" (UID: \"fff81b2c-5ba5-411d-9674-d8f00a824498\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z"
Jul 27 22:06:05.878296 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:05.878238    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fff81b2c-5ba5-411d-9674-d8f00a824498-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z\" (UID: \"fff81b2c-5ba5-411d-9674-d8f00a824498\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z"
Jul 27 22:06:05.878296 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:05.878262    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz87d\" (UniqueName: \"kubernetes.io/projected/fff81b2c-5ba5-411d-9674-d8f00a824498-kube-api-access-zz87d\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z\" (UID: \"fff81b2c-5ba5-411d-9674-d8f00a824498\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z"
Jul 27 22:06:05.903288 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:05.903259    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:06:05.978655 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:05.978630    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fff81b2c-5ba5-411d-9674-d8f00a824498-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z\" (UID: \"fff81b2c-5ba5-411d-9674-d8f00a824498\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z"
Jul 27 22:06:05.978833 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:05.978666    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fff81b2c-5ba5-411d-9674-d8f00a824498-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z\" (UID: \"fff81b2c-5ba5-411d-9674-d8f00a824498\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z"
Jul 27 22:06:05.978833 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:05.978704    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fff81b2c-5ba5-411d-9674-d8f00a824498-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z\" (UID: \"fff81b2c-5ba5-411d-9674-d8f00a824498\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z"
Jul 27 22:06:05.978833 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:05.978726    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fff81b2c-5ba5-411d-9674-d8f00a824498-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z\" (UID: \"fff81b2c-5ba5-411d-9674-d8f00a824498\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z"
Jul 27 22:06:05.978833 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:05.978743    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz87d\" (UniqueName: \"kubernetes.io/projected/fff81b2c-5ba5-411d-9674-d8f00a824498-kube-api-access-zz87d\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z\" (UID: \"fff81b2c-5ba5-411d-9674-d8f00a824498\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z"
Jul 27 22:06:05.979196 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:05.979170    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fff81b2c-5ba5-411d-9674-d8f00a824498-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z\" (UID: \"fff81b2c-5ba5-411d-9674-d8f00a824498\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z"
Jul 27 22:06:05.979494 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:05.979477    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fff81b2c-5ba5-411d-9674-d8f00a824498-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z\" (UID: \"fff81b2c-5ba5-411d-9674-d8f00a824498\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z"
Jul 27 22:06:05.979618 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:05.979596    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fff81b2c-5ba5-411d-9674-d8f00a824498-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z\" (UID: \"fff81b2c-5ba5-411d-9674-d8f00a824498\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z"
Jul 27 22:06:05.981230 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:05.981203    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fff81b2c-5ba5-411d-9674-d8f00a824498-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z\" (UID: \"fff81b2c-5ba5-411d-9674-d8f00a824498\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z"
Jul 27 22:06:05.988797 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:05.988779    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz87d\" (UniqueName: \"kubernetes.io/projected/fff81b2c-5ba5-411d-9674-d8f00a824498-kube-api-access-zz87d\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z\" (UID: \"fff81b2c-5ba5-411d-9674-d8f00a824498\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z"
Jul 27 22:06:05.990372 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:05.990351    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bp5lk" event={"ID":"6485b22a-80db-4f40-b10a-e8a12b76f6cf","Type":"ContainerDied","Data":"69aad5789c07c58db8301187bab25bf1f927e0dd18c1a682ada56206b62397ae"}
Jul 27 22:06:05.990455 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:05.990399    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bp5lk"
Jul 27 22:06:05.990455 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:05.990420    2577 scope.go:117] "RemoveContainer" containerID="5df5422beb51cf35ecacd35bb143e1cabb5bac96507d980e379aa2060dd3af7d"
Jul 27 22:06:06.001046 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:06.001031    2577 scope.go:117] "RemoveContainer" containerID="a85598bbafa02b5c25f421c3769e15237d507b71238dfdcbe033837c08aee357"
Jul 27 22:06:06.008414 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:06.008397    2577 scope.go:117] "RemoveContainer" containerID="cd55c0036038dc285902d383df1acc00b88b8d66099024ac973e4515719c8b63"
Jul 27 22:06:06.009007 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:06:06.008988    2577 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6485b22a_80db_4f40_b10a_e8a12b76f6cf.slice/crio-69aad5789c07c58db8301187bab25bf1f927e0dd18c1a682ada56206b62397ae\": RecentStats: unable to find data in memory cache]"
Jul 27 22:06:06.022622 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:06.022599    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bp5lk"]
Jul 27 22:06:06.029482 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:06.029458    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-bp5lk"]
Jul 27 22:06:06.036655 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:06.036641    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z"
Jul 27 22:06:06.165459 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:06.165434    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z"]
Jul 27 22:06:06.167880 ip-10-0-142-219 kubenswrapper[2577]: W0727 22:06:06.167855    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfff81b2c_5ba5_411d_9674_d8f00a824498.slice/crio-58a393aaea7f763af0d301d6d14952a56e8de99c9e67e48cdc634cb167d8f1b4 WatchSource:0}: Error finding container 58a393aaea7f763af0d301d6d14952a56e8de99c9e67e48cdc634cb167d8f1b4: Status 404 returned error can't find the container with id 58a393aaea7f763af0d301d6d14952a56e8de99c9e67e48cdc634cb167d8f1b4
Jul 27 22:06:06.999593 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:06.999558    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z" event={"ID":"fff81b2c-5ba5-411d-9674-d8f00a824498","Type":"ContainerStarted","Data":"8dcf66ee723c536e34cb1dbf32b9b345a589b53473565f52baeda68262a3f284"}
Jul 27 22:06:07.000073 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:06.999652    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z" event={"ID":"fff81b2c-5ba5-411d-9674-d8f00a824498","Type":"ContainerStarted","Data":"58a393aaea7f763af0d301d6d14952a56e8de99c9e67e48cdc634cb167d8f1b4"}
Jul 27 22:06:07.135712 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:07.135682    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6485b22a-80db-4f40-b10a-e8a12b76f6cf" path="/var/lib/kubelet/pods/6485b22a-80db-4f40-b10a-e8a12b76f6cf/volumes"
Jul 27 22:06:10.012801 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:10.012741    2577 generic.go:356] "Generic (PLEG): container finished" podID="fff81b2c-5ba5-411d-9674-d8f00a824498" containerID="8dcf66ee723c536e34cb1dbf32b9b345a589b53473565f52baeda68262a3f284" exitCode=0
Jul 27 22:06:10.013207 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:10.012796    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z" event={"ID":"fff81b2c-5ba5-411d-9674-d8f00a824498","Type":"ContainerDied","Data":"8dcf66ee723c536e34cb1dbf32b9b345a589b53473565f52baeda68262a3f284"}
Jul 27 22:06:11.019584 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:11.019549    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z" event={"ID":"fff81b2c-5ba5-411d-9674-d8f00a824498","Type":"ContainerStarted","Data":"7f1c4e7c0afd7e8e23e52c3a6a7d532eeaf9457ec3c3162fa154d215bd558d06"}
Jul 27 22:06:11.020086 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:11.019621    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z" event={"ID":"fff81b2c-5ba5-411d-9674-d8f00a824498","Type":"ContainerStarted","Data":"3f807377a14a0a4952b4995aa76ba8f5cc0f00cc554104320d2da42bde0bb064"}
Jul 27 22:06:11.020086 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:11.019936    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z"
Jul 27 22:06:11.020086 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:11.019983    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z"
Jul 27 22:06:11.021249 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:11.021225    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z" podUID="fff81b2c-5ba5-411d-9674-d8f00a824498" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 27 22:06:11.043073 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:11.043033    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z" podStartSLOduration=2.418444285 podStartE2EDuration="6.043020001s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.624575716s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:06:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:06:11.042702739 +0000 UTC m=+3694.409892610" watchObservedRunningTime="2026-07-27 22:06:11.043020001 +0000 UTC m=+3694.410209867"
Jul 27 22:06:12.023478 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:12.023440    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z" podUID="fff81b2c-5ba5-411d-9674-d8f00a824498" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 27 22:06:17.028225 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:17.028197    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z"
Jul 27 22:06:17.028728 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:17.028699    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z" podUID="fff81b2c-5ba5-411d-9674-d8f00a824498" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 27 22:06:27.029102 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:27.029062    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z" podUID="fff81b2c-5ba5-411d-9674-d8f00a824498" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 27 22:06:37.029710 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:37.029674    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z" podUID="fff81b2c-5ba5-411d-9674-d8f00a824498" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 27 22:06:47.028970 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:47.028928    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z" podUID="fff81b2c-5ba5-411d-9674-d8f00a824498" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 27 22:06:57.028945 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:06:57.028901    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z" podUID="fff81b2c-5ba5-411d-9674-d8f00a824498" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 27 22:07:07.028832 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:07.028793    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z" podUID="fff81b2c-5ba5-411d-9674-d8f00a824498" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 27 22:07:17.029069 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:17.029029    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z" podUID="fff81b2c-5ba5-411d-9674-d8f00a824498" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 27 22:07:27.028925 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:27.028896    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z"
Jul 27 22:07:35.812874 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:35.812785    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z"]
Jul 27 22:07:35.813284 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:35.813097    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z" podUID="fff81b2c-5ba5-411d-9674-d8f00a824498" containerName="kube-rbac-proxy" containerID="cri-o://7f1c4e7c0afd7e8e23e52c3a6a7d532eeaf9457ec3c3162fa154d215bd558d06" gracePeriod=30
Jul 27 22:07:35.813284 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:35.813076    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z" podUID="fff81b2c-5ba5-411d-9674-d8f00a824498" containerName="kserve-container" containerID="cri-o://3f807377a14a0a4952b4995aa76ba8f5cc0f00cc554104320d2da42bde0bb064" gracePeriod=30
Jul 27 22:07:35.838905 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:35.838878    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:07:36.339845 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:36.339810    2577 generic.go:356] "Generic (PLEG): container finished" podID="fff81b2c-5ba5-411d-9674-d8f00a824498" containerID="7f1c4e7c0afd7e8e23e52c3a6a7d532eeaf9457ec3c3162fa154d215bd558d06" exitCode=2
Jul 27 22:07:36.339845 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:36.339847    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z" event={"ID":"fff81b2c-5ba5-411d-9674-d8f00a824498","Type":"ContainerDied","Data":"7f1c4e7c0afd7e8e23e52c3a6a7d532eeaf9457ec3c3162fa154d215bd558d06"}
Jul 27 22:07:37.024160 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:37.024111    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z" podUID="fff81b2c-5ba5-411d-9674-d8f00a824498" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.54:8643/healthz\": dial tcp 10.133.0.54:8643: connect: connection refused"
Jul 27 22:07:37.029570 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:37.029540    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z" podUID="fff81b2c-5ba5-411d-9674-d8f00a824498" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 27 22:07:40.852898 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:40.852873    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z"
Jul 27 22:07:40.915664 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:40.915632    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:07:40.934967 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:40.934933    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fff81b2c-5ba5-411d-9674-d8f00a824498-cabundle-cert\") pod \"fff81b2c-5ba5-411d-9674-d8f00a824498\" (UID: \"fff81b2c-5ba5-411d-9674-d8f00a824498\") "
Jul 27 22:07:40.935165 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:40.934993    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fff81b2c-5ba5-411d-9674-d8f00a824498-proxy-tls\") pod \"fff81b2c-5ba5-411d-9674-d8f00a824498\" (UID: \"fff81b2c-5ba5-411d-9674-d8f00a824498\") "
Jul 27 22:07:40.935165 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:40.935036    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fff81b2c-5ba5-411d-9674-d8f00a824498-kube-rbac-proxy-sar-config\") pod \"fff81b2c-5ba5-411d-9674-d8f00a824498\" (UID: \"fff81b2c-5ba5-411d-9674-d8f00a824498\") "
Jul 27 22:07:40.935165 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:40.935079    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fff81b2c-5ba5-411d-9674-d8f00a824498-kserve-provision-location\") pod \"fff81b2c-5ba5-411d-9674-d8f00a824498\" (UID: \"fff81b2c-5ba5-411d-9674-d8f00a824498\") "
Jul 27 22:07:40.935165 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:40.935139    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zz87d\" (UniqueName: \"kubernetes.io/projected/fff81b2c-5ba5-411d-9674-d8f00a824498-kube-api-access-zz87d\") pod \"fff81b2c-5ba5-411d-9674-d8f00a824498\" (UID: \"fff81b2c-5ba5-411d-9674-d8f00a824498\") "
Jul 27 22:07:40.935617 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:40.935346    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fff81b2c-5ba5-411d-9674-d8f00a824498-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fff81b2c-5ba5-411d-9674-d8f00a824498" (UID: "fff81b2c-5ba5-411d-9674-d8f00a824498"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:07:40.935728 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:40.935606    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fff81b2c-5ba5-411d-9674-d8f00a824498-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fff81b2c-5ba5-411d-9674-d8f00a824498" (UID: "fff81b2c-5ba5-411d-9674-d8f00a824498"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:07:40.935728 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:40.935591    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fff81b2c-5ba5-411d-9674-d8f00a824498-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fff81b2c-5ba5-411d-9674-d8f00a824498" (UID: "fff81b2c-5ba5-411d-9674-d8f00a824498"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:07:40.937436 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:40.937406    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fff81b2c-5ba5-411d-9674-d8f00a824498-kube-api-access-zz87d" (OuterVolumeSpecName: "kube-api-access-zz87d") pod "fff81b2c-5ba5-411d-9674-d8f00a824498" (UID: "fff81b2c-5ba5-411d-9674-d8f00a824498"). InnerVolumeSpecName "kube-api-access-zz87d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:07:40.937553 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:40.937494    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fff81b2c-5ba5-411d-9674-d8f00a824498-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fff81b2c-5ba5-411d-9674-d8f00a824498" (UID: "fff81b2c-5ba5-411d-9674-d8f00a824498"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 22:07:40.981462 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:40.981395    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl"]
Jul 27 22:07:40.981850 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:40.981835    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fff81b2c-5ba5-411d-9674-d8f00a824498" containerName="kube-rbac-proxy"
Jul 27 22:07:40.981904 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:40.981854    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fff81b2c-5ba5-411d-9674-d8f00a824498" containerName="kube-rbac-proxy"
Jul 27 22:07:40.981904 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:40.981871    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fff81b2c-5ba5-411d-9674-d8f00a824498" containerName="kserve-container"
Jul 27 22:07:40.981904 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:40.981880    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fff81b2c-5ba5-411d-9674-d8f00a824498" containerName="kserve-container"
Jul 27 22:07:40.981999 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:40.981904    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fff81b2c-5ba5-411d-9674-d8f00a824498" containerName="storage-initializer"
Jul 27 22:07:40.981999 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:40.981913    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fff81b2c-5ba5-411d-9674-d8f00a824498" containerName="storage-initializer"
Jul 27 22:07:40.981999 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:40.981972    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fff81b2c-5ba5-411d-9674-d8f00a824498" containerName="kserve-container"
Jul 27 22:07:40.981999 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:40.981983    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fff81b2c-5ba5-411d-9674-d8f00a824498" containerName="kube-rbac-proxy"
Jul 27 22:07:40.985620 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:40.985598    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl"
Jul 27 22:07:40.988528 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:40.988509    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
Jul 27 22:07:40.989044 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:40.989030    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
Jul 27 22:07:40.995595 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:40.995574    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl"]
Jul 27 22:07:41.036620 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:41.036592    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g88s\" (UniqueName: \"kubernetes.io/projected/9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0-kube-api-access-9g88s\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl\" (UID: \"9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl"
Jul 27 22:07:41.036817 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:41.036630    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl\" (UID: \"9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl"
Jul 27 22:07:41.036817 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:41.036655    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl\" (UID: \"9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl"
Jul 27 22:07:41.036817 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:41.036758    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl\" (UID: \"9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl"
Jul 27 22:07:41.037002 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:41.036835    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl\" (UID: \"9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl"
Jul 27 22:07:41.037002 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:41.036896    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz87d\" (UniqueName: \"kubernetes.io/projected/fff81b2c-5ba5-411d-9674-d8f00a824498-kube-api-access-zz87d\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:07:41.037002 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:41.036911    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fff81b2c-5ba5-411d-9674-d8f00a824498-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:07:41.037002 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:41.036926    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fff81b2c-5ba5-411d-9674-d8f00a824498-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:07:41.037002 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:41.036939    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fff81b2c-5ba5-411d-9674-d8f00a824498-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:07:41.037002 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:41.036954    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fff81b2c-5ba5-411d-9674-d8f00a824498-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:07:41.137464 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:41.137428    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl\" (UID: \"9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl"
Jul 27 22:07:41.137671 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:41.137482    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl\" (UID: \"9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl"
Jul 27 22:07:41.137671 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:41.137527    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl\" (UID: \"9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl"
Jul 27 22:07:41.137671 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:41.137609    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g88s\" (UniqueName: \"kubernetes.io/projected/9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0-kube-api-access-9g88s\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl\" (UID: \"9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl"
Jul 27 22:07:41.137671 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:41.137645    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl\" (UID: \"9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl"
Jul 27 22:07:41.137954 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:07:41.137799    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-serving-cert: secret "isvc-predictive-sklearn-v2-predictor-serving-cert" not found
Jul 27 22:07:41.137954 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:07:41.137881    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0-proxy-tls podName:9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0 nodeName:}" failed. No retries permitted until 2026-07-27 22:07:41.637860436 +0000 UTC m=+3785.005050295 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0-proxy-tls") pod "isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl" (UID: "9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0") : secret "isvc-predictive-sklearn-v2-predictor-serving-cert" not found
Jul 27 22:07:41.138110 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:41.138090    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl\" (UID: \"9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl"
Jul 27 22:07:41.138255 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:41.138236    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl\" (UID: \"9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl"
Jul 27 22:07:41.138293 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:41.138243    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl\" (UID: \"9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl"
Jul 27 22:07:41.147175 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:41.147154    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g88s\" (UniqueName: \"kubernetes.io/projected/9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0-kube-api-access-9g88s\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl\" (UID: \"9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl"
Jul 27 22:07:41.198631 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:41.198592    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:07:41.361584 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:41.361494    2577 generic.go:356] "Generic (PLEG): container finished" podID="fff81b2c-5ba5-411d-9674-d8f00a824498" containerID="3f807377a14a0a4952b4995aa76ba8f5cc0f00cc554104320d2da42bde0bb064" exitCode=0
Jul 27 22:07:41.361584 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:41.361563    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z"
Jul 27 22:07:41.361584 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:41.361575    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z" event={"ID":"fff81b2c-5ba5-411d-9674-d8f00a824498","Type":"ContainerDied","Data":"3f807377a14a0a4952b4995aa76ba8f5cc0f00cc554104320d2da42bde0bb064"}
Jul 27 22:07:41.361909 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:41.361651    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z" event={"ID":"fff81b2c-5ba5-411d-9674-d8f00a824498","Type":"ContainerDied","Data":"58a393aaea7f763af0d301d6d14952a56e8de99c9e67e48cdc634cb167d8f1b4"}
Jul 27 22:07:41.361909 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:41.361687    2577 scope.go:117] "RemoveContainer" containerID="7f1c4e7c0afd7e8e23e52c3a6a7d532eeaf9457ec3c3162fa154d215bd558d06"
Jul 27 22:07:41.370204 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:41.370189    2577 scope.go:117] "RemoveContainer" containerID="3f807377a14a0a4952b4995aa76ba8f5cc0f00cc554104320d2da42bde0bb064"
Jul 27 22:07:41.377184 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:41.377169    2577 scope.go:117] "RemoveContainer" containerID="8dcf66ee723c536e34cb1dbf32b9b345a589b53473565f52baeda68262a3f284"
Jul 27 22:07:41.382642 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:41.382624    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z"]
Jul 27 22:07:41.384599 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:41.384582    2577 scope.go:117] "RemoveContainer" containerID="7f1c4e7c0afd7e8e23e52c3a6a7d532eeaf9457ec3c3162fa154d215bd558d06"
Jul 27 22:07:41.384955 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:07:41.384934    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f1c4e7c0afd7e8e23e52c3a6a7d532eeaf9457ec3c3162fa154d215bd558d06\": container with ID starting with 7f1c4e7c0afd7e8e23e52c3a6a7d532eeaf9457ec3c3162fa154d215bd558d06 not found: ID does not exist" containerID="7f1c4e7c0afd7e8e23e52c3a6a7d532eeaf9457ec3c3162fa154d215bd558d06"
Jul 27 22:07:41.385070 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:41.384965    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f1c4e7c0afd7e8e23e52c3a6a7d532eeaf9457ec3c3162fa154d215bd558d06"} err="failed to get container status \"7f1c4e7c0afd7e8e23e52c3a6a7d532eeaf9457ec3c3162fa154d215bd558d06\": rpc error: code = NotFound desc = could not find container \"7f1c4e7c0afd7e8e23e52c3a6a7d532eeaf9457ec3c3162fa154d215bd558d06\": container with ID starting with 7f1c4e7c0afd7e8e23e52c3a6a7d532eeaf9457ec3c3162fa154d215bd558d06 not found: ID does not exist"
Jul 27 22:07:41.385070 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:41.384990    2577 scope.go:117] "RemoveContainer" containerID="3f807377a14a0a4952b4995aa76ba8f5cc0f00cc554104320d2da42bde0bb064"
Jul 27 22:07:41.385305 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:07:41.385279    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f807377a14a0a4952b4995aa76ba8f5cc0f00cc554104320d2da42bde0bb064\": container with ID starting with 3f807377a14a0a4952b4995aa76ba8f5cc0f00cc554104320d2da42bde0bb064 not found: ID does not exist" containerID="3f807377a14a0a4952b4995aa76ba8f5cc0f00cc554104320d2da42bde0bb064"
Jul 27 22:07:41.385396 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:41.385312    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f807377a14a0a4952b4995aa76ba8f5cc0f00cc554104320d2da42bde0bb064"} err="failed to get container status \"3f807377a14a0a4952b4995aa76ba8f5cc0f00cc554104320d2da42bde0bb064\": rpc error: code = NotFound desc = could not find container \"3f807377a14a0a4952b4995aa76ba8f5cc0f00cc554104320d2da42bde0bb064\": container with ID starting with 3f807377a14a0a4952b4995aa76ba8f5cc0f00cc554104320d2da42bde0bb064 not found: ID does not exist"
Jul 27 22:07:41.385396 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:41.385333    2577 scope.go:117] "RemoveContainer" containerID="8dcf66ee723c536e34cb1dbf32b9b345a589b53473565f52baeda68262a3f284"
Jul 27 22:07:41.385613 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:07:41.385586    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dcf66ee723c536e34cb1dbf32b9b345a589b53473565f52baeda68262a3f284\": container with ID starting with 8dcf66ee723c536e34cb1dbf32b9b345a589b53473565f52baeda68262a3f284 not found: ID does not exist" containerID="8dcf66ee723c536e34cb1dbf32b9b345a589b53473565f52baeda68262a3f284"
Jul 27 22:07:41.385661 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:41.385619    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dcf66ee723c536e34cb1dbf32b9b345a589b53473565f52baeda68262a3f284"} err="failed to get container status \"8dcf66ee723c536e34cb1dbf32b9b345a589b53473565f52baeda68262a3f284\": rpc error: code = NotFound desc = could not find container \"8dcf66ee723c536e34cb1dbf32b9b345a589b53473565f52baeda68262a3f284\": container with ID starting with 8dcf66ee723c536e34cb1dbf32b9b345a589b53473565f52baeda68262a3f284 not found: ID does not exist"
Jul 27 22:07:41.386820 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:41.386802    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hvh8z"]
Jul 27 22:07:41.642270 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:41.642182    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl\" (UID: \"9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl"
Jul 27 22:07:41.644520 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:41.644493    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl\" (UID: \"9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl"
Jul 27 22:07:41.896666 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:41.896582    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl"
Jul 27 22:07:42.026947 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:42.026924    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl"]
Jul 27 22:07:42.028663 ip-10-0-142-219 kubenswrapper[2577]: W0727 22:07:42.028632    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fe1dfdc_591c_48e0_b2bf_2c2f11f61ea0.slice/crio-d27979e52ff8b2216788c5be6dd4138cb861827a2c083137742480f2688d89bb WatchSource:0}: Error finding container d27979e52ff8b2216788c5be6dd4138cb861827a2c083137742480f2688d89bb: Status 404 returned error can't find the container with id d27979e52ff8b2216788c5be6dd4138cb861827a2c083137742480f2688d89bb
Jul 27 22:07:42.369243 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:42.369205    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl" event={"ID":"9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0","Type":"ContainerStarted","Data":"aaf7d43c0b817ac77954f973de6546ffd2d1d0a8f9bedf74afff85e026fbbe34"}
Jul 27 22:07:42.369438 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:42.369278    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl" event={"ID":"9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0","Type":"ContainerStarted","Data":"d27979e52ff8b2216788c5be6dd4138cb861827a2c083137742480f2688d89bb"}
Jul 27 22:07:43.135087 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:43.135052    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fff81b2c-5ba5-411d-9674-d8f00a824498" path="/var/lib/kubelet/pods/fff81b2c-5ba5-411d-9674-d8f00a824498/volumes"
Jul 27 22:07:46.385546 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:46.385463    2577 generic.go:356] "Generic (PLEG): container finished" podID="9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0" containerID="aaf7d43c0b817ac77954f973de6546ffd2d1d0a8f9bedf74afff85e026fbbe34" exitCode=0
Jul 27 22:07:46.385927 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:46.385536    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl" event={"ID":"9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0","Type":"ContainerDied","Data":"aaf7d43c0b817ac77954f973de6546ffd2d1d0a8f9bedf74afff85e026fbbe34"}
Jul 27 22:07:47.391728 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:47.391691    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl" event={"ID":"9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0","Type":"ContainerStarted","Data":"89dffde76e8d8e93856b1dd53c8ebd4492311eac15f0c456c9a67c1885649671"}
Jul 27 22:07:47.392264 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:47.391781    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl" event={"ID":"9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0","Type":"ContainerStarted","Data":"cbaf2148954d445b058525a25ad41bb5bbcf2aee161b7417f49b4c6b6c2d8d1a"}
Jul 27 22:07:47.392264 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:47.392011    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl"
Jul 27 22:07:47.412967 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:47.412915    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl" podStartSLOduration=3.563867252 podStartE2EDuration="7.412900791s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.849033539s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:07:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:07:47.411949151 +0000 UTC m=+3790.779139027" watchObservedRunningTime="2026-07-27 22:07:47.412900791 +0000 UTC m=+3790.780090657"
Jul 27 22:07:48.395825 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:48.395794    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl"
Jul 27 22:07:54.404847 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:07:54.404817    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl"
Jul 27 22:08:24.405719 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:08:24.405679    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl" podUID="9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.55:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.55:8080: connect: connection refused"
Jul 27 22:08:34.405673 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:08:34.405633    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl" podUID="9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.55:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.55:8080: connect: connection refused"
Jul 27 22:08:44.405442 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:08:44.405400    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl" podUID="9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.55:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.55:8080: connect: connection refused"
Jul 27 22:08:54.406305 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:08:54.406262    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl" podUID="9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.55:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.55:8080: connect: connection refused"
Jul 27 22:09:04.215543 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:04.215451    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl"
Jul 27 22:09:11.893601 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:11.893565    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl"]
Jul 27 22:09:11.894173 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:11.893852    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl" podUID="9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0" containerName="kserve-container" containerID="cri-o://cbaf2148954d445b058525a25ad41bb5bbcf2aee161b7417f49b4c6b6c2d8d1a" gracePeriod=30
Jul 27 22:09:11.894173 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:11.893879    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl" podUID="9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0" containerName="kube-rbac-proxy" containerID="cri-o://89dffde76e8d8e93856b1dd53c8ebd4492311eac15f0c456c9a67c1885649671" gracePeriod=30
Jul 27 22:09:11.946583 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:11.946554    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:09:12.722138 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:12.722099    2577 generic.go:356] "Generic (PLEG): container finished" podID="9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0" containerID="89dffde76e8d8e93856b1dd53c8ebd4492311eac15f0c456c9a67c1885649671" exitCode=2
Jul 27 22:09:12.722138 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:12.722136    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl" event={"ID":"9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0","Type":"ContainerDied","Data":"89dffde76e8d8e93856b1dd53c8ebd4492311eac15f0c456c9a67c1885649671"}
Jul 27 22:09:14.130830 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:14.130793    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl" podUID="9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.55:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.55:8080: connect: connection refused"
Jul 27 22:09:14.400967 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:14.400878    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl" podUID="9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.55:8643/healthz\": dial tcp 10.133.0.55:8643: connect: connection refused"
Jul 27 22:09:16.534993 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:16.534968    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl"
Jul 27 22:09:16.706255 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:16.706218    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9g88s\" (UniqueName: \"kubernetes.io/projected/9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0-kube-api-access-9g88s\") pod \"9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0\" (UID: \"9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0\") "
Jul 27 22:09:16.706255 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:16.706257    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0-kube-rbac-proxy-sar-config\") pod \"9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0\" (UID: \"9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0\") "
Jul 27 22:09:16.706480 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:16.706289    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0-cabundle-cert\") pod \"9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0\" (UID: \"9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0\") "
Jul 27 22:09:16.706480 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:16.706329    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0-proxy-tls\") pod \"9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0\" (UID: \"9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0\") "
Jul 27 22:09:16.706480 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:16.706375    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0-kserve-provision-location\") pod \"9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0\" (UID: \"9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0\") "
Jul 27 22:09:16.706737 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:16.706709    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0" (UID: "9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:09:16.706824 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:16.706711    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0" (UID: "9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:09:16.706824 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:16.706717    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0" (UID: "9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:09:16.708534 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:16.708509    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0" (UID: "9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 22:09:16.708657 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:16.708582    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0-kube-api-access-9g88s" (OuterVolumeSpecName: "kube-api-access-9g88s") pod "9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0" (UID: "9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0"). InnerVolumeSpecName "kube-api-access-9g88s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:09:16.738898 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:16.738864    2577 generic.go:356] "Generic (PLEG): container finished" podID="9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0" containerID="cbaf2148954d445b058525a25ad41bb5bbcf2aee161b7417f49b4c6b6c2d8d1a" exitCode=0
Jul 27 22:09:16.739049 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:16.738919    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl"
Jul 27 22:09:16.739049 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:16.738921    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl" event={"ID":"9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0","Type":"ContainerDied","Data":"cbaf2148954d445b058525a25ad41bb5bbcf2aee161b7417f49b4c6b6c2d8d1a"}
Jul 27 22:09:16.739138 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:16.739074    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl" event={"ID":"9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0","Type":"ContainerDied","Data":"d27979e52ff8b2216788c5be6dd4138cb861827a2c083137742480f2688d89bb"}
Jul 27 22:09:16.739138 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:16.739106    2577 scope.go:117] "RemoveContainer" containerID="89dffde76e8d8e93856b1dd53c8ebd4492311eac15f0c456c9a67c1885649671"
Jul 27 22:09:16.747189 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:16.747167    2577 scope.go:117] "RemoveContainer" containerID="cbaf2148954d445b058525a25ad41bb5bbcf2aee161b7417f49b4c6b6c2d8d1a"
Jul 27 22:09:16.754460 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:16.754441    2577 scope.go:117] "RemoveContainer" containerID="aaf7d43c0b817ac77954f973de6546ffd2d1d0a8f9bedf74afff85e026fbbe34"
Jul 27 22:09:16.761149 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:16.761127    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl"]
Jul 27 22:09:16.761648 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:16.761633    2577 scope.go:117] "RemoveContainer" containerID="89dffde76e8d8e93856b1dd53c8ebd4492311eac15f0c456c9a67c1885649671"
Jul 27 22:09:16.762076 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:09:16.762051    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89dffde76e8d8e93856b1dd53c8ebd4492311eac15f0c456c9a67c1885649671\": container with ID starting with 89dffde76e8d8e93856b1dd53c8ebd4492311eac15f0c456c9a67c1885649671 not found: ID does not exist" containerID="89dffde76e8d8e93856b1dd53c8ebd4492311eac15f0c456c9a67c1885649671"
Jul 27 22:09:16.762177 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:16.762085    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89dffde76e8d8e93856b1dd53c8ebd4492311eac15f0c456c9a67c1885649671"} err="failed to get container status \"89dffde76e8d8e93856b1dd53c8ebd4492311eac15f0c456c9a67c1885649671\": rpc error: code = NotFound desc = could not find container \"89dffde76e8d8e93856b1dd53c8ebd4492311eac15f0c456c9a67c1885649671\": container with ID starting with 89dffde76e8d8e93856b1dd53c8ebd4492311eac15f0c456c9a67c1885649671 not found: ID does not exist"
Jul 27 22:09:16.762177 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:16.762104    2577 scope.go:117] "RemoveContainer" containerID="cbaf2148954d445b058525a25ad41bb5bbcf2aee161b7417f49b4c6b6c2d8d1a"
Jul 27 22:09:16.762412 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:09:16.762391    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbaf2148954d445b058525a25ad41bb5bbcf2aee161b7417f49b4c6b6c2d8d1a\": container with ID starting with cbaf2148954d445b058525a25ad41bb5bbcf2aee161b7417f49b4c6b6c2d8d1a not found: ID does not exist" containerID="cbaf2148954d445b058525a25ad41bb5bbcf2aee161b7417f49b4c6b6c2d8d1a"
Jul 27 22:09:16.762505 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:16.762421    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbaf2148954d445b058525a25ad41bb5bbcf2aee161b7417f49b4c6b6c2d8d1a"} err="failed to get container status \"cbaf2148954d445b058525a25ad41bb5bbcf2aee161b7417f49b4c6b6c2d8d1a\": rpc error: code = NotFound desc = could not find container \"cbaf2148954d445b058525a25ad41bb5bbcf2aee161b7417f49b4c6b6c2d8d1a\": container with ID starting with cbaf2148954d445b058525a25ad41bb5bbcf2aee161b7417f49b4c6b6c2d8d1a not found: ID does not exist"
Jul 27 22:09:16.762505 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:16.762443    2577 scope.go:117] "RemoveContainer" containerID="aaf7d43c0b817ac77954f973de6546ffd2d1d0a8f9bedf74afff85e026fbbe34"
Jul 27 22:09:16.762716 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:09:16.762693    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaf7d43c0b817ac77954f973de6546ffd2d1d0a8f9bedf74afff85e026fbbe34\": container with ID starting with aaf7d43c0b817ac77954f973de6546ffd2d1d0a8f9bedf74afff85e026fbbe34 not found: ID does not exist" containerID="aaf7d43c0b817ac77954f973de6546ffd2d1d0a8f9bedf74afff85e026fbbe34"
Jul 27 22:09:16.762757 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:16.762725    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaf7d43c0b817ac77954f973de6546ffd2d1d0a8f9bedf74afff85e026fbbe34"} err="failed to get container status \"aaf7d43c0b817ac77954f973de6546ffd2d1d0a8f9bedf74afff85e026fbbe34\": rpc error: code = NotFound desc = could not find container \"aaf7d43c0b817ac77954f973de6546ffd2d1d0a8f9bedf74afff85e026fbbe34\": container with ID starting with aaf7d43c0b817ac77954f973de6546ffd2d1d0a8f9bedf74afff85e026fbbe34 not found: ID does not exist"
Jul 27 22:09:16.764013 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:16.763996    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-czlwl"]
Jul 27 22:09:16.807150 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:16.807122    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:09:16.807150 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:16.807145    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:09:16.807292 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:16.807158    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g88s\" (UniqueName: \"kubernetes.io/projected/9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0-kube-api-access-9g88s\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:09:16.807292 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:16.807172    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:09:16.807292 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:16.807184    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:09:16.985194 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:16.985112    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:09:17.069606 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:17.069571    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9"]
Jul 27 22:09:17.069977 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:17.069959    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0" containerName="storage-initializer"
Jul 27 22:09:17.070069 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:17.069979    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0" containerName="storage-initializer"
Jul 27 22:09:17.070069 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:17.069999    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0" containerName="kserve-container"
Jul 27 22:09:17.070069 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:17.070008    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0" containerName="kserve-container"
Jul 27 22:09:17.070069 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:17.070038    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0" containerName="kube-rbac-proxy"
Jul 27 22:09:17.070069 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:17.070047    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0" containerName="kube-rbac-proxy"
Jul 27 22:09:17.070341 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:17.070118    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0" containerName="kserve-container"
Jul 27 22:09:17.070341 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:17.070131    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0" containerName="kube-rbac-proxy"
Jul 27 22:09:17.074603 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:17.074581    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9"
Jul 27 22:09:17.077150 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:17.077131    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 22:09:17.077342 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:17.077307    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v7c66\""
Jul 27 22:09:17.077426 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:17.077310    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 22:09:17.077426 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:17.077379    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
Jul 27 22:09:17.077426 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:17.077415    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 22:09:17.077607 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:17.077584    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 22:09:17.077886 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:17.077870    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
Jul 27 22:09:17.086807 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:17.086787    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9"]
Jul 27 22:09:17.135289 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:17.135261    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0" path="/var/lib/kubelet/pods/9fe1dfdc-591c-48e0-b2bf-2c2f11f61ea0/volumes"
Jul 27 22:09:17.209832 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:17.209803    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9aff77da-9522-416e-926f-a8dca9a46f5a-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9\" (UID: \"9aff77da-9522-416e-926f-a8dca9a46f5a\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9"
Jul 27 22:09:17.210020 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:17.209850    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9aff77da-9522-416e-926f-a8dca9a46f5a-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9\" (UID: \"9aff77da-9522-416e-926f-a8dca9a46f5a\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9"
Jul 27 22:09:17.210020 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:17.209884    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9aff77da-9522-416e-926f-a8dca9a46f5a-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9\" (UID: \"9aff77da-9522-416e-926f-a8dca9a46f5a\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9"
Jul 27 22:09:17.210020 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:17.209961    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9aff77da-9522-416e-926f-a8dca9a46f5a-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9\" (UID: \"9aff77da-9522-416e-926f-a8dca9a46f5a\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9"
Jul 27 22:09:17.210020 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:17.210005    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs54c\" (UniqueName: \"kubernetes.io/projected/9aff77da-9522-416e-926f-a8dca9a46f5a-kube-api-access-qs54c\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9\" (UID: \"9aff77da-9522-416e-926f-a8dca9a46f5a\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9"
Jul 27 22:09:17.310667 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:17.310569    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9aff77da-9522-416e-926f-a8dca9a46f5a-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9\" (UID: \"9aff77da-9522-416e-926f-a8dca9a46f5a\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9"
Jul 27 22:09:17.310667 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:17.310612    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9aff77da-9522-416e-926f-a8dca9a46f5a-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9\" (UID: \"9aff77da-9522-416e-926f-a8dca9a46f5a\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9"
Jul 27 22:09:17.310667 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:17.310646    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9aff77da-9522-416e-926f-a8dca9a46f5a-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9\" (UID: \"9aff77da-9522-416e-926f-a8dca9a46f5a\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9"
Jul 27 22:09:17.310994 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:17.310676    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs54c\" (UniqueName: \"kubernetes.io/projected/9aff77da-9522-416e-926f-a8dca9a46f5a-kube-api-access-qs54c\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9\" (UID: \"9aff77da-9522-416e-926f-a8dca9a46f5a\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9"
Jul 27 22:09:17.310994 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:17.310746    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9aff77da-9522-416e-926f-a8dca9a46f5a-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9\" (UID: \"9aff77da-9522-416e-926f-a8dca9a46f5a\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9"
Jul 27 22:09:17.311174 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:17.311149    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9aff77da-9522-416e-926f-a8dca9a46f5a-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9\" (UID: \"9aff77da-9522-416e-926f-a8dca9a46f5a\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9"
Jul 27 22:09:17.311477 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:17.311451    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9aff77da-9522-416e-926f-a8dca9a46f5a-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9\" (UID: \"9aff77da-9522-416e-926f-a8dca9a46f5a\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9"
Jul 27 22:09:17.311477 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:17.311467    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9aff77da-9522-416e-926f-a8dca9a46f5a-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9\" (UID: \"9aff77da-9522-416e-926f-a8dca9a46f5a\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9"
Jul 27 22:09:17.313253 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:17.313233    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9aff77da-9522-416e-926f-a8dca9a46f5a-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9\" (UID: \"9aff77da-9522-416e-926f-a8dca9a46f5a\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9"
Jul 27 22:09:17.319090 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:17.319065    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs54c\" (UniqueName: \"kubernetes.io/projected/9aff77da-9522-416e-926f-a8dca9a46f5a-kube-api-access-qs54c\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9\" (UID: \"9aff77da-9522-416e-926f-a8dca9a46f5a\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9"
Jul 27 22:09:17.321299 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:17.321273    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:09:17.385778 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:17.385725    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9"
Jul 27 22:09:17.515343 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:17.515309    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9"]
Jul 27 22:09:17.517645 ip-10-0-142-219 kubenswrapper[2577]: W0727 22:09:17.517617    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aff77da_9522_416e_926f_a8dca9a46f5a.slice/crio-7c990f6d073c40d1322194c21b8c58699ce840429d42c5d362eaaf6d0cb8c3be WatchSource:0}: Error finding container 7c990f6d073c40d1322194c21b8c58699ce840429d42c5d362eaaf6d0cb8c3be: Status 404 returned error can't find the container with id 7c990f6d073c40d1322194c21b8c58699ce840429d42c5d362eaaf6d0cb8c3be
Jul 27 22:09:17.745391 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:17.745357    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9" event={"ID":"9aff77da-9522-416e-926f-a8dca9a46f5a","Type":"ContainerStarted","Data":"543d45462afeb80a0b7c2eee9b10c32dc24938c7ad139485d7c9b99b511ff084"}
Jul 27 22:09:17.745852 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:17.745416    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9" event={"ID":"9aff77da-9522-416e-926f-a8dca9a46f5a","Type":"ContainerStarted","Data":"7c990f6d073c40d1322194c21b8c58699ce840429d42c5d362eaaf6d0cb8c3be"}
Jul 27 22:09:21.765017 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:21.764980    2577 generic.go:356] "Generic (PLEG): container finished" podID="9aff77da-9522-416e-926f-a8dca9a46f5a" containerID="543d45462afeb80a0b7c2eee9b10c32dc24938c7ad139485d7c9b99b511ff084" exitCode=0
Jul 27 22:09:21.765017 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:21.765019    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9" event={"ID":"9aff77da-9522-416e-926f-a8dca9a46f5a","Type":"ContainerDied","Data":"543d45462afeb80a0b7c2eee9b10c32dc24938c7ad139485d7c9b99b511ff084"}
Jul 27 22:09:22.771826 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:22.771792    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9" event={"ID":"9aff77da-9522-416e-926f-a8dca9a46f5a","Type":"ContainerStarted","Data":"316e1017ef0da28b52fac5cba11884c3a7640f183a3239dbe4506ee83c9dd85d"}
Jul 27 22:09:22.772200 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:22.771855    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9" event={"ID":"9aff77da-9522-416e-926f-a8dca9a46f5a","Type":"ContainerStarted","Data":"ecbd05ccb7b4f518f799a2f49025af26a60d2b2dc027b17a5d52cff6bd7ee365"}
Jul 27 22:09:22.772200 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:22.772022    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9"
Jul 27 22:09:22.790444 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:22.790401    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9" podStartSLOduration=2.163531979 podStartE2EDuration="5.790387846s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.626855867s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:09:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:09:22.789636149 +0000 UTC m=+3886.156826017" watchObservedRunningTime="2026-07-27 22:09:22.790387846 +0000 UTC m=+3886.157577712"
Jul 27 22:09:23.777145 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:23.777114    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9"
Jul 27 22:09:29.785588 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:29.785559    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9"
Jul 27 22:09:59.786622 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:09:59.786587    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9" podUID="9aff77da-9522-416e-926f-a8dca9a46f5a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.56:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.56:8080: connect: connection refused"
Jul 27 22:10:09.787042 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:09.787001    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9" podUID="9aff77da-9522-416e-926f-a8dca9a46f5a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.56:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.56:8080: connect: connection refused"
Jul 27 22:10:12.623344 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:12.623308    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:10:19.786788 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:19.786725    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9" podUID="9aff77da-9522-416e-926f-a8dca9a46f5a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.56:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.56:8080: connect: connection refused"
Jul 27 22:10:29.786281 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:29.786197    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9" podUID="9aff77da-9522-416e-926f-a8dca9a46f5a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.56:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.56:8080: connect: connection refused"
Jul 27 22:10:39.813927 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:39.813880    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9"
Jul 27 22:10:47.172095 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:47.172065    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9"]
Jul 27 22:10:47.174622 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:47.172346    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9" podUID="9aff77da-9522-416e-926f-a8dca9a46f5a" containerName="kserve-container" containerID="cri-o://ecbd05ccb7b4f518f799a2f49025af26a60d2b2dc027b17a5d52cff6bd7ee365" gracePeriod=30
Jul 27 22:10:47.174622 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:47.172348    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9" podUID="9aff77da-9522-416e-926f-a8dca9a46f5a" containerName="kube-rbac-proxy" containerID="cri-o://316e1017ef0da28b52fac5cba11884c3a7640f183a3239dbe4506ee83c9dd85d" gracePeriod=30
Jul 27 22:10:47.185587 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:47.185564    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:10:48.113330 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:48.113294    2577 generic.go:356] "Generic (PLEG): container finished" podID="9aff77da-9522-416e-926f-a8dca9a46f5a" containerID="316e1017ef0da28b52fac5cba11884c3a7640f183a3239dbe4506ee83c9dd85d" exitCode=2
Jul 27 22:10:48.113501 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:48.113367    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9" event={"ID":"9aff77da-9522-416e-926f-a8dca9a46f5a","Type":"ContainerDied","Data":"316e1017ef0da28b52fac5cba11884c3a7640f183a3239dbe4506ee83c9dd85d"}
Jul 27 22:10:49.781819 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:49.781758    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9" podUID="9aff77da-9522-416e-926f-a8dca9a46f5a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.56:8643/healthz\": dial tcp 10.133.0.56:8643: connect: connection refused"
Jul 27 22:10:49.786174 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:49.786147    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9" podUID="9aff77da-9522-416e-926f-a8dca9a46f5a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.56:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.56:8080: connect: connection refused"
Jul 27 22:10:51.811657 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:51.811632    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9"
Jul 27 22:10:51.880622 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:51.880535    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9aff77da-9522-416e-926f-a8dca9a46f5a-cabundle-cert\") pod \"9aff77da-9522-416e-926f-a8dca9a46f5a\" (UID: \"9aff77da-9522-416e-926f-a8dca9a46f5a\") "
Jul 27 22:10:51.880622 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:51.880584    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qs54c\" (UniqueName: \"kubernetes.io/projected/9aff77da-9522-416e-926f-a8dca9a46f5a-kube-api-access-qs54c\") pod \"9aff77da-9522-416e-926f-a8dca9a46f5a\" (UID: \"9aff77da-9522-416e-926f-a8dca9a46f5a\") "
Jul 27 22:10:51.880881 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:51.880627    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9aff77da-9522-416e-926f-a8dca9a46f5a-kserve-provision-location\") pod \"9aff77da-9522-416e-926f-a8dca9a46f5a\" (UID: \"9aff77da-9522-416e-926f-a8dca9a46f5a\") "
Jul 27 22:10:51.880881 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:51.880663    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9aff77da-9522-416e-926f-a8dca9a46f5a-kube-rbac-proxy-sar-config\") pod \"9aff77da-9522-416e-926f-a8dca9a46f5a\" (UID: \"9aff77da-9522-416e-926f-a8dca9a46f5a\") "
Jul 27 22:10:51.880881 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:51.880708    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9aff77da-9522-416e-926f-a8dca9a46f5a-proxy-tls\") pod \"9aff77da-9522-416e-926f-a8dca9a46f5a\" (UID: \"9aff77da-9522-416e-926f-a8dca9a46f5a\") "
Jul 27 22:10:51.881065 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:51.881042    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aff77da-9522-416e-926f-a8dca9a46f5a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9aff77da-9522-416e-926f-a8dca9a46f5a" (UID: "9aff77da-9522-416e-926f-a8dca9a46f5a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:10:51.881115 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:51.880985    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9aff77da-9522-416e-926f-a8dca9a46f5a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9aff77da-9522-416e-926f-a8dca9a46f5a" (UID: "9aff77da-9522-416e-926f-a8dca9a46f5a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:10:51.881153 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:51.881112    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9aff77da-9522-416e-926f-a8dca9a46f5a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9aff77da-9522-416e-926f-a8dca9a46f5a" (UID: "9aff77da-9522-416e-926f-a8dca9a46f5a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:10:51.882838 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:51.882811    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aff77da-9522-416e-926f-a8dca9a46f5a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9aff77da-9522-416e-926f-a8dca9a46f5a" (UID: "9aff77da-9522-416e-926f-a8dca9a46f5a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 22:10:51.882838 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:51.882830    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aff77da-9522-416e-926f-a8dca9a46f5a-kube-api-access-qs54c" (OuterVolumeSpecName: "kube-api-access-qs54c") pod "9aff77da-9522-416e-926f-a8dca9a46f5a" (UID: "9aff77da-9522-416e-926f-a8dca9a46f5a"). InnerVolumeSpecName "kube-api-access-qs54c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:10:51.981808 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:51.981750    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9aff77da-9522-416e-926f-a8dca9a46f5a-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:10:51.981808 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:51.981803    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9aff77da-9522-416e-926f-a8dca9a46f5a-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:10:51.981808 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:51.981814    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs54c\" (UniqueName: \"kubernetes.io/projected/9aff77da-9522-416e-926f-a8dca9a46f5a-kube-api-access-qs54c\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:10:51.982047 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:51.981824    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9aff77da-9522-416e-926f-a8dca9a46f5a-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:10:51.982047 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:51.981834    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9aff77da-9522-416e-926f-a8dca9a46f5a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:10:52.130608 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:52.130566    2577 generic.go:356] "Generic (PLEG): container finished" podID="9aff77da-9522-416e-926f-a8dca9a46f5a" containerID="ecbd05ccb7b4f518f799a2f49025af26a60d2b2dc027b17a5d52cff6bd7ee365" exitCode=0
Jul 27 22:10:52.130608 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:52.130603    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9" event={"ID":"9aff77da-9522-416e-926f-a8dca9a46f5a","Type":"ContainerDied","Data":"ecbd05ccb7b4f518f799a2f49025af26a60d2b2dc027b17a5d52cff6bd7ee365"}
Jul 27 22:10:52.130880 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:52.130642    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9"
Jul 27 22:10:52.130880 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:52.130664    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9" event={"ID":"9aff77da-9522-416e-926f-a8dca9a46f5a","Type":"ContainerDied","Data":"7c990f6d073c40d1322194c21b8c58699ce840429d42c5d362eaaf6d0cb8c3be"}
Jul 27 22:10:52.130880 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:52.130697    2577 scope.go:117] "RemoveContainer" containerID="316e1017ef0da28b52fac5cba11884c3a7640f183a3239dbe4506ee83c9dd85d"
Jul 27 22:10:52.139183 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:52.139118    2577 scope.go:117] "RemoveContainer" containerID="ecbd05ccb7b4f518f799a2f49025af26a60d2b2dc027b17a5d52cff6bd7ee365"
Jul 27 22:10:52.147175 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:52.147143    2577 scope.go:117] "RemoveContainer" containerID="543d45462afeb80a0b7c2eee9b10c32dc24938c7ad139485d7c9b99b511ff084"
Jul 27 22:10:52.154054 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:52.154033    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9"]
Jul 27 22:10:52.154724 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:52.154706    2577 scope.go:117] "RemoveContainer" containerID="316e1017ef0da28b52fac5cba11884c3a7640f183a3239dbe4506ee83c9dd85d"
Jul 27 22:10:52.155011 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:10:52.154993    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"316e1017ef0da28b52fac5cba11884c3a7640f183a3239dbe4506ee83c9dd85d\": container with ID starting with 316e1017ef0da28b52fac5cba11884c3a7640f183a3239dbe4506ee83c9dd85d not found: ID does not exist" containerID="316e1017ef0da28b52fac5cba11884c3a7640f183a3239dbe4506ee83c9dd85d"
Jul 27 22:10:52.155082 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:52.155020    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"316e1017ef0da28b52fac5cba11884c3a7640f183a3239dbe4506ee83c9dd85d"} err="failed to get container status \"316e1017ef0da28b52fac5cba11884c3a7640f183a3239dbe4506ee83c9dd85d\": rpc error: code = NotFound desc = could not find container \"316e1017ef0da28b52fac5cba11884c3a7640f183a3239dbe4506ee83c9dd85d\": container with ID starting with 316e1017ef0da28b52fac5cba11884c3a7640f183a3239dbe4506ee83c9dd85d not found: ID does not exist"
Jul 27 22:10:52.155082 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:52.155037    2577 scope.go:117] "RemoveContainer" containerID="ecbd05ccb7b4f518f799a2f49025af26a60d2b2dc027b17a5d52cff6bd7ee365"
Jul 27 22:10:52.155268 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:10:52.155246    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecbd05ccb7b4f518f799a2f49025af26a60d2b2dc027b17a5d52cff6bd7ee365\": container with ID starting with ecbd05ccb7b4f518f799a2f49025af26a60d2b2dc027b17a5d52cff6bd7ee365 not found: ID does not exist" containerID="ecbd05ccb7b4f518f799a2f49025af26a60d2b2dc027b17a5d52cff6bd7ee365"
Jul 27 22:10:52.155308 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:52.155278    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecbd05ccb7b4f518f799a2f49025af26a60d2b2dc027b17a5d52cff6bd7ee365"} err="failed to get container status \"ecbd05ccb7b4f518f799a2f49025af26a60d2b2dc027b17a5d52cff6bd7ee365\": rpc error: code = NotFound desc = could not find container \"ecbd05ccb7b4f518f799a2f49025af26a60d2b2dc027b17a5d52cff6bd7ee365\": container with ID starting with ecbd05ccb7b4f518f799a2f49025af26a60d2b2dc027b17a5d52cff6bd7ee365 not found: ID does not exist"
Jul 27 22:10:52.155308 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:52.155300    2577 scope.go:117] "RemoveContainer" containerID="543d45462afeb80a0b7c2eee9b10c32dc24938c7ad139485d7c9b99b511ff084"
Jul 27 22:10:52.155502 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:10:52.155487    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"543d45462afeb80a0b7c2eee9b10c32dc24938c7ad139485d7c9b99b511ff084\": container with ID starting with 543d45462afeb80a0b7c2eee9b10c32dc24938c7ad139485d7c9b99b511ff084 not found: ID does not exist" containerID="543d45462afeb80a0b7c2eee9b10c32dc24938c7ad139485d7c9b99b511ff084"
Jul 27 22:10:52.155543 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:52.155506    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"543d45462afeb80a0b7c2eee9b10c32dc24938c7ad139485d7c9b99b511ff084"} err="failed to get container status \"543d45462afeb80a0b7c2eee9b10c32dc24938c7ad139485d7c9b99b511ff084\": rpc error: code = NotFound desc = could not find container \"543d45462afeb80a0b7c2eee9b10c32dc24938c7ad139485d7c9b99b511ff084\": container with ID starting with 543d45462afeb80a0b7c2eee9b10c32dc24938c7ad139485d7c9b99b511ff084 not found: ID does not exist"
Jul 27 22:10:52.159824 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:52.159801    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-5b9l9"]
Jul 27 22:10:52.228693 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:52.228656    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:10:52.279360 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:52.279328    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g"]
Jul 27 22:10:52.279718 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:52.279703    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9aff77da-9522-416e-926f-a8dca9a46f5a" containerName="storage-initializer"
Jul 27 22:10:52.279785 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:52.279720    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9aff77da-9522-416e-926f-a8dca9a46f5a" containerName="storage-initializer"
Jul 27 22:10:52.279785 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:52.279741    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9aff77da-9522-416e-926f-a8dca9a46f5a" containerName="kserve-container"
Jul 27 22:10:52.279785 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:52.279751    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9aff77da-9522-416e-926f-a8dca9a46f5a" containerName="kserve-container"
Jul 27 22:10:52.279785 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:52.279774    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9aff77da-9522-416e-926f-a8dca9a46f5a" containerName="kube-rbac-proxy"
Jul 27 22:10:52.279785 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:52.279784    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9aff77da-9522-416e-926f-a8dca9a46f5a" containerName="kube-rbac-proxy"
Jul 27 22:10:52.279954 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:52.279868    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9aff77da-9522-416e-926f-a8dca9a46f5a" containerName="kube-rbac-proxy"
Jul 27 22:10:52.279954 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:52.279881    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9aff77da-9522-416e-926f-a8dca9a46f5a" containerName="kserve-container"
Jul 27 22:10:52.284520 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:52.284502    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g"
Jul 27 22:10:52.287188 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:52.287162    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
Jul 27 22:10:52.287309 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:52.287245    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 22:10:52.287431 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:52.287417    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 22:10:52.287504 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:52.287463    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
Jul 27 22:10:52.287504 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:52.287421    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v7c66\""
Jul 27 22:10:52.287686 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:52.287671    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 22:10:52.287777 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:52.287734    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 22:10:52.293038 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:52.293014    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g"]
Jul 27 22:10:52.384672 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:52.384590    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c211b339-0257-44e0-b847-2a589dd65417-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g\" (UID: \"c211b339-0257-44e0-b847-2a589dd65417\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g"
Jul 27 22:10:52.384672 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:52.384653    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c211b339-0257-44e0-b847-2a589dd65417-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g\" (UID: \"c211b339-0257-44e0-b847-2a589dd65417\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g"
Jul 27 22:10:52.384891 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:52.384687    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vjcg\" (UniqueName: \"kubernetes.io/projected/c211b339-0257-44e0-b847-2a589dd65417-kube-api-access-2vjcg\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g\" (UID: \"c211b339-0257-44e0-b847-2a589dd65417\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g"
Jul 27 22:10:52.384891 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:52.384757    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c211b339-0257-44e0-b847-2a589dd65417-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g\" (UID: \"c211b339-0257-44e0-b847-2a589dd65417\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g"
Jul 27 22:10:52.384891 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:52.384816    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c211b339-0257-44e0-b847-2a589dd65417-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g\" (UID: \"c211b339-0257-44e0-b847-2a589dd65417\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g"
Jul 27 22:10:52.431459 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:52.431430    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:10:52.485886 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:52.485859    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c211b339-0257-44e0-b847-2a589dd65417-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g\" (UID: \"c211b339-0257-44e0-b847-2a589dd65417\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g"
Jul 27 22:10:52.486009 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:52.485891    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c211b339-0257-44e0-b847-2a589dd65417-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g\" (UID: \"c211b339-0257-44e0-b847-2a589dd65417\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g"
Jul 27 22:10:52.486009 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:52.485933    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c211b339-0257-44e0-b847-2a589dd65417-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g\" (UID: \"c211b339-0257-44e0-b847-2a589dd65417\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g"
Jul 27 22:10:52.486009 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:52.485950    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c211b339-0257-44e0-b847-2a589dd65417-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g\" (UID: \"c211b339-0257-44e0-b847-2a589dd65417\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g"
Jul 27 22:10:52.486009 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:52.485970    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vjcg\" (UniqueName: \"kubernetes.io/projected/c211b339-0257-44e0-b847-2a589dd65417-kube-api-access-2vjcg\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g\" (UID: \"c211b339-0257-44e0-b847-2a589dd65417\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g"
Jul 27 22:10:52.486285 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:52.486262    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c211b339-0257-44e0-b847-2a589dd65417-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g\" (UID: \"c211b339-0257-44e0-b847-2a589dd65417\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g"
Jul 27 22:10:52.486710 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:52.486689    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c211b339-0257-44e0-b847-2a589dd65417-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g\" (UID: \"c211b339-0257-44e0-b847-2a589dd65417\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g"
Jul 27 22:10:52.486801 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:52.486691    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c211b339-0257-44e0-b847-2a589dd65417-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g\" (UID: \"c211b339-0257-44e0-b847-2a589dd65417\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g"
Jul 27 22:10:52.488305 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:52.488286    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c211b339-0257-44e0-b847-2a589dd65417-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g\" (UID: \"c211b339-0257-44e0-b847-2a589dd65417\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g"
Jul 27 22:10:52.494482 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:52.494461    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vjcg\" (UniqueName: \"kubernetes.io/projected/c211b339-0257-44e0-b847-2a589dd65417-kube-api-access-2vjcg\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g\" (UID: \"c211b339-0257-44e0-b847-2a589dd65417\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g"
Jul 27 22:10:52.595731 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:52.595699    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g"
Jul 27 22:10:52.719536 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:52.719075    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g"]
Jul 27 22:10:52.722010 ip-10-0-142-219 kubenswrapper[2577]: W0727 22:10:52.721986    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc211b339_0257_44e0_b847_2a589dd65417.slice/crio-f3131af03bec39f3ed2b1294e22528b8119944dd6b8ad6e2b5b42f18b9513740 WatchSource:0}: Error finding container f3131af03bec39f3ed2b1294e22528b8119944dd6b8ad6e2b5b42f18b9513740: Status 404 returned error can't find the container with id f3131af03bec39f3ed2b1294e22528b8119944dd6b8ad6e2b5b42f18b9513740
Jul 27 22:10:53.138298 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:53.138258    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9aff77da-9522-416e-926f-a8dca9a46f5a" path="/var/lib/kubelet/pods/9aff77da-9522-416e-926f-a8dca9a46f5a/volumes"
Jul 27 22:10:53.139691 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:53.139657    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g" event={"ID":"c211b339-0257-44e0-b847-2a589dd65417","Type":"ContainerStarted","Data":"e923144b8cf75a9d95b6d53778e67cfa2aa9ac39b0494a7a937f1012d88328be"}
Jul 27 22:10:53.139862 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:53.139741    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g" event={"ID":"c211b339-0257-44e0-b847-2a589dd65417","Type":"ContainerStarted","Data":"f3131af03bec39f3ed2b1294e22528b8119944dd6b8ad6e2b5b42f18b9513740"}
Jul 27 22:10:57.158441 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:57.158405    2577 generic.go:356] "Generic (PLEG): container finished" podID="c211b339-0257-44e0-b847-2a589dd65417" containerID="e923144b8cf75a9d95b6d53778e67cfa2aa9ac39b0494a7a937f1012d88328be" exitCode=0
Jul 27 22:10:57.158441 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:57.158438    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g" event={"ID":"c211b339-0257-44e0-b847-2a589dd65417","Type":"ContainerDied","Data":"e923144b8cf75a9d95b6d53778e67cfa2aa9ac39b0494a7a937f1012d88328be"}
Jul 27 22:10:58.164771 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:58.164727    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g" event={"ID":"c211b339-0257-44e0-b847-2a589dd65417","Type":"ContainerStarted","Data":"c2b8f7f95f7ab8f2e188d73fd811f474e93687cbc937c1bb5c2888348920b150"}
Jul 27 22:10:58.165249 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:58.164812    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g" event={"ID":"c211b339-0257-44e0-b847-2a589dd65417","Type":"ContainerStarted","Data":"b9e9409d21198c859828f14ef9bb45bad10b907f7923d48ae7288c7702345c91"}
Jul 27 22:10:58.165249 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:58.165033    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g"
Jul 27 22:10:59.168488 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:10:59.168457    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g"
Jul 27 22:11:05.177525 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:11:05.177495    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g"
Jul 27 22:11:05.211798 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:11:05.211721    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g" podStartSLOduration=9.60615313 podStartE2EDuration="13.211700229s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.605547099s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:10:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:10:58.194110225 +0000 UTC m=+3981.561300092" watchObservedRunningTime="2026-07-27 22:11:05.211700229 +0000 UTC m=+3988.578890096"
Jul 27 22:11:33.289229 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:11:33.289196    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:11:35.178456 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:11:35.178412    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g" podUID="c211b339-0257-44e0-b847-2a589dd65417" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.57:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.57:8080: connect: connection refused"
Jul 27 22:11:45.178902 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:11:45.178866    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g" podUID="c211b339-0257-44e0-b847-2a589dd65417" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.57:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.57:8080: connect: connection refused"
Jul 27 22:11:55.178065 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:11:55.178024    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g" podUID="c211b339-0257-44e0-b847-2a589dd65417" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.57:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.57:8080: connect: connection refused"
Jul 27 22:12:05.178497 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:05.178410    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g" podUID="c211b339-0257-44e0-b847-2a589dd65417" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.57:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.57:8080: connect: connection refused"
Jul 27 22:12:09.213420 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:09.213390    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g"
Jul 27 22:12:12.355315 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:12.355284    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g"]
Jul 27 22:12:12.355705 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:12.355477    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g" podUID="c211b339-0257-44e0-b847-2a589dd65417" containerName="kserve-container" containerID="cri-o://b9e9409d21198c859828f14ef9bb45bad10b907f7923d48ae7288c7702345c91" gracePeriod=30
Jul 27 22:12:12.355705 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:12.355534    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g" podUID="c211b339-0257-44e0-b847-2a589dd65417" containerName="kube-rbac-proxy" containerID="cri-o://c2b8f7f95f7ab8f2e188d73fd811f474e93687cbc937c1bb5c2888348920b150" gracePeriod=30
Jul 27 22:12:12.378707 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:12.378684    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:12:13.459779 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:13.459722    2577 generic.go:356] "Generic (PLEG): container finished" podID="c211b339-0257-44e0-b847-2a589dd65417" containerID="c2b8f7f95f7ab8f2e188d73fd811f474e93687cbc937c1bb5c2888348920b150" exitCode=2
Jul 27 22:12:13.460161 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:13.459778    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g" event={"ID":"c211b339-0257-44e0-b847-2a589dd65417","Type":"ContainerDied","Data":"c2b8f7f95f7ab8f2e188d73fd811f474e93687cbc937c1bb5c2888348920b150"}
Jul 27 22:12:15.173164 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:15.173132    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g" podUID="c211b339-0257-44e0-b847-2a589dd65417" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.57:8643/healthz\": dial tcp 10.133.0.57:8643: connect: connection refused"
Jul 27 22:12:17.296531 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:17.296509    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g"
Jul 27 22:12:17.384961 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:17.384932    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c211b339-0257-44e0-b847-2a589dd65417-kserve-provision-location\") pod \"c211b339-0257-44e0-b847-2a589dd65417\" (UID: \"c211b339-0257-44e0-b847-2a589dd65417\") "
Jul 27 22:12:17.385152 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:17.384996    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c211b339-0257-44e0-b847-2a589dd65417-proxy-tls\") pod \"c211b339-0257-44e0-b847-2a589dd65417\" (UID: \"c211b339-0257-44e0-b847-2a589dd65417\") "
Jul 27 22:12:17.385152 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:17.385022    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c211b339-0257-44e0-b847-2a589dd65417-cabundle-cert\") pod \"c211b339-0257-44e0-b847-2a589dd65417\" (UID: \"c211b339-0257-44e0-b847-2a589dd65417\") "
Jul 27 22:12:17.385152 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:17.385040    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2vjcg\" (UniqueName: \"kubernetes.io/projected/c211b339-0257-44e0-b847-2a589dd65417-kube-api-access-2vjcg\") pod \"c211b339-0257-44e0-b847-2a589dd65417\" (UID: \"c211b339-0257-44e0-b847-2a589dd65417\") "
Jul 27 22:12:17.385152 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:17.385055    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c211b339-0257-44e0-b847-2a589dd65417-kube-rbac-proxy-sar-config\") pod \"c211b339-0257-44e0-b847-2a589dd65417\" (UID: \"c211b339-0257-44e0-b847-2a589dd65417\") "
Jul 27 22:12:17.385400 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:17.385372    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c211b339-0257-44e0-b847-2a589dd65417-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c211b339-0257-44e0-b847-2a589dd65417" (UID: "c211b339-0257-44e0-b847-2a589dd65417"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:12:17.385452 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:17.385386    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c211b339-0257-44e0-b847-2a589dd65417-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c211b339-0257-44e0-b847-2a589dd65417" (UID: "c211b339-0257-44e0-b847-2a589dd65417"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:12:17.385452 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:17.385420    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c211b339-0257-44e0-b847-2a589dd65417-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c211b339-0257-44e0-b847-2a589dd65417" (UID: "c211b339-0257-44e0-b847-2a589dd65417"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:12:17.387168 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:17.387138    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c211b339-0257-44e0-b847-2a589dd65417-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c211b339-0257-44e0-b847-2a589dd65417" (UID: "c211b339-0257-44e0-b847-2a589dd65417"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 22:12:17.387298 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:17.387208    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c211b339-0257-44e0-b847-2a589dd65417-kube-api-access-2vjcg" (OuterVolumeSpecName: "kube-api-access-2vjcg") pod "c211b339-0257-44e0-b847-2a589dd65417" (UID: "c211b339-0257-44e0-b847-2a589dd65417"). InnerVolumeSpecName "kube-api-access-2vjcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:12:17.469388 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:17.469347    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:12:17.476044 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:17.476019    2577 generic.go:356] "Generic (PLEG): container finished" podID="c211b339-0257-44e0-b847-2a589dd65417" containerID="b9e9409d21198c859828f14ef9bb45bad10b907f7923d48ae7288c7702345c91" exitCode=0
Jul 27 22:12:17.476187 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:17.476069    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g" event={"ID":"c211b339-0257-44e0-b847-2a589dd65417","Type":"ContainerDied","Data":"b9e9409d21198c859828f14ef9bb45bad10b907f7923d48ae7288c7702345c91"}
Jul 27 22:12:17.476187 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:17.476081    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g"
Jul 27 22:12:17.476187 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:17.476113    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g" event={"ID":"c211b339-0257-44e0-b847-2a589dd65417","Type":"ContainerDied","Data":"f3131af03bec39f3ed2b1294e22528b8119944dd6b8ad6e2b5b42f18b9513740"}
Jul 27 22:12:17.476187 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:17.476140    2577 scope.go:117] "RemoveContainer" containerID="c2b8f7f95f7ab8f2e188d73fd811f474e93687cbc937c1bb5c2888348920b150"
Jul 27 22:12:17.484649 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:17.484597    2577 scope.go:117] "RemoveContainer" containerID="b9e9409d21198c859828f14ef9bb45bad10b907f7923d48ae7288c7702345c91"
Jul 27 22:12:17.485560 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:17.485542    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c211b339-0257-44e0-b847-2a589dd65417-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:12:17.485560 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:17.485557    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c211b339-0257-44e0-b847-2a589dd65417-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:12:17.485644 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:17.485566    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vjcg\" (UniqueName: \"kubernetes.io/projected/c211b339-0257-44e0-b847-2a589dd65417-kube-api-access-2vjcg\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:12:17.485644 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:17.485576    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c211b339-0257-44e0-b847-2a589dd65417-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:12:17.485644 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:17.485584    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c211b339-0257-44e0-b847-2a589dd65417-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:12:17.492403 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:17.492381    2577 scope.go:117] "RemoveContainer" containerID="e923144b8cf75a9d95b6d53778e67cfa2aa9ac39b0494a7a937f1012d88328be"
Jul 27 22:12:17.499521 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:17.499505    2577 scope.go:117] "RemoveContainer" containerID="c2b8f7f95f7ab8f2e188d73fd811f474e93687cbc937c1bb5c2888348920b150"
Jul 27 22:12:17.499800 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:12:17.499779    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2b8f7f95f7ab8f2e188d73fd811f474e93687cbc937c1bb5c2888348920b150\": container with ID starting with c2b8f7f95f7ab8f2e188d73fd811f474e93687cbc937c1bb5c2888348920b150 not found: ID does not exist" containerID="c2b8f7f95f7ab8f2e188d73fd811f474e93687cbc937c1bb5c2888348920b150"
Jul 27 22:12:17.499854 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:17.499809    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2b8f7f95f7ab8f2e188d73fd811f474e93687cbc937c1bb5c2888348920b150"} err="failed to get container status \"c2b8f7f95f7ab8f2e188d73fd811f474e93687cbc937c1bb5c2888348920b150\": rpc error: code = NotFound desc = could not find container \"c2b8f7f95f7ab8f2e188d73fd811f474e93687cbc937c1bb5c2888348920b150\": container with ID starting with c2b8f7f95f7ab8f2e188d73fd811f474e93687cbc937c1bb5c2888348920b150 not found: ID does not exist"
Jul 27 22:12:17.499854 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:17.499828    2577 scope.go:117] "RemoveContainer" containerID="b9e9409d21198c859828f14ef9bb45bad10b907f7923d48ae7288c7702345c91"
Jul 27 22:12:17.500078 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:12:17.500065    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9e9409d21198c859828f14ef9bb45bad10b907f7923d48ae7288c7702345c91\": container with ID starting with b9e9409d21198c859828f14ef9bb45bad10b907f7923d48ae7288c7702345c91 not found: ID does not exist" containerID="b9e9409d21198c859828f14ef9bb45bad10b907f7923d48ae7288c7702345c91"
Jul 27 22:12:17.500121 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:17.500082    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9e9409d21198c859828f14ef9bb45bad10b907f7923d48ae7288c7702345c91"} err="failed to get container status \"b9e9409d21198c859828f14ef9bb45bad10b907f7923d48ae7288c7702345c91\": rpc error: code = NotFound desc = could not find container \"b9e9409d21198c859828f14ef9bb45bad10b907f7923d48ae7288c7702345c91\": container with ID starting with b9e9409d21198c859828f14ef9bb45bad10b907f7923d48ae7288c7702345c91 not found: ID does not exist"
Jul 27 22:12:17.500121 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:17.500093    2577 scope.go:117] "RemoveContainer" containerID="e923144b8cf75a9d95b6d53778e67cfa2aa9ac39b0494a7a937f1012d88328be"
Jul 27 22:12:17.500262 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:12:17.500249    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e923144b8cf75a9d95b6d53778e67cfa2aa9ac39b0494a7a937f1012d88328be\": container with ID starting with e923144b8cf75a9d95b6d53778e67cfa2aa9ac39b0494a7a937f1012d88328be not found: ID does not exist" containerID="e923144b8cf75a9d95b6d53778e67cfa2aa9ac39b0494a7a937f1012d88328be"
Jul 27 22:12:17.500301 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:17.500264    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e923144b8cf75a9d95b6d53778e67cfa2aa9ac39b0494a7a937f1012d88328be"} err="failed to get container status \"e923144b8cf75a9d95b6d53778e67cfa2aa9ac39b0494a7a937f1012d88328be\": rpc error: code = NotFound desc = could not find container \"e923144b8cf75a9d95b6d53778e67cfa2aa9ac39b0494a7a937f1012d88328be\": container with ID starting with e923144b8cf75a9d95b6d53778e67cfa2aa9ac39b0494a7a937f1012d88328be not found: ID does not exist"
Jul 27 22:12:17.507553 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:17.507531    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g"]
Jul 27 22:12:17.512320 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:17.512300    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jfz6g"]
Jul 27 22:12:17.838893 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:17.838740    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:12:19.135379 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:19.135346    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c211b339-0257-44e0-b847-2a589dd65417" path="/var/lib/kubelet/pods/c211b339-0257-44e0-b847-2a589dd65417/volumes"
Jul 27 22:12:19.542462 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:19.542431    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:12:24.646373 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:24.646335    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:12:24.679473 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:24.679434    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-b8g97"]
Jul 27 22:12:24.679909 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:24.679894    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c211b339-0257-44e0-b847-2a589dd65417" containerName="kserve-container"
Jul 27 22:12:24.679968 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:24.679914    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c211b339-0257-44e0-b847-2a589dd65417" containerName="kserve-container"
Jul 27 22:12:24.679968 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:24.679927    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c211b339-0257-44e0-b847-2a589dd65417" containerName="storage-initializer"
Jul 27 22:12:24.679968 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:24.679936    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c211b339-0257-44e0-b847-2a589dd65417" containerName="storage-initializer"
Jul 27 22:12:24.679968 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:24.679959    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c211b339-0257-44e0-b847-2a589dd65417" containerName="kube-rbac-proxy"
Jul 27 22:12:24.680113 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:24.679968    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c211b339-0257-44e0-b847-2a589dd65417" containerName="kube-rbac-proxy"
Jul 27 22:12:24.680113 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:24.680025    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c211b339-0257-44e0-b847-2a589dd65417" containerName="kube-rbac-proxy"
Jul 27 22:12:24.680113 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:24.680036    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c211b339-0257-44e0-b847-2a589dd65417" containerName="kserve-container"
Jul 27 22:12:24.685019 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:24.684997    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-b8g97"
Jul 27 22:12:24.687363 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:24.687332    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
Jul 27 22:12:24.687509 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:24.687432    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 22:12:24.687573 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:24.687542    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
Jul 27 22:12:24.687633 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:24.687576    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v7c66\""
Jul 27 22:12:24.687633 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:24.687574    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 22:12:24.687738 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:24.687635    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 22:12:24.687738 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:24.687695    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 22:12:24.692704 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:24.692683    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-b8g97"]
Jul 27 22:12:24.845114 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:24.845082    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a57a0b7-d716-4737-8e98-b6b865a27c81-kserve-provision-location\") pod \"isvc-sklearn-predictor-5f4d89cc44-b8g97\" (UID: \"1a57a0b7-d716-4737-8e98-b6b865a27c81\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-b8g97"
Jul 27 22:12:24.845114 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:24.845117    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4n52\" (UniqueName: \"kubernetes.io/projected/1a57a0b7-d716-4737-8e98-b6b865a27c81-kube-api-access-s4n52\") pod \"isvc-sklearn-predictor-5f4d89cc44-b8g97\" (UID: \"1a57a0b7-d716-4737-8e98-b6b865a27c81\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-b8g97"
Jul 27 22:12:24.845391 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:24.845148    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a57a0b7-d716-4737-8e98-b6b865a27c81-cabundle-cert\") pod \"isvc-sklearn-predictor-5f4d89cc44-b8g97\" (UID: \"1a57a0b7-d716-4737-8e98-b6b865a27c81\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-b8g97"
Jul 27 22:12:24.845391 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:24.845239    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a57a0b7-d716-4737-8e98-b6b865a27c81-proxy-tls\") pod \"isvc-sklearn-predictor-5f4d89cc44-b8g97\" (UID: \"1a57a0b7-d716-4737-8e98-b6b865a27c81\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-b8g97"
Jul 27 22:12:24.845391 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:24.845288    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a57a0b7-d716-4737-8e98-b6b865a27c81-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-5f4d89cc44-b8g97\" (UID: \"1a57a0b7-d716-4737-8e98-b6b865a27c81\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-b8g97"
Jul 27 22:12:24.897385 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:24.897308    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:12:24.947842 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:24.947801    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a57a0b7-d716-4737-8e98-b6b865a27c81-kserve-provision-location\") pod \"isvc-sklearn-predictor-5f4d89cc44-b8g97\" (UID: \"1a57a0b7-d716-4737-8e98-b6b865a27c81\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-b8g97"
Jul 27 22:12:24.948028 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:24.947886    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4n52\" (UniqueName: \"kubernetes.io/projected/1a57a0b7-d716-4737-8e98-b6b865a27c81-kube-api-access-s4n52\") pod \"isvc-sklearn-predictor-5f4d89cc44-b8g97\" (UID: \"1a57a0b7-d716-4737-8e98-b6b865a27c81\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-b8g97"
Jul 27 22:12:24.948028 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:24.947944    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a57a0b7-d716-4737-8e98-b6b865a27c81-cabundle-cert\") pod \"isvc-sklearn-predictor-5f4d89cc44-b8g97\" (UID: \"1a57a0b7-d716-4737-8e98-b6b865a27c81\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-b8g97"
Jul 27 22:12:24.948028 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:24.948003    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a57a0b7-d716-4737-8e98-b6b865a27c81-proxy-tls\") pod \"isvc-sklearn-predictor-5f4d89cc44-b8g97\" (UID: \"1a57a0b7-d716-4737-8e98-b6b865a27c81\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-b8g97"
Jul 27 22:12:24.948188 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:24.948066    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a57a0b7-d716-4737-8e98-b6b865a27c81-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-5f4d89cc44-b8g97\" (UID: \"1a57a0b7-d716-4737-8e98-b6b865a27c81\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-b8g97"
Jul 27 22:12:24.948553 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:24.948527    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a57a0b7-d716-4737-8e98-b6b865a27c81-kserve-provision-location\") pod \"isvc-sklearn-predictor-5f4d89cc44-b8g97\" (UID: \"1a57a0b7-d716-4737-8e98-b6b865a27c81\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-b8g97"
Jul 27 22:12:24.948958 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:24.948888    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a57a0b7-d716-4737-8e98-b6b865a27c81-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-5f4d89cc44-b8g97\" (UID: \"1a57a0b7-d716-4737-8e98-b6b865a27c81\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-b8g97"
Jul 27 22:12:24.949550 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:24.949528    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a57a0b7-d716-4737-8e98-b6b865a27c81-cabundle-cert\") pod \"isvc-sklearn-predictor-5f4d89cc44-b8g97\" (UID: \"1a57a0b7-d716-4737-8e98-b6b865a27c81\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-b8g97"
Jul 27 22:12:24.952233 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:24.952204    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a57a0b7-d716-4737-8e98-b6b865a27c81-proxy-tls\") pod \"isvc-sklearn-predictor-5f4d89cc44-b8g97\" (UID: \"1a57a0b7-d716-4737-8e98-b6b865a27c81\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-b8g97"
Jul 27 22:12:24.956814 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:24.956792    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4n52\" (UniqueName: \"kubernetes.io/projected/1a57a0b7-d716-4737-8e98-b6b865a27c81-kube-api-access-s4n52\") pod \"isvc-sklearn-predictor-5f4d89cc44-b8g97\" (UID: \"1a57a0b7-d716-4737-8e98-b6b865a27c81\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-b8g97"
Jul 27 22:12:24.996076 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:24.996053    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-b8g97"
Jul 27 22:12:25.126955 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:25.126930    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-b8g97"]
Jul 27 22:12:25.128603 ip-10-0-142-219 kubenswrapper[2577]: W0727 22:12:25.128575    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a57a0b7_d716_4737_8e98_b6b865a27c81.slice/crio-5ec3646b8611fed8f8c5fb82700e169116af6647e1a73efcf180b960efa9a73c WatchSource:0}: Error finding container 5ec3646b8611fed8f8c5fb82700e169116af6647e1a73efcf180b960efa9a73c: Status 404 returned error can't find the container with id 5ec3646b8611fed8f8c5fb82700e169116af6647e1a73efcf180b960efa9a73c
Jul 27 22:12:25.510085 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:25.510050    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-b8g97" event={"ID":"1a57a0b7-d716-4737-8e98-b6b865a27c81","Type":"ContainerStarted","Data":"ec77f536b40dfa07704bc31219739157958941501a575db08f857323256870dc"}
Jul 27 22:12:25.510233 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:25.510105    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-b8g97" event={"ID":"1a57a0b7-d716-4737-8e98-b6b865a27c81","Type":"ContainerStarted","Data":"5ec3646b8611fed8f8c5fb82700e169116af6647e1a73efcf180b960efa9a73c"}
Jul 27 22:12:29.527199 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:29.527166    2577 generic.go:356] "Generic (PLEG): container finished" podID="1a57a0b7-d716-4737-8e98-b6b865a27c81" containerID="ec77f536b40dfa07704bc31219739157958941501a575db08f857323256870dc" exitCode=0
Jul 27 22:12:29.527590 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:29.527241    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-b8g97" event={"ID":"1a57a0b7-d716-4737-8e98-b6b865a27c81","Type":"ContainerDied","Data":"ec77f536b40dfa07704bc31219739157958941501a575db08f857323256870dc"}
Jul 27 22:12:30.533211 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:30.533170    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-b8g97" event={"ID":"1a57a0b7-d716-4737-8e98-b6b865a27c81","Type":"ContainerStarted","Data":"cf6c7b0ef8abfe8b2ca1f6a8317616ff1d74e98629a3134d7eb84477daa76e11"}
Jul 27 22:12:30.533587 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:30.533234    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-b8g97" event={"ID":"1a57a0b7-d716-4737-8e98-b6b865a27c81","Type":"ContainerStarted","Data":"ead5d19fb401c64f96225ef2f44c916e8167a3e35be7f9a95d7aab3ce1a50cbb"}
Jul 27 22:12:30.533587 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:30.533442    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-b8g97"
Jul 27 22:12:30.552737 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:30.552681    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-b8g97" podStartSLOduration=2.719851664 podStartE2EDuration="6.552664822s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.832813158s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:12:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:12:30.551072701 +0000 UTC m=+4073.918262566" watchObservedRunningTime="2026-07-27 22:12:30.552664822 +0000 UTC m=+4073.919854730"
Jul 27 22:12:31.541507 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:31.540545    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-b8g97"
Jul 27 22:12:31.541918 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:31.541551    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-b8g97" podUID="1a57a0b7-d716-4737-8e98-b6b865a27c81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Jul 27 22:12:32.541984 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:32.541939    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-b8g97" podUID="1a57a0b7-d716-4737-8e98-b6b865a27c81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Jul 27 22:12:37.546195 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:37.546168    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-b8g97"
Jul 27 22:12:37.546757 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:37.546729    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-b8g97" podUID="1a57a0b7-d716-4737-8e98-b6b865a27c81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Jul 27 22:12:47.547039 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:47.546998    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-b8g97" podUID="1a57a0b7-d716-4737-8e98-b6b865a27c81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Jul 27 22:12:57.546923 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:12:57.546881    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-b8g97" podUID="1a57a0b7-d716-4737-8e98-b6b865a27c81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Jul 27 22:13:07.546681 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:13:07.546638    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-b8g97" podUID="1a57a0b7-d716-4737-8e98-b6b865a27c81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Jul 27 22:13:17.546631 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:13:17.546590    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-b8g97" podUID="1a57a0b7-d716-4737-8e98-b6b865a27c81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Jul 27 22:13:27.546995 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:13:27.546954    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-b8g97" podUID="1a57a0b7-d716-4737-8e98-b6b865a27c81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Jul 27 22:13:37.546676 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:13:37.546593    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-b8g97" podUID="1a57a0b7-d716-4737-8e98-b6b865a27c81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Jul 27 22:13:47.547480 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:13:47.547450    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-b8g97"
Jul 27 22:13:54.827260 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:13:54.827227    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-b8g97"]
Jul 27 22:13:54.827754 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:13:54.827470    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-b8g97" podUID="1a57a0b7-d716-4737-8e98-b6b865a27c81" containerName="kserve-container" containerID="cri-o://ead5d19fb401c64f96225ef2f44c916e8167a3e35be7f9a95d7aab3ce1a50cbb" gracePeriod=30
Jul 27 22:13:54.827754 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:13:54.827528    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-b8g97" podUID="1a57a0b7-d716-4737-8e98-b6b865a27c81" containerName="kube-rbac-proxy" containerID="cri-o://cf6c7b0ef8abfe8b2ca1f6a8317616ff1d74e98629a3134d7eb84477daa76e11" gracePeriod=30
Jul 27 22:13:54.849102 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:13:54.849075    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:13:55.860928 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:13:55.860892    2577 generic.go:356] "Generic (PLEG): container finished" podID="1a57a0b7-d716-4737-8e98-b6b865a27c81" containerID="cf6c7b0ef8abfe8b2ca1f6a8317616ff1d74e98629a3134d7eb84477daa76e11" exitCode=2
Jul 27 22:13:55.861294 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:13:55.860958    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-b8g97" event={"ID":"1a57a0b7-d716-4737-8e98-b6b865a27c81","Type":"ContainerDied","Data":"cf6c7b0ef8abfe8b2ca1f6a8317616ff1d74e98629a3134d7eb84477daa76e11"}
Jul 27 22:13:57.542467 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:13:57.542423    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-b8g97" podUID="1a57a0b7-d716-4737-8e98-b6b865a27c81" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.58:8643/healthz\": dial tcp 10.133.0.58:8643: connect: connection refused"
Jul 27 22:13:57.546740 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:13:57.546712    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-b8g97" podUID="1a57a0b7-d716-4737-8e98-b6b865a27c81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Jul 27 22:13:59.367919 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:13:59.367897    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-b8g97"
Jul 27 22:13:59.396749 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:13:59.396720    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a57a0b7-d716-4737-8e98-b6b865a27c81-kube-rbac-proxy-sar-config\") pod \"1a57a0b7-d716-4737-8e98-b6b865a27c81\" (UID: \"1a57a0b7-d716-4737-8e98-b6b865a27c81\") "
Jul 27 22:13:59.396890 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:13:59.396817    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a57a0b7-d716-4737-8e98-b6b865a27c81-proxy-tls\") pod \"1a57a0b7-d716-4737-8e98-b6b865a27c81\" (UID: \"1a57a0b7-d716-4737-8e98-b6b865a27c81\") "
Jul 27 22:13:59.396890 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:13:59.396843    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4n52\" (UniqueName: \"kubernetes.io/projected/1a57a0b7-d716-4737-8e98-b6b865a27c81-kube-api-access-s4n52\") pod \"1a57a0b7-d716-4737-8e98-b6b865a27c81\" (UID: \"1a57a0b7-d716-4737-8e98-b6b865a27c81\") "
Jul 27 22:13:59.396890 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:13:59.396864    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a57a0b7-d716-4737-8e98-b6b865a27c81-kserve-provision-location\") pod \"1a57a0b7-d716-4737-8e98-b6b865a27c81\" (UID: \"1a57a0b7-d716-4737-8e98-b6b865a27c81\") "
Jul 27 22:13:59.397038 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:13:59.396893    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a57a0b7-d716-4737-8e98-b6b865a27c81-cabundle-cert\") pod \"1a57a0b7-d716-4737-8e98-b6b865a27c81\" (UID: \"1a57a0b7-d716-4737-8e98-b6b865a27c81\") "
Jul 27 22:13:59.397220 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:13:59.397073    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a57a0b7-d716-4737-8e98-b6b865a27c81-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1a57a0b7-d716-4737-8e98-b6b865a27c81" (UID: "1a57a0b7-d716-4737-8e98-b6b865a27c81"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:13:59.397326 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:13:59.397262    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a57a0b7-d716-4737-8e98-b6b865a27c81-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1a57a0b7-d716-4737-8e98-b6b865a27c81" (UID: "1a57a0b7-d716-4737-8e98-b6b865a27c81"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:13:59.397387 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:13:59.397325    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a57a0b7-d716-4737-8e98-b6b865a27c81-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1a57a0b7-d716-4737-8e98-b6b865a27c81" (UID: "1a57a0b7-d716-4737-8e98-b6b865a27c81"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:13:59.398976 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:13:59.398956    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a57a0b7-d716-4737-8e98-b6b865a27c81-kube-api-access-s4n52" (OuterVolumeSpecName: "kube-api-access-s4n52") pod "1a57a0b7-d716-4737-8e98-b6b865a27c81" (UID: "1a57a0b7-d716-4737-8e98-b6b865a27c81"). InnerVolumeSpecName "kube-api-access-s4n52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:13:59.398976 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:13:59.398968    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a57a0b7-d716-4737-8e98-b6b865a27c81-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1a57a0b7-d716-4737-8e98-b6b865a27c81" (UID: "1a57a0b7-d716-4737-8e98-b6b865a27c81"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 22:13:59.497664 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:13:59.497569    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a57a0b7-d716-4737-8e98-b6b865a27c81-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:13:59.497664 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:13:59.497591    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a57a0b7-d716-4737-8e98-b6b865a27c81-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:13:59.497664 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:13:59.497600    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4n52\" (UniqueName: \"kubernetes.io/projected/1a57a0b7-d716-4737-8e98-b6b865a27c81-kube-api-access-s4n52\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:13:59.497664 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:13:59.497617    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a57a0b7-d716-4737-8e98-b6b865a27c81-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:13:59.497664 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:13:59.497626    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a57a0b7-d716-4737-8e98-b6b865a27c81-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:13:59.879328 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:13:59.879246    2577 generic.go:356] "Generic (PLEG): container finished" podID="1a57a0b7-d716-4737-8e98-b6b865a27c81" containerID="ead5d19fb401c64f96225ef2f44c916e8167a3e35be7f9a95d7aab3ce1a50cbb" exitCode=0
Jul 27 22:13:59.879328 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:13:59.879305    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-b8g97"
Jul 27 22:13:59.879500 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:13:59.879337    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-b8g97" event={"ID":"1a57a0b7-d716-4737-8e98-b6b865a27c81","Type":"ContainerDied","Data":"ead5d19fb401c64f96225ef2f44c916e8167a3e35be7f9a95d7aab3ce1a50cbb"}
Jul 27 22:13:59.879500 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:13:59.879401    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-b8g97" event={"ID":"1a57a0b7-d716-4737-8e98-b6b865a27c81","Type":"ContainerDied","Data":"5ec3646b8611fed8f8c5fb82700e169116af6647e1a73efcf180b960efa9a73c"}
Jul 27 22:13:59.879500 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:13:59.879428    2577 scope.go:117] "RemoveContainer" containerID="cf6c7b0ef8abfe8b2ca1f6a8317616ff1d74e98629a3134d7eb84477daa76e11"
Jul 27 22:13:59.889072 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:13:59.889058    2577 scope.go:117] "RemoveContainer" containerID="ead5d19fb401c64f96225ef2f44c916e8167a3e35be7f9a95d7aab3ce1a50cbb"
Jul 27 22:13:59.898071 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:13:59.898051    2577 scope.go:117] "RemoveContainer" containerID="ec77f536b40dfa07704bc31219739157958941501a575db08f857323256870dc"
Jul 27 22:13:59.905281 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:13:59.905264    2577 scope.go:117] "RemoveContainer" containerID="cf6c7b0ef8abfe8b2ca1f6a8317616ff1d74e98629a3134d7eb84477daa76e11"
Jul 27 22:13:59.905536 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:13:59.905514    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf6c7b0ef8abfe8b2ca1f6a8317616ff1d74e98629a3134d7eb84477daa76e11\": container with ID starting with cf6c7b0ef8abfe8b2ca1f6a8317616ff1d74e98629a3134d7eb84477daa76e11 not found: ID does not exist" containerID="cf6c7b0ef8abfe8b2ca1f6a8317616ff1d74e98629a3134d7eb84477daa76e11"
Jul 27 22:13:59.905644 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:13:59.905542    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf6c7b0ef8abfe8b2ca1f6a8317616ff1d74e98629a3134d7eb84477daa76e11"} err="failed to get container status \"cf6c7b0ef8abfe8b2ca1f6a8317616ff1d74e98629a3134d7eb84477daa76e11\": rpc error: code = NotFound desc = could not find container \"cf6c7b0ef8abfe8b2ca1f6a8317616ff1d74e98629a3134d7eb84477daa76e11\": container with ID starting with cf6c7b0ef8abfe8b2ca1f6a8317616ff1d74e98629a3134d7eb84477daa76e11 not found: ID does not exist"
Jul 27 22:13:59.905644 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:13:59.905577    2577 scope.go:117] "RemoveContainer" containerID="ead5d19fb401c64f96225ef2f44c916e8167a3e35be7f9a95d7aab3ce1a50cbb"
Jul 27 22:13:59.905894 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:13:59.905867    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ead5d19fb401c64f96225ef2f44c916e8167a3e35be7f9a95d7aab3ce1a50cbb\": container with ID starting with ead5d19fb401c64f96225ef2f44c916e8167a3e35be7f9a95d7aab3ce1a50cbb not found: ID does not exist" containerID="ead5d19fb401c64f96225ef2f44c916e8167a3e35be7f9a95d7aab3ce1a50cbb"
Jul 27 22:13:59.905948 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:13:59.905901    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ead5d19fb401c64f96225ef2f44c916e8167a3e35be7f9a95d7aab3ce1a50cbb"} err="failed to get container status \"ead5d19fb401c64f96225ef2f44c916e8167a3e35be7f9a95d7aab3ce1a50cbb\": rpc error: code = NotFound desc = could not find container \"ead5d19fb401c64f96225ef2f44c916e8167a3e35be7f9a95d7aab3ce1a50cbb\": container with ID starting with ead5d19fb401c64f96225ef2f44c916e8167a3e35be7f9a95d7aab3ce1a50cbb not found: ID does not exist"
Jul 27 22:13:59.905948 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:13:59.905917    2577 scope.go:117] "RemoveContainer" containerID="ec77f536b40dfa07704bc31219739157958941501a575db08f857323256870dc"
Jul 27 22:13:59.906166 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:13:59.906151    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec77f536b40dfa07704bc31219739157958941501a575db08f857323256870dc\": container with ID starting with ec77f536b40dfa07704bc31219739157958941501a575db08f857323256870dc not found: ID does not exist" containerID="ec77f536b40dfa07704bc31219739157958941501a575db08f857323256870dc"
Jul 27 22:13:59.906221 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:13:59.906169    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec77f536b40dfa07704bc31219739157958941501a575db08f857323256870dc"} err="failed to get container status \"ec77f536b40dfa07704bc31219739157958941501a575db08f857323256870dc\": rpc error: code = NotFound desc = could not find container \"ec77f536b40dfa07704bc31219739157958941501a575db08f857323256870dc\": container with ID starting with ec77f536b40dfa07704bc31219739157958941501a575db08f857323256870dc not found: ID does not exist"
Jul 27 22:13:59.939566 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:13:59.939542    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-b8g97"]
Jul 27 22:13:59.956213 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:13:59.956193    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-b8g97"]
Jul 27 22:13:59.983994 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:13:59.983962    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:14:00.053000 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:14:00.052969    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5"]
Jul 27 22:14:00.053253 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:14:00.053240    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a57a0b7-d716-4737-8e98-b6b865a27c81" containerName="storage-initializer"
Jul 27 22:14:00.053296 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:14:00.053254    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a57a0b7-d716-4737-8e98-b6b865a27c81" containerName="storage-initializer"
Jul 27 22:14:00.053296 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:14:00.053266    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a57a0b7-d716-4737-8e98-b6b865a27c81" containerName="kube-rbac-proxy"
Jul 27 22:14:00.053296 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:14:00.053272    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a57a0b7-d716-4737-8e98-b6b865a27c81" containerName="kube-rbac-proxy"
Jul 27 22:14:00.053296 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:14:00.053286    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a57a0b7-d716-4737-8e98-b6b865a27c81" containerName="kserve-container"
Jul 27 22:14:00.053296 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:14:00.053292    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a57a0b7-d716-4737-8e98-b6b865a27c81" containerName="kserve-container"
Jul 27 22:14:00.053454 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:14:00.053340    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a57a0b7-d716-4737-8e98-b6b865a27c81" containerName="kube-rbac-proxy"
Jul 27 22:14:00.053454 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:14:00.053349    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a57a0b7-d716-4737-8e98-b6b865a27c81" containerName="kserve-container"
Jul 27 22:14:00.057806 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:14:00.057787    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5"
Jul 27 22:14:00.060279 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:14:00.060258    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
Jul 27 22:14:00.060391 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:14:00.060258    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 22:14:00.060450 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:14:00.060419    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v7c66\""
Jul 27 22:14:00.060528 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:14:00.060509    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 22:14:00.060723 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:14:00.060701    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 22:14:00.060965 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:14:00.060947    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 22:14:00.061060 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:14:00.060985    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
Jul 27 22:14:00.071298 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:14:00.071276    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5"]
Jul 27 22:14:00.103364 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:14:00.103345    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-254tr\" (UniqueName: \"kubernetes.io/projected/988f9ffe-754b-4c15-b775-77170d2fcd3a-kube-api-access-254tr\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5\" (UID: \"988f9ffe-754b-4c15-b775-77170d2fcd3a\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5"
Jul 27 22:14:00.103468 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:14:00.103370    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/988f9ffe-754b-4c15-b775-77170d2fcd3a-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5\" (UID: \"988f9ffe-754b-4c15-b775-77170d2fcd3a\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5"
Jul 27 22:14:00.103468 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:14:00.103393    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/988f9ffe-754b-4c15-b775-77170d2fcd3a-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5\" (UID: \"988f9ffe-754b-4c15-b775-77170d2fcd3a\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5"
Jul 27 22:14:00.103556 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:14:00.103480    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/988f9ffe-754b-4c15-b775-77170d2fcd3a-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5\" (UID: \"988f9ffe-754b-4c15-b775-77170d2fcd3a\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5"
Jul 27 22:14:00.103556 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:14:00.103501    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/988f9ffe-754b-4c15-b775-77170d2fcd3a-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5\" (UID: \"988f9ffe-754b-4c15-b775-77170d2fcd3a\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5"
Jul 27 22:14:00.203903 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:14:00.203878    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-254tr\" (UniqueName: \"kubernetes.io/projected/988f9ffe-754b-4c15-b775-77170d2fcd3a-kube-api-access-254tr\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5\" (UID: \"988f9ffe-754b-4c15-b775-77170d2fcd3a\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5"
Jul 27 22:14:00.204026 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:14:00.203908    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/988f9ffe-754b-4c15-b775-77170d2fcd3a-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5\" (UID: \"988f9ffe-754b-4c15-b775-77170d2fcd3a\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5"
Jul 27 22:14:00.204026 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:14:00.203928    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/988f9ffe-754b-4c15-b775-77170d2fcd3a-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5\" (UID: \"988f9ffe-754b-4c15-b775-77170d2fcd3a\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5"
Jul 27 22:14:00.204026 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:14:00.203981    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/988f9ffe-754b-4c15-b775-77170d2fcd3a-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5\" (UID: \"988f9ffe-754b-4c15-b775-77170d2fcd3a\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5"
Jul 27 22:14:00.204026 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:14:00.203999    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/988f9ffe-754b-4c15-b775-77170d2fcd3a-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5\" (UID: \"988f9ffe-754b-4c15-b775-77170d2fcd3a\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5"
Jul 27 22:14:00.204328 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:14:00.204308    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/988f9ffe-754b-4c15-b775-77170d2fcd3a-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5\" (UID: \"988f9ffe-754b-4c15-b775-77170d2fcd3a\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5"
Jul 27 22:14:00.204639 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:14:00.204614    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/988f9ffe-754b-4c15-b775-77170d2fcd3a-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5\" (UID: \"988f9ffe-754b-4c15-b775-77170d2fcd3a\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5"
Jul 27 22:14:00.204733 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:14:00.204616    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/988f9ffe-754b-4c15-b775-77170d2fcd3a-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5\" (UID: \"988f9ffe-754b-4c15-b775-77170d2fcd3a\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5"
Jul 27 22:14:00.206434 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:14:00.206413    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/988f9ffe-754b-4c15-b775-77170d2fcd3a-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5\" (UID: \"988f9ffe-754b-4c15-b775-77170d2fcd3a\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5"
Jul 27 22:14:00.212799 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:14:00.212757    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-254tr\" (UniqueName: \"kubernetes.io/projected/988f9ffe-754b-4c15-b775-77170d2fcd3a-kube-api-access-254tr\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5\" (UID: \"988f9ffe-754b-4c15-b775-77170d2fcd3a\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5"
Jul 27 22:14:00.234274 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:14:00.234249    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:14:00.368331 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:14:00.368297    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5"
Jul 27 22:14:00.495571 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:14:00.495499    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5"]
Jul 27 22:14:00.498307 ip-10-0-142-219 kubenswrapper[2577]: W0727 22:14:00.498284    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod988f9ffe_754b_4c15_b775_77170d2fcd3a.slice/crio-ddde8cd6b6ebfaab6804794ba376d28424c3df6d127565f5100f93f2aa1bcb3e WatchSource:0}: Error finding container ddde8cd6b6ebfaab6804794ba376d28424c3df6d127565f5100f93f2aa1bcb3e: Status 404 returned error can't find the container with id ddde8cd6b6ebfaab6804794ba376d28424c3df6d127565f5100f93f2aa1bcb3e
Jul 27 22:14:00.885132 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:14:00.885027    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5" event={"ID":"988f9ffe-754b-4c15-b775-77170d2fcd3a","Type":"ContainerStarted","Data":"3158cd15e24033ed7dd3ebd306a813d21d2c8e91d8c1097994cac928673dbf67"}
Jul 27 22:14:00.885132 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:14:00.885100    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5" event={"ID":"988f9ffe-754b-4c15-b775-77170d2fcd3a","Type":"ContainerStarted","Data":"ddde8cd6b6ebfaab6804794ba376d28424c3df6d127565f5100f93f2aa1bcb3e"}
Jul 27 22:14:01.135463 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:14:01.135389    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a57a0b7-d716-4737-8e98-b6b865a27c81" path="/var/lib/kubelet/pods/1a57a0b7-d716-4737-8e98-b6b865a27c81/volumes"
Jul 27 22:14:04.903524 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:14:04.903485    2577 generic.go:356] "Generic (PLEG): container finished" podID="988f9ffe-754b-4c15-b775-77170d2fcd3a" containerID="3158cd15e24033ed7dd3ebd306a813d21d2c8e91d8c1097994cac928673dbf67" exitCode=0
Jul 27 22:14:04.903918 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:14:04.903554    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5" event={"ID":"988f9ffe-754b-4c15-b775-77170d2fcd3a","Type":"ContainerDied","Data":"3158cd15e24033ed7dd3ebd306a813d21d2c8e91d8c1097994cac928673dbf67"}
Jul 27 22:14:05.909732 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:14:05.909692    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5" event={"ID":"988f9ffe-754b-4c15-b775-77170d2fcd3a","Type":"ContainerStarted","Data":"78643eff79d3e4d8fe93fffdba523a4390eac71450975094a94b1ae28a471ed2"}
Jul 27 22:14:05.910124 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:14:05.909752    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5" event={"ID":"988f9ffe-754b-4c15-b775-77170d2fcd3a","Type":"ContainerStarted","Data":"1327adb06ef24b713ceb8051a8ddeacd52b04046c849d795c8e0d69a4081e5dd"}
Jul 27 22:14:05.910124 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:14:05.909962    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5"
Jul 27 22:14:05.910124 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:14:05.910014    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5"
Jul 27 22:14:05.931418 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:14:05.931379    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5" podStartSLOduration=2.361298442 podStartE2EDuration="5.931367761s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.570069319s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:14:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:14:05.929967906 +0000 UTC m=+4169.297157773" watchObservedRunningTime="2026-07-27 22:14:05.931367761 +0000 UTC m=+4169.298557627"
Jul 27 22:14:11.918707 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:14:11.918676    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5"
Jul 27 22:14:42.009949 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:14:42.009904    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5" podUID="988f9ffe-754b-4c15-b775-77170d2fcd3a" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Jul 27 22:14:51.921508 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:14:51.921478    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5"
Jul 27 22:15:00.053693 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:00.053611    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5"]
Jul 27 22:15:00.054120 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:00.053897    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5" podUID="988f9ffe-754b-4c15-b775-77170d2fcd3a" containerName="kserve-container" containerID="cri-o://1327adb06ef24b713ceb8051a8ddeacd52b04046c849d795c8e0d69a4081e5dd" gracePeriod=30
Jul 27 22:15:00.054120 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:00.053953    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5" podUID="988f9ffe-754b-4c15-b775-77170d2fcd3a" containerName="kube-rbac-proxy" containerID="cri-o://78643eff79d3e4d8fe93fffdba523a4390eac71450975094a94b1ae28a471ed2" gracePeriod=30
Jul 27 22:15:00.086035 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:00.086007    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:15:01.125086 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:01.125056    2577 generic.go:356] "Generic (PLEG): container finished" podID="988f9ffe-754b-4c15-b775-77170d2fcd3a" containerID="78643eff79d3e4d8fe93fffdba523a4390eac71450975094a94b1ae28a471ed2" exitCode=2
Jul 27 22:15:01.125477 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:01.125133    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5" event={"ID":"988f9ffe-754b-4c15-b775-77170d2fcd3a","Type":"ContainerDied","Data":"78643eff79d3e4d8fe93fffdba523a4390eac71450975094a94b1ae28a471ed2"}
Jul 27 22:15:01.914085 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:01.914035    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5" podUID="988f9ffe-754b-4c15-b775-77170d2fcd3a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.59:8643/healthz\": dial tcp 10.133.0.59:8643: connect: connection refused"
Jul 27 22:15:02.960940 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:02.960898    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5" podUID="988f9ffe-754b-4c15-b775-77170d2fcd3a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.59:8080/v2/models/sklearn-v2-mlserver/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 27 22:15:05.247490 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:05.247460    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:15:05.277178 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:05.277146    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-rx6hh"]
Jul 27 22:15:05.280548 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:05.280531    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-rx6hh"
Jul 27 22:15:05.282934 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:05.282906    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
Jul 27 22:15:05.283052 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:05.282916    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
Jul 27 22:15:05.289716 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:05.289688    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-rx6hh"]
Jul 27 22:15:05.335248 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:05.335213    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db27e5a8-9082-4e83-94c6-f1779bb0affe-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-rx6hh\" (UID: \"db27e5a8-9082-4e83-94c6-f1779bb0affe\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-rx6hh"
Jul 27 22:15:05.335428 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:05.335254    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkzrf\" (UniqueName: \"kubernetes.io/projected/db27e5a8-9082-4e83-94c6-f1779bb0affe-kube-api-access-gkzrf\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-rx6hh\" (UID: \"db27e5a8-9082-4e83-94c6-f1779bb0affe\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-rx6hh"
Jul 27 22:15:05.335428 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:05.335284    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db27e5a8-9082-4e83-94c6-f1779bb0affe-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-rx6hh\" (UID: \"db27e5a8-9082-4e83-94c6-f1779bb0affe\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-rx6hh"
Jul 27 22:15:05.335428 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:05.335328    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db27e5a8-9082-4e83-94c6-f1779bb0affe-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-rx6hh\" (UID: \"db27e5a8-9082-4e83-94c6-f1779bb0affe\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-rx6hh"
Jul 27 22:15:05.335428 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:05.335359    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db27e5a8-9082-4e83-94c6-f1779bb0affe-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-rx6hh\" (UID: \"db27e5a8-9082-4e83-94c6-f1779bb0affe\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-rx6hh"
Jul 27 22:15:05.436465 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:05.436427    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db27e5a8-9082-4e83-94c6-f1779bb0affe-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-rx6hh\" (UID: \"db27e5a8-9082-4e83-94c6-f1779bb0affe\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-rx6hh"
Jul 27 22:15:05.436647 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:05.436515    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db27e5a8-9082-4e83-94c6-f1779bb0affe-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-rx6hh\" (UID: \"db27e5a8-9082-4e83-94c6-f1779bb0affe\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-rx6hh"
Jul 27 22:15:05.436647 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:05.436551    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkzrf\" (UniqueName: \"kubernetes.io/projected/db27e5a8-9082-4e83-94c6-f1779bb0affe-kube-api-access-gkzrf\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-rx6hh\" (UID: \"db27e5a8-9082-4e83-94c6-f1779bb0affe\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-rx6hh"
Jul 27 22:15:05.436647 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:15:05.436568    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-serving-cert: secret "isvc-sklearn-runtime-predictor-serving-cert" not found
Jul 27 22:15:05.436647 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:15:05.436624    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db27e5a8-9082-4e83-94c6-f1779bb0affe-proxy-tls podName:db27e5a8-9082-4e83-94c6-f1779bb0affe nodeName:}" failed. No retries permitted until 2026-07-27 22:15:05.936608177 +0000 UTC m=+4229.303798020 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/db27e5a8-9082-4e83-94c6-f1779bb0affe-proxy-tls") pod "isvc-sklearn-runtime-predictor-599b58cf6-rx6hh" (UID: "db27e5a8-9082-4e83-94c6-f1779bb0affe") : secret "isvc-sklearn-runtime-predictor-serving-cert" not found
Jul 27 22:15:05.436934 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:05.436720    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db27e5a8-9082-4e83-94c6-f1779bb0affe-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-rx6hh\" (UID: \"db27e5a8-9082-4e83-94c6-f1779bb0affe\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-rx6hh"
Jul 27 22:15:05.436934 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:05.436786    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db27e5a8-9082-4e83-94c6-f1779bb0affe-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-rx6hh\" (UID: \"db27e5a8-9082-4e83-94c6-f1779bb0affe\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-rx6hh"
Jul 27 22:15:05.437174 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:05.437154    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db27e5a8-9082-4e83-94c6-f1779bb0affe-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-rx6hh\" (UID: \"db27e5a8-9082-4e83-94c6-f1779bb0affe\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-rx6hh"
Jul 27 22:15:05.437308 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:05.437290    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db27e5a8-9082-4e83-94c6-f1779bb0affe-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-rx6hh\" (UID: \"db27e5a8-9082-4e83-94c6-f1779bb0affe\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-rx6hh"
Jul 27 22:15:05.437369 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:05.437306    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db27e5a8-9082-4e83-94c6-f1779bb0affe-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-rx6hh\" (UID: \"db27e5a8-9082-4e83-94c6-f1779bb0affe\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-rx6hh"
Jul 27 22:15:05.445447 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:05.445427    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkzrf\" (UniqueName: \"kubernetes.io/projected/db27e5a8-9082-4e83-94c6-f1779bb0affe-kube-api-access-gkzrf\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-rx6hh\" (UID: \"db27e5a8-9082-4e83-94c6-f1779bb0affe\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-rx6hh"
Jul 27 22:15:05.578027 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:05.577938    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:15:05.941422 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:05.941381    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db27e5a8-9082-4e83-94c6-f1779bb0affe-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-rx6hh\" (UID: \"db27e5a8-9082-4e83-94c6-f1779bb0affe\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-rx6hh"
Jul 27 22:15:05.943959 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:05.943939    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db27e5a8-9082-4e83-94c6-f1779bb0affe-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-599b58cf6-rx6hh\" (UID: \"db27e5a8-9082-4e83-94c6-f1779bb0affe\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-rx6hh"
Jul 27 22:15:06.191229 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:06.191193    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-rx6hh"
Jul 27 22:15:06.325985 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:06.322817    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-rx6hh"]
Jul 27 22:15:06.330287 ip-10-0-142-219 kubenswrapper[2577]: W0727 22:15:06.330259    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb27e5a8_9082_4e83_94c6_f1779bb0affe.slice/crio-8e68668ffda9cd497617bf56856833c2d726cb3ad27a762d5048a3bca4e09aea WatchSource:0}: Error finding container 8e68668ffda9cd497617bf56856833c2d726cb3ad27a762d5048a3bca4e09aea: Status 404 returned error can't find the container with id 8e68668ffda9cd497617bf56856833c2d726cb3ad27a762d5048a3bca4e09aea
Jul 27 22:15:06.914000 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:06.913956    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5" podUID="988f9ffe-754b-4c15-b775-77170d2fcd3a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.59:8643/healthz\": dial tcp 10.133.0.59:8643: connect: connection refused"
Jul 27 22:15:07.149606 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:07.149571    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-rx6hh" event={"ID":"db27e5a8-9082-4e83-94c6-f1779bb0affe","Type":"ContainerStarted","Data":"a400f7cef2808e383cb0cf46d532ec055e7210114ae5150f2cc7861b546c5c2d"}
Jul 27 22:15:07.149819 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:07.149656    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-rx6hh" event={"ID":"db27e5a8-9082-4e83-94c6-f1779bb0affe","Type":"ContainerStarted","Data":"8e68668ffda9cd497617bf56856833c2d726cb3ad27a762d5048a3bca4e09aea"}
Jul 27 22:15:07.285849 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:07.285829    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5"
Jul 27 22:15:07.351824 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:07.351791    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/988f9ffe-754b-4c15-b775-77170d2fcd3a-cabundle-cert\") pod \"988f9ffe-754b-4c15-b775-77170d2fcd3a\" (UID: \"988f9ffe-754b-4c15-b775-77170d2fcd3a\") "
Jul 27 22:15:07.352235 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:07.351832    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/988f9ffe-754b-4c15-b775-77170d2fcd3a-kserve-provision-location\") pod \"988f9ffe-754b-4c15-b775-77170d2fcd3a\" (UID: \"988f9ffe-754b-4c15-b775-77170d2fcd3a\") "
Jul 27 22:15:07.352235 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:07.351882    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/988f9ffe-754b-4c15-b775-77170d2fcd3a-kube-rbac-proxy-sar-config\") pod \"988f9ffe-754b-4c15-b775-77170d2fcd3a\" (UID: \"988f9ffe-754b-4c15-b775-77170d2fcd3a\") "
Jul 27 22:15:07.352235 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:07.351939    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/988f9ffe-754b-4c15-b775-77170d2fcd3a-proxy-tls\") pod \"988f9ffe-754b-4c15-b775-77170d2fcd3a\" (UID: \"988f9ffe-754b-4c15-b775-77170d2fcd3a\") "
Jul 27 22:15:07.352235 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:07.351959    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-254tr\" (UniqueName: \"kubernetes.io/projected/988f9ffe-754b-4c15-b775-77170d2fcd3a-kube-api-access-254tr\") pod \"988f9ffe-754b-4c15-b775-77170d2fcd3a\" (UID: \"988f9ffe-754b-4c15-b775-77170d2fcd3a\") "
Jul 27 22:15:07.352235 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:07.352201    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/988f9ffe-754b-4c15-b775-77170d2fcd3a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "988f9ffe-754b-4c15-b775-77170d2fcd3a" (UID: "988f9ffe-754b-4c15-b775-77170d2fcd3a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:15:07.352486 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:07.352252    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/988f9ffe-754b-4c15-b775-77170d2fcd3a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "988f9ffe-754b-4c15-b775-77170d2fcd3a" (UID: "988f9ffe-754b-4c15-b775-77170d2fcd3a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:15:07.352486 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:07.352316    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/988f9ffe-754b-4c15-b775-77170d2fcd3a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "988f9ffe-754b-4c15-b775-77170d2fcd3a" (UID: "988f9ffe-754b-4c15-b775-77170d2fcd3a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:15:07.354154 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:07.354131    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/988f9ffe-754b-4c15-b775-77170d2fcd3a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "988f9ffe-754b-4c15-b775-77170d2fcd3a" (UID: "988f9ffe-754b-4c15-b775-77170d2fcd3a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 22:15:07.354253 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:07.354204    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/988f9ffe-754b-4c15-b775-77170d2fcd3a-kube-api-access-254tr" (OuterVolumeSpecName: "kube-api-access-254tr") pod "988f9ffe-754b-4c15-b775-77170d2fcd3a" (UID: "988f9ffe-754b-4c15-b775-77170d2fcd3a"). InnerVolumeSpecName "kube-api-access-254tr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:15:07.452443 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:07.452416    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/988f9ffe-754b-4c15-b775-77170d2fcd3a-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:15:07.452443 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:07.452441    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/988f9ffe-754b-4c15-b775-77170d2fcd3a-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:15:07.452627 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:07.452451    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/988f9ffe-754b-4c15-b775-77170d2fcd3a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:15:07.452627 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:07.452460    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/988f9ffe-754b-4c15-b775-77170d2fcd3a-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:15:07.452627 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:07.452468    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-254tr\" (UniqueName: \"kubernetes.io/projected/988f9ffe-754b-4c15-b775-77170d2fcd3a-kube-api-access-254tr\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:15:08.155398 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:08.155365    2577 generic.go:356] "Generic (PLEG): container finished" podID="988f9ffe-754b-4c15-b775-77170d2fcd3a" containerID="1327adb06ef24b713ceb8051a8ddeacd52b04046c849d795c8e0d69a4081e5dd" exitCode=0
Jul 27 22:15:08.155621 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:08.155418    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5"
Jul 27 22:15:08.155621 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:08.155453    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5" event={"ID":"988f9ffe-754b-4c15-b775-77170d2fcd3a","Type":"ContainerDied","Data":"1327adb06ef24b713ceb8051a8ddeacd52b04046c849d795c8e0d69a4081e5dd"}
Jul 27 22:15:08.155621 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:08.155528    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5" event={"ID":"988f9ffe-754b-4c15-b775-77170d2fcd3a","Type":"ContainerDied","Data":"ddde8cd6b6ebfaab6804794ba376d28424c3df6d127565f5100f93f2aa1bcb3e"}
Jul 27 22:15:08.155621 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:08.155556    2577 scope.go:117] "RemoveContainer" containerID="78643eff79d3e4d8fe93fffdba523a4390eac71450975094a94b1ae28a471ed2"
Jul 27 22:15:08.166413 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:08.166391    2577 scope.go:117] "RemoveContainer" containerID="1327adb06ef24b713ceb8051a8ddeacd52b04046c849d795c8e0d69a4081e5dd"
Jul 27 22:15:08.173898 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:08.173883    2577 scope.go:117] "RemoveContainer" containerID="3158cd15e24033ed7dd3ebd306a813d21d2c8e91d8c1097994cac928673dbf67"
Jul 27 22:15:08.181015 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:08.180999    2577 scope.go:117] "RemoveContainer" containerID="78643eff79d3e4d8fe93fffdba523a4390eac71450975094a94b1ae28a471ed2"
Jul 27 22:15:08.181318 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:15:08.181299    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78643eff79d3e4d8fe93fffdba523a4390eac71450975094a94b1ae28a471ed2\": container with ID starting with 78643eff79d3e4d8fe93fffdba523a4390eac71450975094a94b1ae28a471ed2 not found: ID does not exist" containerID="78643eff79d3e4d8fe93fffdba523a4390eac71450975094a94b1ae28a471ed2"
Jul 27 22:15:08.181368 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:08.181326    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78643eff79d3e4d8fe93fffdba523a4390eac71450975094a94b1ae28a471ed2"} err="failed to get container status \"78643eff79d3e4d8fe93fffdba523a4390eac71450975094a94b1ae28a471ed2\": rpc error: code = NotFound desc = could not find container \"78643eff79d3e4d8fe93fffdba523a4390eac71450975094a94b1ae28a471ed2\": container with ID starting with 78643eff79d3e4d8fe93fffdba523a4390eac71450975094a94b1ae28a471ed2 not found: ID does not exist"
Jul 27 22:15:08.181368 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:08.181344    2577 scope.go:117] "RemoveContainer" containerID="1327adb06ef24b713ceb8051a8ddeacd52b04046c849d795c8e0d69a4081e5dd"
Jul 27 22:15:08.181557 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:15:08.181544    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1327adb06ef24b713ceb8051a8ddeacd52b04046c849d795c8e0d69a4081e5dd\": container with ID starting with 1327adb06ef24b713ceb8051a8ddeacd52b04046c849d795c8e0d69a4081e5dd not found: ID does not exist" containerID="1327adb06ef24b713ceb8051a8ddeacd52b04046c849d795c8e0d69a4081e5dd"
Jul 27 22:15:08.181610 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:08.181560    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1327adb06ef24b713ceb8051a8ddeacd52b04046c849d795c8e0d69a4081e5dd"} err="failed to get container status \"1327adb06ef24b713ceb8051a8ddeacd52b04046c849d795c8e0d69a4081e5dd\": rpc error: code = NotFound desc = could not find container \"1327adb06ef24b713ceb8051a8ddeacd52b04046c849d795c8e0d69a4081e5dd\": container with ID starting with 1327adb06ef24b713ceb8051a8ddeacd52b04046c849d795c8e0d69a4081e5dd not found: ID does not exist"
Jul 27 22:15:08.181610 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:08.181572    2577 scope.go:117] "RemoveContainer" containerID="3158cd15e24033ed7dd3ebd306a813d21d2c8e91d8c1097994cac928673dbf67"
Jul 27 22:15:08.181842 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:15:08.181808    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3158cd15e24033ed7dd3ebd306a813d21d2c8e91d8c1097994cac928673dbf67\": container with ID starting with 3158cd15e24033ed7dd3ebd306a813d21d2c8e91d8c1097994cac928673dbf67 not found: ID does not exist" containerID="3158cd15e24033ed7dd3ebd306a813d21d2c8e91d8c1097994cac928673dbf67"
Jul 27 22:15:08.181895 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:08.181851    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3158cd15e24033ed7dd3ebd306a813d21d2c8e91d8c1097994cac928673dbf67"} err="failed to get container status \"3158cd15e24033ed7dd3ebd306a813d21d2c8e91d8c1097994cac928673dbf67\": rpc error: code = NotFound desc = could not find container \"3158cd15e24033ed7dd3ebd306a813d21d2c8e91d8c1097994cac928673dbf67\": container with ID starting with 3158cd15e24033ed7dd3ebd306a813d21d2c8e91d8c1097994cac928673dbf67 not found: ID does not exist"
Jul 27 22:15:08.182537 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:08.182521    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5"]
Jul 27 22:15:08.198254 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:08.198228    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcgr5"]
Jul 27 22:15:09.135506 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:09.135458    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="988f9ffe-754b-4c15-b775-77170d2fcd3a" path="/var/lib/kubelet/pods/988f9ffe-754b-4c15-b775-77170d2fcd3a/volumes"
Jul 27 22:15:12.172769 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:12.172709    2577 generic.go:356] "Generic (PLEG): container finished" podID="db27e5a8-9082-4e83-94c6-f1779bb0affe" containerID="a400f7cef2808e383cb0cf46d532ec055e7210114ae5150f2cc7861b546c5c2d" exitCode=0
Jul 27 22:15:12.173172 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:12.172791    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-rx6hh" event={"ID":"db27e5a8-9082-4e83-94c6-f1779bb0affe","Type":"ContainerDied","Data":"a400f7cef2808e383cb0cf46d532ec055e7210114ae5150f2cc7861b546c5c2d"}
Jul 27 22:15:13.179102 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:13.179071    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-rx6hh" event={"ID":"db27e5a8-9082-4e83-94c6-f1779bb0affe","Type":"ContainerStarted","Data":"afab2cfb0fc177ba8d5169c542cb12049cdbc483146b5a104a3b0ee0ac19b399"}
Jul 27 22:15:13.179466 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:13.179122    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-rx6hh" event={"ID":"db27e5a8-9082-4e83-94c6-f1779bb0affe","Type":"ContainerStarted","Data":"8f0f632a8116e74824f0cfaa39374c98899c762a51fe2672d918d9833e28694a"}
Jul 27 22:15:13.179466 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:13.179390    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-rx6hh"
Jul 27 22:15:13.179590 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:13.179577    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-rx6hh"
Jul 27 22:15:13.180774 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:13.180733    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-rx6hh" podUID="db27e5a8-9082-4e83-94c6-f1779bb0affe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Jul 27 22:15:13.198819 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:13.198783    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-rx6hh" podStartSLOduration=2.583756099 podStartE2EDuration="8.19875819s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.615002091s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:15:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:15:13.197871565 +0000 UTC m=+4236.565061430" watchObservedRunningTime="2026-07-27 22:15:13.19875819 +0000 UTC m=+4236.565948056"
Jul 27 22:15:14.183694 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:14.183651    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-rx6hh" podUID="db27e5a8-9082-4e83-94c6-f1779bb0affe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Jul 27 22:15:19.188360 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:19.188333    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-rx6hh"
Jul 27 22:15:19.188873 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:19.188846    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-rx6hh" podUID="db27e5a8-9082-4e83-94c6-f1779bb0affe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Jul 27 22:15:29.190931 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:29.190901    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-rx6hh"
Jul 27 22:15:36.949202 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:36.949171    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-599b58cf6-rx6hh_db27e5a8-9082-4e83-94c6-f1779bb0affe/kserve-container/0.log"
Jul 27 22:15:37.104066 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:37.104026    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-rx6hh"]
Jul 27 22:15:37.104321 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:37.104290    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-rx6hh" podUID="db27e5a8-9082-4e83-94c6-f1779bb0affe" containerName="kserve-container" containerID="cri-o://8f0f632a8116e74824f0cfaa39374c98899c762a51fe2672d918d9833e28694a" gracePeriod=30
Jul 27 22:15:37.104416 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:37.104304    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-rx6hh" podUID="db27e5a8-9082-4e83-94c6-f1779bb0affe" containerName="kube-rbac-proxy" containerID="cri-o://afab2cfb0fc177ba8d5169c542cb12049cdbc483146b5a104a3b0ee0ac19b399" gracePeriod=30
Jul 27 22:15:37.120796 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:37.120753    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:15:37.286263 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:37.286232    2577 generic.go:356] "Generic (PLEG): container finished" podID="db27e5a8-9082-4e83-94c6-f1779bb0affe" containerID="afab2cfb0fc177ba8d5169c542cb12049cdbc483146b5a104a3b0ee0ac19b399" exitCode=2
Jul 27 22:15:37.286395 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:37.286304    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-rx6hh" event={"ID":"db27e5a8-9082-4e83-94c6-f1779bb0affe","Type":"ContainerDied","Data":"afab2cfb0fc177ba8d5169c542cb12049cdbc483146b5a104a3b0ee0ac19b399"}
Jul 27 22:15:38.158146 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:38.158125    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-rx6hh"
Jul 27 22:15:38.291838 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:38.291809    2577 generic.go:356] "Generic (PLEG): container finished" podID="db27e5a8-9082-4e83-94c6-f1779bb0affe" containerID="8f0f632a8116e74824f0cfaa39374c98899c762a51fe2672d918d9833e28694a" exitCode=0
Jul 27 22:15:38.292013 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:38.291860    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-rx6hh"
Jul 27 22:15:38.292013 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:38.291896    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-rx6hh" event={"ID":"db27e5a8-9082-4e83-94c6-f1779bb0affe","Type":"ContainerDied","Data":"8f0f632a8116e74824f0cfaa39374c98899c762a51fe2672d918d9833e28694a"}
Jul 27 22:15:38.292013 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:38.291963    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-rx6hh" event={"ID":"db27e5a8-9082-4e83-94c6-f1779bb0affe","Type":"ContainerDied","Data":"8e68668ffda9cd497617bf56856833c2d726cb3ad27a762d5048a3bca4e09aea"}
Jul 27 22:15:38.292013 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:38.291991    2577 scope.go:117] "RemoveContainer" containerID="afab2cfb0fc177ba8d5169c542cb12049cdbc483146b5a104a3b0ee0ac19b399"
Jul 27 22:15:38.302112 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:38.302093    2577 scope.go:117] "RemoveContainer" containerID="8f0f632a8116e74824f0cfaa39374c98899c762a51fe2672d918d9833e28694a"
Jul 27 22:15:38.307289 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:38.307267    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gkzrf\" (UniqueName: \"kubernetes.io/projected/db27e5a8-9082-4e83-94c6-f1779bb0affe-kube-api-access-gkzrf\") pod \"db27e5a8-9082-4e83-94c6-f1779bb0affe\" (UID: \"db27e5a8-9082-4e83-94c6-f1779bb0affe\") "
Jul 27 22:15:38.307371 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:38.307322    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db27e5a8-9082-4e83-94c6-f1779bb0affe-cabundle-cert\") pod \"db27e5a8-9082-4e83-94c6-f1779bb0affe\" (UID: \"db27e5a8-9082-4e83-94c6-f1779bb0affe\") "
Jul 27 22:15:38.307371 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:38.307348    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db27e5a8-9082-4e83-94c6-f1779bb0affe-kube-rbac-proxy-sar-config\") pod \"db27e5a8-9082-4e83-94c6-f1779bb0affe\" (UID: \"db27e5a8-9082-4e83-94c6-f1779bb0affe\") "
Jul 27 22:15:38.307448 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:38.307393    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db27e5a8-9082-4e83-94c6-f1779bb0affe-proxy-tls\") pod \"db27e5a8-9082-4e83-94c6-f1779bb0affe\" (UID: \"db27e5a8-9082-4e83-94c6-f1779bb0affe\") "
Jul 27 22:15:38.307448 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:38.307432    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db27e5a8-9082-4e83-94c6-f1779bb0affe-kserve-provision-location\") pod \"db27e5a8-9082-4e83-94c6-f1779bb0affe\" (UID: \"db27e5a8-9082-4e83-94c6-f1779bb0affe\") "
Jul 27 22:15:38.307684 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:38.307662    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db27e5a8-9082-4e83-94c6-f1779bb0affe-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "db27e5a8-9082-4e83-94c6-f1779bb0affe" (UID: "db27e5a8-9082-4e83-94c6-f1779bb0affe"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:15:38.307814 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:38.307690    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db27e5a8-9082-4e83-94c6-f1779bb0affe-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "db27e5a8-9082-4e83-94c6-f1779bb0affe" (UID: "db27e5a8-9082-4e83-94c6-f1779bb0affe"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:15:38.309387 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:38.309364    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db27e5a8-9082-4e83-94c6-f1779bb0affe-kube-api-access-gkzrf" (OuterVolumeSpecName: "kube-api-access-gkzrf") pod "db27e5a8-9082-4e83-94c6-f1779bb0affe" (UID: "db27e5a8-9082-4e83-94c6-f1779bb0affe"). InnerVolumeSpecName "kube-api-access-gkzrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:15:38.309488 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:38.309406    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db27e5a8-9082-4e83-94c6-f1779bb0affe-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "db27e5a8-9082-4e83-94c6-f1779bb0affe" (UID: "db27e5a8-9082-4e83-94c6-f1779bb0affe"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 22:15:38.309793 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:38.309751    2577 scope.go:117] "RemoveContainer" containerID="a400f7cef2808e383cb0cf46d532ec055e7210114ae5150f2cc7861b546c5c2d"
Jul 27 22:15:38.314699 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:38.314678    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db27e5a8-9082-4e83-94c6-f1779bb0affe-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "db27e5a8-9082-4e83-94c6-f1779bb0affe" (UID: "db27e5a8-9082-4e83-94c6-f1779bb0affe"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:15:38.322453 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:38.322435    2577 scope.go:117] "RemoveContainer" containerID="afab2cfb0fc177ba8d5169c542cb12049cdbc483146b5a104a3b0ee0ac19b399"
Jul 27 22:15:38.322803 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:15:38.322753    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afab2cfb0fc177ba8d5169c542cb12049cdbc483146b5a104a3b0ee0ac19b399\": container with ID starting with afab2cfb0fc177ba8d5169c542cb12049cdbc483146b5a104a3b0ee0ac19b399 not found: ID does not exist" containerID="afab2cfb0fc177ba8d5169c542cb12049cdbc483146b5a104a3b0ee0ac19b399"
Jul 27 22:15:38.322923 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:38.322814    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afab2cfb0fc177ba8d5169c542cb12049cdbc483146b5a104a3b0ee0ac19b399"} err="failed to get container status \"afab2cfb0fc177ba8d5169c542cb12049cdbc483146b5a104a3b0ee0ac19b399\": rpc error: code = NotFound desc = could not find container \"afab2cfb0fc177ba8d5169c542cb12049cdbc483146b5a104a3b0ee0ac19b399\": container with ID starting with afab2cfb0fc177ba8d5169c542cb12049cdbc483146b5a104a3b0ee0ac19b399 not found: ID does not exist"
Jul 27 22:15:38.322923 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:38.322851    2577 scope.go:117] "RemoveContainer" containerID="8f0f632a8116e74824f0cfaa39374c98899c762a51fe2672d918d9833e28694a"
Jul 27 22:15:38.323126 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:15:38.323108    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f0f632a8116e74824f0cfaa39374c98899c762a51fe2672d918d9833e28694a\": container with ID starting with 8f0f632a8116e74824f0cfaa39374c98899c762a51fe2672d918d9833e28694a not found: ID does not exist" containerID="8f0f632a8116e74824f0cfaa39374c98899c762a51fe2672d918d9833e28694a"
Jul 27 22:15:38.323223 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:38.323130    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f0f632a8116e74824f0cfaa39374c98899c762a51fe2672d918d9833e28694a"} err="failed to get container status \"8f0f632a8116e74824f0cfaa39374c98899c762a51fe2672d918d9833e28694a\": rpc error: code = NotFound desc = could not find container \"8f0f632a8116e74824f0cfaa39374c98899c762a51fe2672d918d9833e28694a\": container with ID starting with 8f0f632a8116e74824f0cfaa39374c98899c762a51fe2672d918d9833e28694a not found: ID does not exist"
Jul 27 22:15:38.323223 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:38.323149    2577 scope.go:117] "RemoveContainer" containerID="a400f7cef2808e383cb0cf46d532ec055e7210114ae5150f2cc7861b546c5c2d"
Jul 27 22:15:38.323450 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:15:38.323423    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a400f7cef2808e383cb0cf46d532ec055e7210114ae5150f2cc7861b546c5c2d\": container with ID starting with a400f7cef2808e383cb0cf46d532ec055e7210114ae5150f2cc7861b546c5c2d not found: ID does not exist" containerID="a400f7cef2808e383cb0cf46d532ec055e7210114ae5150f2cc7861b546c5c2d"
Jul 27 22:15:38.323528 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:38.323450    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a400f7cef2808e383cb0cf46d532ec055e7210114ae5150f2cc7861b546c5c2d"} err="failed to get container status \"a400f7cef2808e383cb0cf46d532ec055e7210114ae5150f2cc7861b546c5c2d\": rpc error: code = NotFound desc = could not find container \"a400f7cef2808e383cb0cf46d532ec055e7210114ae5150f2cc7861b546c5c2d\": container with ID starting with a400f7cef2808e383cb0cf46d532ec055e7210114ae5150f2cc7861b546c5c2d not found: ID does not exist"
Jul 27 22:15:38.408434 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:38.408406    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkzrf\" (UniqueName: \"kubernetes.io/projected/db27e5a8-9082-4e83-94c6-f1779bb0affe-kube-api-access-gkzrf\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:15:38.408434 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:38.408429    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db27e5a8-9082-4e83-94c6-f1779bb0affe-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:15:38.408434 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:38.408438    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db27e5a8-9082-4e83-94c6-f1779bb0affe-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:15:38.408650 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:38.408447    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db27e5a8-9082-4e83-94c6-f1779bb0affe-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:15:38.408650 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:38.408456    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db27e5a8-9082-4e83-94c6-f1779bb0affe-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:15:38.621353 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:38.621311    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-rx6hh"]
Jul 27 22:15:38.621525 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:38.621370    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-599b58cf6-rx6hh"]
Jul 27 22:15:39.135400 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:39.135367    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db27e5a8-9082-4e83-94c6-f1779bb0affe" path="/var/lib/kubelet/pods/db27e5a8-9082-4e83-94c6-f1779bb0affe/volumes"
Jul 27 22:15:42.205297 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:42.205266    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:15:42.247362 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:42.247326    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq"]
Jul 27 22:15:42.247605 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:42.247594    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="988f9ffe-754b-4c15-b775-77170d2fcd3a" containerName="kube-rbac-proxy"
Jul 27 22:15:42.247605 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:42.247618    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="988f9ffe-754b-4c15-b775-77170d2fcd3a" containerName="kube-rbac-proxy"
Jul 27 22:15:42.247752 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:42.247631    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db27e5a8-9082-4e83-94c6-f1779bb0affe" containerName="storage-initializer"
Jul 27 22:15:42.247752 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:42.247637    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="db27e5a8-9082-4e83-94c6-f1779bb0affe" containerName="storage-initializer"
Jul 27 22:15:42.247752 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:42.247645    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db27e5a8-9082-4e83-94c6-f1779bb0affe" containerName="kserve-container"
Jul 27 22:15:42.247752 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:42.247650    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="db27e5a8-9082-4e83-94c6-f1779bb0affe" containerName="kserve-container"
Jul 27 22:15:42.247752 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:42.247678    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db27e5a8-9082-4e83-94c6-f1779bb0affe" containerName="kube-rbac-proxy"
Jul 27 22:15:42.247752 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:42.247683    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="db27e5a8-9082-4e83-94c6-f1779bb0affe" containerName="kube-rbac-proxy"
Jul 27 22:15:42.247752 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:42.247690    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="988f9ffe-754b-4c15-b775-77170d2fcd3a" containerName="storage-initializer"
Jul 27 22:15:42.247752 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:42.247696    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="988f9ffe-754b-4c15-b775-77170d2fcd3a" containerName="storage-initializer"
Jul 27 22:15:42.247752 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:42.247706    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="988f9ffe-754b-4c15-b775-77170d2fcd3a" containerName="kserve-container"
Jul 27 22:15:42.247752 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:42.247711    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="988f9ffe-754b-4c15-b775-77170d2fcd3a" containerName="kserve-container"
Jul 27 22:15:42.247752 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:42.247756    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="988f9ffe-754b-4c15-b775-77170d2fcd3a" containerName="kube-rbac-proxy"
Jul 27 22:15:42.248281 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:42.247778    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="db27e5a8-9082-4e83-94c6-f1779bb0affe" containerName="kserve-container"
Jul 27 22:15:42.248281 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:42.247784    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="988f9ffe-754b-4c15-b775-77170d2fcd3a" containerName="kserve-container"
Jul 27 22:15:42.248281 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:42.247790    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="db27e5a8-9082-4e83-94c6-f1779bb0affe" containerName="kube-rbac-proxy"
Jul 27 22:15:42.252481 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:42.252465    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq"
Jul 27 22:15:42.254677 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:42.254658    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 22:15:42.254811 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:42.254794    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 22:15:42.254884 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:42.254806    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
Jul 27 22:15:42.254884 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:42.254808    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 27 22:15:42.254884 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:42.254839    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 22:15:42.254884 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:42.254842    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v7c66\""
Jul 27 22:15:42.255857 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:42.255839    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 22:15:42.260741 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:42.260721    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq"]
Jul 27 22:15:42.305967 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:42.305922    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:15:42.338521 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:42.338499    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9lhk\" (UniqueName: \"kubernetes.io/projected/068d1618-4c36-42d2-b57a-d04cbe9be77d-kube-api-access-p9lhk\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq\" (UID: \"068d1618-4c36-42d2-b57a-d04cbe9be77d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq"
Jul 27 22:15:42.338625 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:42.338526    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/068d1618-4c36-42d2-b57a-d04cbe9be77d-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq\" (UID: \"068d1618-4c36-42d2-b57a-d04cbe9be77d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq"
Jul 27 22:15:42.338625 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:42.338595    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/068d1618-4c36-42d2-b57a-d04cbe9be77d-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq\" (UID: \"068d1618-4c36-42d2-b57a-d04cbe9be77d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq"
Jul 27 22:15:42.338711 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:42.338624    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/068d1618-4c36-42d2-b57a-d04cbe9be77d-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq\" (UID: \"068d1618-4c36-42d2-b57a-d04cbe9be77d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq"
Jul 27 22:15:42.338711 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:42.338641    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/068d1618-4c36-42d2-b57a-d04cbe9be77d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq\" (UID: \"068d1618-4c36-42d2-b57a-d04cbe9be77d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq"
Jul 27 22:15:42.439396 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:42.439368    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/068d1618-4c36-42d2-b57a-d04cbe9be77d-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq\" (UID: \"068d1618-4c36-42d2-b57a-d04cbe9be77d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq"
Jul 27 22:15:42.439396 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:42.439396    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/068d1618-4c36-42d2-b57a-d04cbe9be77d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq\" (UID: \"068d1618-4c36-42d2-b57a-d04cbe9be77d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq"
Jul 27 22:15:42.439559 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:42.439422    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9lhk\" (UniqueName: \"kubernetes.io/projected/068d1618-4c36-42d2-b57a-d04cbe9be77d-kube-api-access-p9lhk\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq\" (UID: \"068d1618-4c36-42d2-b57a-d04cbe9be77d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq"
Jul 27 22:15:42.439634 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:42.439616    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/068d1618-4c36-42d2-b57a-d04cbe9be77d-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq\" (UID: \"068d1618-4c36-42d2-b57a-d04cbe9be77d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq"
Jul 27 22:15:42.439738 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:42.439723    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/068d1618-4c36-42d2-b57a-d04cbe9be77d-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq\" (UID: \"068d1618-4c36-42d2-b57a-d04cbe9be77d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq"
Jul 27 22:15:42.439971 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:42.439950    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/068d1618-4c36-42d2-b57a-d04cbe9be77d-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq\" (UID: \"068d1618-4c36-42d2-b57a-d04cbe9be77d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq"
Jul 27 22:15:42.440131 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:42.440111    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/068d1618-4c36-42d2-b57a-d04cbe9be77d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq\" (UID: \"068d1618-4c36-42d2-b57a-d04cbe9be77d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq"
Jul 27 22:15:42.440326 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:42.440309    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/068d1618-4c36-42d2-b57a-d04cbe9be77d-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq\" (UID: \"068d1618-4c36-42d2-b57a-d04cbe9be77d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq"
Jul 27 22:15:42.442049 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:42.442023    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/068d1618-4c36-42d2-b57a-d04cbe9be77d-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq\" (UID: \"068d1618-4c36-42d2-b57a-d04cbe9be77d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq"
Jul 27 22:15:42.447256 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:42.447234    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9lhk\" (UniqueName: \"kubernetes.io/projected/068d1618-4c36-42d2-b57a-d04cbe9be77d-kube-api-access-p9lhk\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq\" (UID: \"068d1618-4c36-42d2-b57a-d04cbe9be77d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq"
Jul 27 22:15:42.563207 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:42.563119    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq"
Jul 27 22:15:42.692668 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:42.692645    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq"]
Jul 27 22:15:42.694823 ip-10-0-142-219 kubenswrapper[2577]: W0727 22:15:42.694793    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod068d1618_4c36_42d2_b57a_d04cbe9be77d.slice/crio-1e77b047e5f208d8b7613f64922b27d1254d2ca4b6391c50383510f4009ce510 WatchSource:0}: Error finding container 1e77b047e5f208d8b7613f64922b27d1254d2ca4b6391c50383510f4009ce510: Status 404 returned error can't find the container with id 1e77b047e5f208d8b7613f64922b27d1254d2ca4b6391c50383510f4009ce510
Jul 27 22:15:43.312975 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:43.312940    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq" event={"ID":"068d1618-4c36-42d2-b57a-d04cbe9be77d","Type":"ContainerStarted","Data":"f98d46df0c36f7828fb39fb6c46616928776172c7dc9fcca0c8a3ca8b4c6ea31"}
Jul 27 22:15:43.313416 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:43.313005    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq" event={"ID":"068d1618-4c36-42d2-b57a-d04cbe9be77d","Type":"ContainerStarted","Data":"1e77b047e5f208d8b7613f64922b27d1254d2ca4b6391c50383510f4009ce510"}
Jul 27 22:15:47.332468 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:47.332337    2577 generic.go:356] "Generic (PLEG): container finished" podID="068d1618-4c36-42d2-b57a-d04cbe9be77d" containerID="f98d46df0c36f7828fb39fb6c46616928776172c7dc9fcca0c8a3ca8b4c6ea31" exitCode=0
Jul 27 22:15:47.332468 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:47.332424    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq" event={"ID":"068d1618-4c36-42d2-b57a-d04cbe9be77d","Type":"ContainerDied","Data":"f98d46df0c36f7828fb39fb6c46616928776172c7dc9fcca0c8a3ca8b4c6ea31"}
Jul 27 22:15:48.338597 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:48.338564    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq" event={"ID":"068d1618-4c36-42d2-b57a-d04cbe9be77d","Type":"ContainerStarted","Data":"5974366ccae644671d83331cbc1caf08eddf59b43f6d3125a2551a8e3e23d7eb"}
Jul 27 22:15:48.339078 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:48.338623    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq" event={"ID":"068d1618-4c36-42d2-b57a-d04cbe9be77d","Type":"ContainerStarted","Data":"58649611c0ef261e91a564f6a2e7d0acd60e8d8aaaee4ab3eab7df607fbb7ea6"}
Jul 27 22:15:48.339078 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:48.338836    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq"
Jul 27 22:15:48.339078 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:48.338995    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq"
Jul 27 22:15:48.360468 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:48.360425    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq" podStartSLOduration=2.6980702020000003 podStartE2EDuration="6.360412876s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.662342674s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:15:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:15:48.359253719 +0000 UTC m=+4271.726443579" watchObservedRunningTime="2026-07-27 22:15:48.360412876 +0000 UTC m=+4271.727602741"
Jul 27 22:15:54.348466 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:15:54.348440    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq"
Jul 27 22:16:24.409566 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:24.409515    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq" podUID="068d1618-4c36-42d2-b57a-d04cbe9be77d" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Jul 27 22:16:34.351595 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:34.351512    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq"
Jul 27 22:16:42.342789 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:42.342743    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq"]
Jul 27 22:16:42.343281 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:42.342997    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq" podUID="068d1618-4c36-42d2-b57a-d04cbe9be77d" containerName="kserve-container" containerID="cri-o://58649611c0ef261e91a564f6a2e7d0acd60e8d8aaaee4ab3eab7df607fbb7ea6" gracePeriod=30
Jul 27 22:16:42.343281 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:42.343078    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq" podUID="068d1618-4c36-42d2-b57a-d04cbe9be77d" containerName="kube-rbac-proxy" containerID="cri-o://5974366ccae644671d83331cbc1caf08eddf59b43f6d3125a2551a8e3e23d7eb" gracePeriod=30
Jul 27 22:16:42.368780 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:42.368717    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:16:42.550236 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:42.550205    2577 generic.go:356] "Generic (PLEG): container finished" podID="068d1618-4c36-42d2-b57a-d04cbe9be77d" containerID="5974366ccae644671d83331cbc1caf08eddf59b43f6d3125a2551a8e3e23d7eb" exitCode=2
Jul 27 22:16:42.550373 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:42.550263    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq" event={"ID":"068d1618-4c36-42d2-b57a-d04cbe9be77d","Type":"ContainerDied","Data":"5974366ccae644671d83331cbc1caf08eddf59b43f6d3125a2551a8e3e23d7eb"}
Jul 27 22:16:44.344304 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:44.344267    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq" podUID="068d1618-4c36-42d2-b57a-d04cbe9be77d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.61:8643/healthz\": dial tcp 10.133.0.61:8643: connect: connection refused"
Jul 27 22:16:45.390936 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:45.390894    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq" podUID="068d1618-4c36-42d2-b57a-d04cbe9be77d" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.61:8080/v2/models/isvc-sklearn-v2-runtime/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 27 22:16:47.450459 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:47.450419    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:16:47.480474 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:47.480442    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq"]
Jul 27 22:16:47.483932 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:47.483917    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq"
Jul 27 22:16:47.486311 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:47.486286    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
Jul 27 22:16:47.486439 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:47.486317    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
Jul 27 22:16:47.496258 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:47.496237    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq"]
Jul 27 22:16:47.562106 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:47.562074    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a177a020-a0e5-42c2-ab39-42d951b1c779-proxy-tls\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq\" (UID: \"a177a020-a0e5-42c2-ab39-42d951b1c779\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq"
Jul 27 22:16:47.562106 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:47.562103    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a177a020-a0e5-42c2-ab39-42d951b1c779-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq\" (UID: \"a177a020-a0e5-42c2-ab39-42d951b1c779\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq"
Jul 27 22:16:47.562350 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:47.562177    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a177a020-a0e5-42c2-ab39-42d951b1c779-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq\" (UID: \"a177a020-a0e5-42c2-ab39-42d951b1c779\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq"
Jul 27 22:16:47.562350 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:47.562220    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs7lb\" (UniqueName: \"kubernetes.io/projected/a177a020-a0e5-42c2-ab39-42d951b1c779-kube-api-access-cs7lb\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq\" (UID: \"a177a020-a0e5-42c2-ab39-42d951b1c779\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq"
Jul 27 22:16:47.562350 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:47.562254    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a177a020-a0e5-42c2-ab39-42d951b1c779-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq\" (UID: \"a177a020-a0e5-42c2-ab39-42d951b1c779\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq"
Jul 27 22:16:47.662603 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:47.662568    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a177a020-a0e5-42c2-ab39-42d951b1c779-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq\" (UID: \"a177a020-a0e5-42c2-ab39-42d951b1c779\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq"
Jul 27 22:16:47.662819 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:47.662616    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs7lb\" (UniqueName: \"kubernetes.io/projected/a177a020-a0e5-42c2-ab39-42d951b1c779-kube-api-access-cs7lb\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq\" (UID: \"a177a020-a0e5-42c2-ab39-42d951b1c779\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq"
Jul 27 22:16:47.662819 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:47.662651    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a177a020-a0e5-42c2-ab39-42d951b1c779-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq\" (UID: \"a177a020-a0e5-42c2-ab39-42d951b1c779\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq"
Jul 27 22:16:47.662819 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:47.662675    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a177a020-a0e5-42c2-ab39-42d951b1c779-proxy-tls\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq\" (UID: \"a177a020-a0e5-42c2-ab39-42d951b1c779\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq"
Jul 27 22:16:47.662819 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:47.662697    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a177a020-a0e5-42c2-ab39-42d951b1c779-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq\" (UID: \"a177a020-a0e5-42c2-ab39-42d951b1c779\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq"
Jul 27 22:16:47.663039 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:16:47.662821    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-predictor-serving-cert: secret "isvc-sklearn-v2-predictor-serving-cert" not found
Jul 27 22:16:47.663039 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:16:47.662896    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a177a020-a0e5-42c2-ab39-42d951b1c779-proxy-tls podName:a177a020-a0e5-42c2-ab39-42d951b1c779 nodeName:}" failed. No retries permitted until 2026-07-27 22:16:48.162872939 +0000 UTC m=+4331.530062797 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a177a020-a0e5-42c2-ab39-42d951b1c779-proxy-tls") pod "isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq" (UID: "a177a020-a0e5-42c2-ab39-42d951b1c779") : secret "isvc-sklearn-v2-predictor-serving-cert" not found
Jul 27 22:16:47.663155 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:47.663098    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a177a020-a0e5-42c2-ab39-42d951b1c779-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq\" (UID: \"a177a020-a0e5-42c2-ab39-42d951b1c779\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq"
Jul 27 22:16:47.663362 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:47.663344    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a177a020-a0e5-42c2-ab39-42d951b1c779-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq\" (UID: \"a177a020-a0e5-42c2-ab39-42d951b1c779\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq"
Jul 27 22:16:47.663417 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:47.663356    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a177a020-a0e5-42c2-ab39-42d951b1c779-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq\" (UID: \"a177a020-a0e5-42c2-ab39-42d951b1c779\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq"
Jul 27 22:16:47.671251 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:47.671224    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs7lb\" (UniqueName: \"kubernetes.io/projected/a177a020-a0e5-42c2-ab39-42d951b1c779-kube-api-access-cs7lb\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq\" (UID: \"a177a020-a0e5-42c2-ab39-42d951b1c779\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq"
Jul 27 22:16:48.000847 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:48.000812    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:16:48.166414 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:48.166370    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a177a020-a0e5-42c2-ab39-42d951b1c779-proxy-tls\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq\" (UID: \"a177a020-a0e5-42c2-ab39-42d951b1c779\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq"
Jul 27 22:16:48.168830 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:48.168800    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a177a020-a0e5-42c2-ab39-42d951b1c779-proxy-tls\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq\" (UID: \"a177a020-a0e5-42c2-ab39-42d951b1c779\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq"
Jul 27 22:16:48.393801 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:48.393702    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq"
Jul 27 22:16:48.524638 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:48.524602    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq"]
Jul 27 22:16:48.527470 ip-10-0-142-219 kubenswrapper[2577]: W0727 22:16:48.527434    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda177a020_a0e5_42c2_ab39_42d951b1c779.slice/crio-f9656d628cd963f961c2fe21d1040bf343a04894c052937da3433f4dc414e4ef WatchSource:0}: Error finding container f9656d628cd963f961c2fe21d1040bf343a04894c052937da3433f4dc414e4ef: Status 404 returned error can't find the container with id f9656d628cd963f961c2fe21d1040bf343a04894c052937da3433f4dc414e4ef
Jul 27 22:16:48.577094 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:48.577066    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq" event={"ID":"a177a020-a0e5-42c2-ab39-42d951b1c779","Type":"ContainerStarted","Data":"f9656d628cd963f961c2fe21d1040bf343a04894c052937da3433f4dc414e4ef"}
Jul 27 22:16:49.344580 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:49.344536    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq" podUID="068d1618-4c36-42d2-b57a-d04cbe9be77d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.61:8643/healthz\": dial tcp 10.133.0.61:8643: connect: connection refused"
Jul 27 22:16:49.585551 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:49.585320    2577 generic.go:356] "Generic (PLEG): container finished" podID="068d1618-4c36-42d2-b57a-d04cbe9be77d" containerID="58649611c0ef261e91a564f6a2e7d0acd60e8d8aaaee4ab3eab7df607fbb7ea6" exitCode=0
Jul 27 22:16:49.585551 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:49.585437    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq" event={"ID":"068d1618-4c36-42d2-b57a-d04cbe9be77d","Type":"ContainerDied","Data":"58649611c0ef261e91a564f6a2e7d0acd60e8d8aaaee4ab3eab7df607fbb7ea6"}
Jul 27 22:16:49.592743 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:49.592711    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq" event={"ID":"a177a020-a0e5-42c2-ab39-42d951b1c779","Type":"ContainerStarted","Data":"21af10c528420261f28f3102ea4c805efcdcf31f6139233396ef0d8c340acc14"}
Jul 27 22:16:49.697295 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:49.697270    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq"
Jul 27 22:16:49.775213 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:49.775187    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9lhk\" (UniqueName: \"kubernetes.io/projected/068d1618-4c36-42d2-b57a-d04cbe9be77d-kube-api-access-p9lhk\") pod \"068d1618-4c36-42d2-b57a-d04cbe9be77d\" (UID: \"068d1618-4c36-42d2-b57a-d04cbe9be77d\") "
Jul 27 22:16:49.775368 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:49.775218    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/068d1618-4c36-42d2-b57a-d04cbe9be77d-cabundle-cert\") pod \"068d1618-4c36-42d2-b57a-d04cbe9be77d\" (UID: \"068d1618-4c36-42d2-b57a-d04cbe9be77d\") "
Jul 27 22:16:49.775368 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:49.775351    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/068d1618-4c36-42d2-b57a-d04cbe9be77d-kserve-provision-location\") pod \"068d1618-4c36-42d2-b57a-d04cbe9be77d\" (UID: \"068d1618-4c36-42d2-b57a-d04cbe9be77d\") "
Jul 27 22:16:49.775465 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:49.775379    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/068d1618-4c36-42d2-b57a-d04cbe9be77d-kube-rbac-proxy-sar-config\") pod \"068d1618-4c36-42d2-b57a-d04cbe9be77d\" (UID: \"068d1618-4c36-42d2-b57a-d04cbe9be77d\") "
Jul 27 22:16:49.775465 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:49.775412    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/068d1618-4c36-42d2-b57a-d04cbe9be77d-proxy-tls\") pod \"068d1618-4c36-42d2-b57a-d04cbe9be77d\" (UID: \"068d1618-4c36-42d2-b57a-d04cbe9be77d\") "
Jul 27 22:16:49.775650 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:49.775626    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/068d1618-4c36-42d2-b57a-d04cbe9be77d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "068d1618-4c36-42d2-b57a-d04cbe9be77d" (UID: "068d1618-4c36-42d2-b57a-d04cbe9be77d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:16:49.775720 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:49.775677    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/068d1618-4c36-42d2-b57a-d04cbe9be77d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "068d1618-4c36-42d2-b57a-d04cbe9be77d" (UID: "068d1618-4c36-42d2-b57a-d04cbe9be77d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:16:49.775792 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:49.775714    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/068d1618-4c36-42d2-b57a-d04cbe9be77d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "068d1618-4c36-42d2-b57a-d04cbe9be77d" (UID: "068d1618-4c36-42d2-b57a-d04cbe9be77d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:16:49.777267 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:49.777246    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/068d1618-4c36-42d2-b57a-d04cbe9be77d-kube-api-access-p9lhk" (OuterVolumeSpecName: "kube-api-access-p9lhk") pod "068d1618-4c36-42d2-b57a-d04cbe9be77d" (UID: "068d1618-4c36-42d2-b57a-d04cbe9be77d"). InnerVolumeSpecName "kube-api-access-p9lhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:16:49.777330 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:49.777313    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/068d1618-4c36-42d2-b57a-d04cbe9be77d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "068d1618-4c36-42d2-b57a-d04cbe9be77d" (UID: "068d1618-4c36-42d2-b57a-d04cbe9be77d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 22:16:49.876875 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:49.876806    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/068d1618-4c36-42d2-b57a-d04cbe9be77d-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:16:49.876875 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:49.876830    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/068d1618-4c36-42d2-b57a-d04cbe9be77d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:16:49.876875 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:49.876840    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/068d1618-4c36-42d2-b57a-d04cbe9be77d-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:16:49.876875 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:49.876848    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9lhk\" (UniqueName: \"kubernetes.io/projected/068d1618-4c36-42d2-b57a-d04cbe9be77d-kube-api-access-p9lhk\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:16:49.876875 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:49.876856    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/068d1618-4c36-42d2-b57a-d04cbe9be77d-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:16:50.598940 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:50.598912    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq"
Jul 27 22:16:50.598940 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:50.598932    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq" event={"ID":"068d1618-4c36-42d2-b57a-d04cbe9be77d","Type":"ContainerDied","Data":"1e77b047e5f208d8b7613f64922b27d1254d2ca4b6391c50383510f4009ce510"}
Jul 27 22:16:50.599478 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:50.599005    2577 scope.go:117] "RemoveContainer" containerID="5974366ccae644671d83331cbc1caf08eddf59b43f6d3125a2551a8e3e23d7eb"
Jul 27 22:16:50.607189 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:50.607167    2577 scope.go:117] "RemoveContainer" containerID="58649611c0ef261e91a564f6a2e7d0acd60e8d8aaaee4ab3eab7df607fbb7ea6"
Jul 27 22:16:50.614263 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:50.614242    2577 scope.go:117] "RemoveContainer" containerID="f98d46df0c36f7828fb39fb6c46616928776172c7dc9fcca0c8a3ca8b4c6ea31"
Jul 27 22:16:50.620471 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:50.620449    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq"]
Jul 27 22:16:50.624417 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:50.624396    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-2wmqq"]
Jul 27 22:16:51.135068 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:51.135036    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="068d1618-4c36-42d2-b57a-d04cbe9be77d" path="/var/lib/kubelet/pods/068d1618-4c36-42d2-b57a-d04cbe9be77d/volumes"
Jul 27 22:16:52.612444 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:52.612361    2577 generic.go:356] "Generic (PLEG): container finished" podID="a177a020-a0e5-42c2-ab39-42d951b1c779" containerID="21af10c528420261f28f3102ea4c805efcdcf31f6139233396ef0d8c340acc14" exitCode=0
Jul 27 22:16:52.612839 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:52.612436    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq" event={"ID":"a177a020-a0e5-42c2-ab39-42d951b1c779","Type":"ContainerDied","Data":"21af10c528420261f28f3102ea4c805efcdcf31f6139233396ef0d8c340acc14"}
Jul 27 22:16:53.619550 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:53.619515    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq" event={"ID":"a177a020-a0e5-42c2-ab39-42d951b1c779","Type":"ContainerStarted","Data":"59aa92645062329259e44ede5b672a6cad4a8585e324c233ae21ee6c67a2bbc5"}
Jul 27 22:16:53.619966 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:53.619578    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq" event={"ID":"a177a020-a0e5-42c2-ab39-42d951b1c779","Type":"ContainerStarted","Data":"bfe86afad721cb17b1190ace68cb38870a4188d07e8f0443d9e08cf5bc201abc"}
Jul 27 22:16:53.619966 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:53.619801    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq"
Jul 27 22:16:53.639235 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:53.639188    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq" podStartSLOduration=2.869056497 podStartE2EDuration="6.639172901s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.770116404s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:16:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:16:53.637735337 +0000 UTC m=+4337.004925217" watchObservedRunningTime="2026-07-27 22:16:53.639172901 +0000 UTC m=+4337.006362766"
Jul 27 22:16:54.623317 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:54.623279    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq"
Jul 27 22:16:54.624576 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:54.624549    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq" podUID="a177a020-a0e5-42c2-ab39-42d951b1c779" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Jul 27 22:16:55.626511 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:16:55.626472    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq" podUID="a177a020-a0e5-42c2-ab39-42d951b1c779" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Jul 27 22:17:00.631050 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:17:00.631023    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq"
Jul 27 22:17:00.631510 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:17:00.631483    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq" podUID="a177a020-a0e5-42c2-ab39-42d951b1c779" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Jul 27 22:17:10.632386 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:17:10.632345    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq" podUID="a177a020-a0e5-42c2-ab39-42d951b1c779" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Jul 27 22:17:20.632309 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:17:20.632262    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq" podUID="a177a020-a0e5-42c2-ab39-42d951b1c779" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Jul 27 22:17:30.631531 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:17:30.631485    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq" podUID="a177a020-a0e5-42c2-ab39-42d951b1c779" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Jul 27 22:17:31.672443 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:17:31.672408    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:17:40.632102 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:17:40.632045    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq" podUID="a177a020-a0e5-42c2-ab39-42d951b1c779" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Jul 27 22:17:41.839510 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:17:41.839472    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:17:50.631693 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:17:50.631652    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq" podUID="a177a020-a0e5-42c2-ab39-42d951b1c779" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Jul 27 22:18:00.632211 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:00.632117    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq" podUID="a177a020-a0e5-42c2-ab39-42d951b1c779" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Jul 27 22:18:01.131333 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:01.131292    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq" podUID="a177a020-a0e5-42c2-ab39-42d951b1c779" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Jul 27 22:18:11.134393 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:11.134363    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq"
Jul 27 22:18:17.704466 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:17.704431    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq"]
Jul 27 22:18:17.704866 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:17.704665    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq" podUID="a177a020-a0e5-42c2-ab39-42d951b1c779" containerName="kserve-container" containerID="cri-o://bfe86afad721cb17b1190ace68cb38870a4188d07e8f0443d9e08cf5bc201abc" gracePeriod=30
Jul 27 22:18:17.704866 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:17.704697    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq" podUID="a177a020-a0e5-42c2-ab39-42d951b1c779" containerName="kube-rbac-proxy" containerID="cri-o://59aa92645062329259e44ede5b672a6cad4a8585e324c233ae21ee6c67a2bbc5" gracePeriod=30
Jul 27 22:18:17.725279 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:17.725251    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:18:17.941862 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:17.941824    2577 generic.go:356] "Generic (PLEG): container finished" podID="a177a020-a0e5-42c2-ab39-42d951b1c779" containerID="59aa92645062329259e44ede5b672a6cad4a8585e324c233ae21ee6c67a2bbc5" exitCode=2
Jul 27 22:18:17.942020 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:17.941879    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq" event={"ID":"a177a020-a0e5-42c2-ab39-42d951b1c779","Type":"ContainerDied","Data":"59aa92645062329259e44ede5b672a6cad4a8585e324c233ae21ee6c67a2bbc5"}
Jul 27 22:18:20.627003 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:20.626962    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq" podUID="a177a020-a0e5-42c2-ab39-42d951b1c779" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.62:8643/healthz\": dial tcp 10.133.0.62:8643: connect: connection refused"
Jul 27 22:18:21.131543 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:21.131501    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq" podUID="a177a020-a0e5-42c2-ab39-42d951b1c779" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Jul 27 22:18:22.248098 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.248075    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq"
Jul 27 22:18:22.315469 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.315400    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a177a020-a0e5-42c2-ab39-42d951b1c779-kserve-provision-location\") pod \"a177a020-a0e5-42c2-ab39-42d951b1c779\" (UID: \"a177a020-a0e5-42c2-ab39-42d951b1c779\") "
Jul 27 22:18:22.315469 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.315431    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a177a020-a0e5-42c2-ab39-42d951b1c779-kube-rbac-proxy-sar-config\") pod \"a177a020-a0e5-42c2-ab39-42d951b1c779\" (UID: \"a177a020-a0e5-42c2-ab39-42d951b1c779\") "
Jul 27 22:18:22.315469 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.315450    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cs7lb\" (UniqueName: \"kubernetes.io/projected/a177a020-a0e5-42c2-ab39-42d951b1c779-kube-api-access-cs7lb\") pod \"a177a020-a0e5-42c2-ab39-42d951b1c779\" (UID: \"a177a020-a0e5-42c2-ab39-42d951b1c779\") "
Jul 27 22:18:22.315706 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.315545    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a177a020-a0e5-42c2-ab39-42d951b1c779-proxy-tls\") pod \"a177a020-a0e5-42c2-ab39-42d951b1c779\" (UID: \"a177a020-a0e5-42c2-ab39-42d951b1c779\") "
Jul 27 22:18:22.315706 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.315582    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a177a020-a0e5-42c2-ab39-42d951b1c779-cabundle-cert\") pod \"a177a020-a0e5-42c2-ab39-42d951b1c779\" (UID: \"a177a020-a0e5-42c2-ab39-42d951b1c779\") "
Jul 27 22:18:22.315852 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.315745    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a177a020-a0e5-42c2-ab39-42d951b1c779-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a177a020-a0e5-42c2-ab39-42d951b1c779" (UID: "a177a020-a0e5-42c2-ab39-42d951b1c779"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:18:22.315943 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.315814    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a177a020-a0e5-42c2-ab39-42d951b1c779-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a177a020-a0e5-42c2-ab39-42d951b1c779" (UID: "a177a020-a0e5-42c2-ab39-42d951b1c779"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:18:22.316007 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.315987    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a177a020-a0e5-42c2-ab39-42d951b1c779-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a177a020-a0e5-42c2-ab39-42d951b1c779" (UID: "a177a020-a0e5-42c2-ab39-42d951b1c779"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:18:22.317524 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.317499    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a177a020-a0e5-42c2-ab39-42d951b1c779-kube-api-access-cs7lb" (OuterVolumeSpecName: "kube-api-access-cs7lb") pod "a177a020-a0e5-42c2-ab39-42d951b1c779" (UID: "a177a020-a0e5-42c2-ab39-42d951b1c779"). InnerVolumeSpecName "kube-api-access-cs7lb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:18:22.317629 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.317580    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a177a020-a0e5-42c2-ab39-42d951b1c779-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a177a020-a0e5-42c2-ab39-42d951b1c779" (UID: "a177a020-a0e5-42c2-ab39-42d951b1c779"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 22:18:22.416422 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.416393    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a177a020-a0e5-42c2-ab39-42d951b1c779-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:18:22.416422 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.416422    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a177a020-a0e5-42c2-ab39-42d951b1c779-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:18:22.416588 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.416432    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a177a020-a0e5-42c2-ab39-42d951b1c779-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:18:22.416588 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.416441    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs7lb\" (UniqueName: \"kubernetes.io/projected/a177a020-a0e5-42c2-ab39-42d951b1c779-kube-api-access-cs7lb\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:18:22.416588 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.416451    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a177a020-a0e5-42c2-ab39-42d951b1c779-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:18:22.774440 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.774410    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:18:22.828563 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.828531    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-dntff"]
Jul 27 22:18:22.828824 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.828812    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="068d1618-4c36-42d2-b57a-d04cbe9be77d" containerName="kserve-container"
Jul 27 22:18:22.828874 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.828825    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="068d1618-4c36-42d2-b57a-d04cbe9be77d" containerName="kserve-container"
Jul 27 22:18:22.828874 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.828838    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a177a020-a0e5-42c2-ab39-42d951b1c779" containerName="storage-initializer"
Jul 27 22:18:22.828874 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.828844    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a177a020-a0e5-42c2-ab39-42d951b1c779" containerName="storage-initializer"
Jul 27 22:18:22.828874 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.828854    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="068d1618-4c36-42d2-b57a-d04cbe9be77d" containerName="storage-initializer"
Jul 27 22:18:22.828874 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.828860    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="068d1618-4c36-42d2-b57a-d04cbe9be77d" containerName="storage-initializer"
Jul 27 22:18:22.828874 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.828871    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="068d1618-4c36-42d2-b57a-d04cbe9be77d" containerName="kube-rbac-proxy"
Jul 27 22:18:22.829047 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.828878    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="068d1618-4c36-42d2-b57a-d04cbe9be77d" containerName="kube-rbac-proxy"
Jul 27 22:18:22.829047 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.828886    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a177a020-a0e5-42c2-ab39-42d951b1c779" containerName="kserve-container"
Jul 27 22:18:22.829047 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.828891    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a177a020-a0e5-42c2-ab39-42d951b1c779" containerName="kserve-container"
Jul 27 22:18:22.829047 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.828896    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a177a020-a0e5-42c2-ab39-42d951b1c779" containerName="kube-rbac-proxy"
Jul 27 22:18:22.829047 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.828901    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a177a020-a0e5-42c2-ab39-42d951b1c779" containerName="kube-rbac-proxy"
Jul 27 22:18:22.829047 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.828946    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="068d1618-4c36-42d2-b57a-d04cbe9be77d" containerName="kserve-container"
Jul 27 22:18:22.829047 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.828953    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a177a020-a0e5-42c2-ab39-42d951b1c779" containerName="kube-rbac-proxy"
Jul 27 22:18:22.829047 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.828959    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="068d1618-4c36-42d2-b57a-d04cbe9be77d" containerName="kube-rbac-proxy"
Jul 27 22:18:22.829047 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.828972    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a177a020-a0e5-42c2-ab39-42d951b1c779" containerName="kserve-container"
Jul 27 22:18:22.832266 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.832249    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-dntff"
Jul 27 22:18:22.834724 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.834704    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Jul 27 22:18:22.834857 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.834702    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Jul 27 22:18:22.842693 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.842671    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-dntff"]
Jul 27 22:18:22.868175 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.868147    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:18:22.920659 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.920635    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76fb6faa-350a-413f-b335-0dd449c1eb6f-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-dntff\" (UID: \"76fb6faa-350a-413f-b335-0dd449c1eb6f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-dntff"
Jul 27 22:18:22.920746 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.920688    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76fb6faa-350a-413f-b335-0dd449c1eb6f-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-dntff\" (UID: \"76fb6faa-350a-413f-b335-0dd449c1eb6f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-dntff"
Jul 27 22:18:22.920832 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.920738    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76fb6faa-350a-413f-b335-0dd449c1eb6f-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-dntff\" (UID: \"76fb6faa-350a-413f-b335-0dd449c1eb6f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-dntff"
Jul 27 22:18:22.920832 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.920815    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ppkw\" (UniqueName: \"kubernetes.io/projected/76fb6faa-350a-413f-b335-0dd449c1eb6f-kube-api-access-9ppkw\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-dntff\" (UID: \"76fb6faa-350a-413f-b335-0dd449c1eb6f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-dntff"
Jul 27 22:18:22.920903 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.920842    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/76fb6faa-350a-413f-b335-0dd449c1eb6f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-dntff\" (UID: \"76fb6faa-350a-413f-b335-0dd449c1eb6f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-dntff"
Jul 27 22:18:22.963590 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.963566    2577 generic.go:356] "Generic (PLEG): container finished" podID="a177a020-a0e5-42c2-ab39-42d951b1c779" containerID="bfe86afad721cb17b1190ace68cb38870a4188d07e8f0443d9e08cf5bc201abc" exitCode=0
Jul 27 22:18:22.963677 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.963611    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq" event={"ID":"a177a020-a0e5-42c2-ab39-42d951b1c779","Type":"ContainerDied","Data":"bfe86afad721cb17b1190ace68cb38870a4188d07e8f0443d9e08cf5bc201abc"}
Jul 27 22:18:22.963677 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.963625    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq"
Jul 27 22:18:22.963677 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.963663    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq" event={"ID":"a177a020-a0e5-42c2-ab39-42d951b1c779","Type":"ContainerDied","Data":"f9656d628cd963f961c2fe21d1040bf343a04894c052937da3433f4dc414e4ef"}
Jul 27 22:18:22.963837 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.963690    2577 scope.go:117] "RemoveContainer" containerID="59aa92645062329259e44ede5b672a6cad4a8585e324c233ae21ee6c67a2bbc5"
Jul 27 22:18:22.972015 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.971992    2577 scope.go:117] "RemoveContainer" containerID="bfe86afad721cb17b1190ace68cb38870a4188d07e8f0443d9e08cf5bc201abc"
Jul 27 22:18:22.979064 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.979048    2577 scope.go:117] "RemoveContainer" containerID="21af10c528420261f28f3102ea4c805efcdcf31f6139233396ef0d8c340acc14"
Jul 27 22:18:22.985968 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.985950    2577 scope.go:117] "RemoveContainer" containerID="59aa92645062329259e44ede5b672a6cad4a8585e324c233ae21ee6c67a2bbc5"
Jul 27 22:18:22.986243 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:18:22.986225    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59aa92645062329259e44ede5b672a6cad4a8585e324c233ae21ee6c67a2bbc5\": container with ID starting with 59aa92645062329259e44ede5b672a6cad4a8585e324c233ae21ee6c67a2bbc5 not found: ID does not exist" containerID="59aa92645062329259e44ede5b672a6cad4a8585e324c233ae21ee6c67a2bbc5"
Jul 27 22:18:22.986309 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.986252    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59aa92645062329259e44ede5b672a6cad4a8585e324c233ae21ee6c67a2bbc5"} err="failed to get container status \"59aa92645062329259e44ede5b672a6cad4a8585e324c233ae21ee6c67a2bbc5\": rpc error: code = NotFound desc = could not find container \"59aa92645062329259e44ede5b672a6cad4a8585e324c233ae21ee6c67a2bbc5\": container with ID starting with 59aa92645062329259e44ede5b672a6cad4a8585e324c233ae21ee6c67a2bbc5 not found: ID does not exist"
Jul 27 22:18:22.986309 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.986273    2577 scope.go:117] "RemoveContainer" containerID="bfe86afad721cb17b1190ace68cb38870a4188d07e8f0443d9e08cf5bc201abc"
Jul 27 22:18:22.986503 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:18:22.986486    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfe86afad721cb17b1190ace68cb38870a4188d07e8f0443d9e08cf5bc201abc\": container with ID starting with bfe86afad721cb17b1190ace68cb38870a4188d07e8f0443d9e08cf5bc201abc not found: ID does not exist" containerID="bfe86afad721cb17b1190ace68cb38870a4188d07e8f0443d9e08cf5bc201abc"
Jul 27 22:18:22.986547 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.986509    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfe86afad721cb17b1190ace68cb38870a4188d07e8f0443d9e08cf5bc201abc"} err="failed to get container status \"bfe86afad721cb17b1190ace68cb38870a4188d07e8f0443d9e08cf5bc201abc\": rpc error: code = NotFound desc = could not find container \"bfe86afad721cb17b1190ace68cb38870a4188d07e8f0443d9e08cf5bc201abc\": container with ID starting with bfe86afad721cb17b1190ace68cb38870a4188d07e8f0443d9e08cf5bc201abc not found: ID does not exist"
Jul 27 22:18:22.986547 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.986527    2577 scope.go:117] "RemoveContainer" containerID="21af10c528420261f28f3102ea4c805efcdcf31f6139233396ef0d8c340acc14"
Jul 27 22:18:22.986737 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:18:22.986722    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21af10c528420261f28f3102ea4c805efcdcf31f6139233396ef0d8c340acc14\": container with ID starting with 21af10c528420261f28f3102ea4c805efcdcf31f6139233396ef0d8c340acc14 not found: ID does not exist" containerID="21af10c528420261f28f3102ea4c805efcdcf31f6139233396ef0d8c340acc14"
Jul 27 22:18:22.986859 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.986740    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21af10c528420261f28f3102ea4c805efcdcf31f6139233396ef0d8c340acc14"} err="failed to get container status \"21af10c528420261f28f3102ea4c805efcdcf31f6139233396ef0d8c340acc14\": rpc error: code = NotFound desc = could not find container \"21af10c528420261f28f3102ea4c805efcdcf31f6139233396ef0d8c340acc14\": container with ID starting with 21af10c528420261f28f3102ea4c805efcdcf31f6139233396ef0d8c340acc14 not found: ID does not exist"
Jul 27 22:18:22.986950 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.986933    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq"]
Jul 27 22:18:22.990440 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:22.990423    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-cdrnq"]
Jul 27 22:18:23.021626 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:23.021604    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76fb6faa-350a-413f-b335-0dd449c1eb6f-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-dntff\" (UID: \"76fb6faa-350a-413f-b335-0dd449c1eb6f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-dntff"
Jul 27 22:18:23.021695 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:23.021634    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76fb6faa-350a-413f-b335-0dd449c1eb6f-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-dntff\" (UID: \"76fb6faa-350a-413f-b335-0dd449c1eb6f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-dntff"
Jul 27 22:18:23.021695 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:23.021657    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ppkw\" (UniqueName: \"kubernetes.io/projected/76fb6faa-350a-413f-b335-0dd449c1eb6f-kube-api-access-9ppkw\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-dntff\" (UID: \"76fb6faa-350a-413f-b335-0dd449c1eb6f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-dntff"
Jul 27 22:18:23.021695 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:23.021677    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/76fb6faa-350a-413f-b335-0dd449c1eb6f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-dntff\" (UID: \"76fb6faa-350a-413f-b335-0dd449c1eb6f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-dntff"
Jul 27 22:18:23.021874 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:23.021719    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76fb6faa-350a-413f-b335-0dd449c1eb6f-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-dntff\" (UID: \"76fb6faa-350a-413f-b335-0dd449c1eb6f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-dntff"
Jul 27 22:18:23.021997 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:23.021977    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76fb6faa-350a-413f-b335-0dd449c1eb6f-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-dntff\" (UID: \"76fb6faa-350a-413f-b335-0dd449c1eb6f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-dntff"
Jul 27 22:18:23.022391 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:23.022369    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76fb6faa-350a-413f-b335-0dd449c1eb6f-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-dntff\" (UID: \"76fb6faa-350a-413f-b335-0dd449c1eb6f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-dntff"
Jul 27 22:18:23.022438 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:23.022377    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/76fb6faa-350a-413f-b335-0dd449c1eb6f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-dntff\" (UID: \"76fb6faa-350a-413f-b335-0dd449c1eb6f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-dntff"
Jul 27 22:18:23.023928 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:23.023911    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76fb6faa-350a-413f-b335-0dd449c1eb6f-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-dntff\" (UID: \"76fb6faa-350a-413f-b335-0dd449c1eb6f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-dntff"
Jul 27 22:18:23.029806 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:23.029731    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ppkw\" (UniqueName: \"kubernetes.io/projected/76fb6faa-350a-413f-b335-0dd449c1eb6f-kube-api-access-9ppkw\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-dntff\" (UID: \"76fb6faa-350a-413f-b335-0dd449c1eb6f\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-dntff"
Jul 27 22:18:23.134986 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:23.134961    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a177a020-a0e5-42c2-ab39-42d951b1c779" path="/var/lib/kubelet/pods/a177a020-a0e5-42c2-ab39-42d951b1c779/volumes"
Jul 27 22:18:23.143184 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:23.143165    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-dntff"
Jul 27 22:18:23.270675 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:23.270646    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-dntff"]
Jul 27 22:18:23.273197 ip-10-0-142-219 kubenswrapper[2577]: W0727 22:18:23.273169    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76fb6faa_350a_413f_b335_0dd449c1eb6f.slice/crio-da6bf722f6f314901e4b3a538ad830a646ca83e39be88885089fee9f3626e15c WatchSource:0}: Error finding container da6bf722f6f314901e4b3a538ad830a646ca83e39be88885089fee9f3626e15c: Status 404 returned error can't find the container with id da6bf722f6f314901e4b3a538ad830a646ca83e39be88885089fee9f3626e15c
Jul 27 22:18:23.970129 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:23.970090    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-dntff" event={"ID":"76fb6faa-350a-413f-b335-0dd449c1eb6f","Type":"ContainerStarted","Data":"d6c4a4036e0d430717cf0da150da03963dfb2db9f9b0c298e701d236d6acddd1"}
Jul 27 22:18:23.970306 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:23.970149    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-dntff" event={"ID":"76fb6faa-350a-413f-b335-0dd449c1eb6f","Type":"ContainerStarted","Data":"da6bf722f6f314901e4b3a538ad830a646ca83e39be88885089fee9f3626e15c"}
Jul 27 22:18:26.983309 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:26.983275    2577 generic.go:356] "Generic (PLEG): container finished" podID="76fb6faa-350a-413f-b335-0dd449c1eb6f" containerID="d6c4a4036e0d430717cf0da150da03963dfb2db9f9b0c298e701d236d6acddd1" exitCode=0
Jul 27 22:18:26.983633 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:26.983338    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-dntff" event={"ID":"76fb6faa-350a-413f-b335-0dd449c1eb6f","Type":"ContainerDied","Data":"d6c4a4036e0d430717cf0da150da03963dfb2db9f9b0c298e701d236d6acddd1"}
Jul 27 22:18:27.989037 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:27.989000    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-dntff" event={"ID":"76fb6faa-350a-413f-b335-0dd449c1eb6f","Type":"ContainerStarted","Data":"347a929ecc9c91527f25fa0b5eeb01668066f196355bc071685e06ad7e16a8f3"}
Jul 27 22:18:27.989526 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:27.989069    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-dntff" event={"ID":"76fb6faa-350a-413f-b335-0dd449c1eb6f","Type":"ContainerStarted","Data":"e019cb2c518e435b9af88892c12cbe800da2240156c8e6d17f1d610aa1970706"}
Jul 27 22:18:27.989526 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:27.989280    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-dntff"
Jul 27 22:18:28.008569 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:28.008517    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-dntff" podStartSLOduration=2.46792897 podStartE2EDuration="6.008505183s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.540576213s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:18:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:18:28.006795361 +0000 UTC m=+4431.373985265" watchObservedRunningTime="2026-07-27 22:18:28.008505183 +0000 UTC m=+4431.375695048"
Jul 27 22:18:28.992576 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:28.992539    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-dntff"
Jul 27 22:18:28.993797 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:28.993735    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-dntff" podUID="76fb6faa-350a-413f-b335-0dd449c1eb6f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Jul 27 22:18:29.996029 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:29.995984    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-dntff" podUID="76fb6faa-350a-413f-b335-0dd449c1eb6f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Jul 27 22:18:35.000565 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:35.000534    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-dntff"
Jul 27 22:18:35.001218 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:35.001190    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-dntff" podUID="76fb6faa-350a-413f-b335-0dd449c1eb6f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Jul 27 22:18:45.001133 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:45.001083    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-dntff" podUID="76fb6faa-350a-413f-b335-0dd449c1eb6f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Jul 27 22:18:55.001740 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:18:55.001700    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-dntff" podUID="76fb6faa-350a-413f-b335-0dd449c1eb6f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Jul 27 22:19:05.002055 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:05.002009    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-dntff" podUID="76fb6faa-350a-413f-b335-0dd449c1eb6f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Jul 27 22:19:15.001203 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:15.001159    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-dntff" podUID="76fb6faa-350a-413f-b335-0dd449c1eb6f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Jul 27 22:19:25.001384 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:25.001346    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-dntff" podUID="76fb6faa-350a-413f-b335-0dd449c1eb6f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Jul 27 22:19:35.001249 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:35.001160    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-dntff" podUID="76fb6faa-350a-413f-b335-0dd449c1eb6f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Jul 27 22:19:38.130333 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:38.130295    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-dntff" podUID="76fb6faa-350a-413f-b335-0dd449c1eb6f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Jul 27 22:19:48.130942 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:48.130906    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-dntff"
Jul 27 22:19:53.004555 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:53.004521    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-dntff"]
Jul 27 22:19:53.005088 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:53.004808    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-dntff" podUID="76fb6faa-350a-413f-b335-0dd449c1eb6f" containerName="kserve-container" containerID="cri-o://e019cb2c518e435b9af88892c12cbe800da2240156c8e6d17f1d610aa1970706" gracePeriod=30
Jul 27 22:19:53.005088 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:53.004870    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-dntff" podUID="76fb6faa-350a-413f-b335-0dd449c1eb6f" containerName="kube-rbac-proxy" containerID="cri-o://347a929ecc9c91527f25fa0b5eeb01668066f196355bc071685e06ad7e16a8f3" gracePeriod=30
Jul 27 22:19:53.052933 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:53.052883    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:19:53.317144 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:53.317043    2577 generic.go:356] "Generic (PLEG): container finished" podID="76fb6faa-350a-413f-b335-0dd449c1eb6f" containerID="347a929ecc9c91527f25fa0b5eeb01668066f196355bc071685e06ad7e16a8f3" exitCode=2
Jul 27 22:19:53.317144 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:53.317111    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-dntff" event={"ID":"76fb6faa-350a-413f-b335-0dd449c1eb6f","Type":"ContainerDied","Data":"347a929ecc9c91527f25fa0b5eeb01668066f196355bc071685e06ad7e16a8f3"}
Jul 27 22:19:53.845505 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:53.845455    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:19:54.996406 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:54.996365    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-dntff" podUID="76fb6faa-350a-413f-b335-0dd449c1eb6f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.63:8643/healthz\": dial tcp 10.133.0.63:8643: connect: connection refused"
Jul 27 22:19:57.342704 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:57.342668    2577 generic.go:356] "Generic (PLEG): container finished" podID="76fb6faa-350a-413f-b335-0dd449c1eb6f" containerID="e019cb2c518e435b9af88892c12cbe800da2240156c8e6d17f1d610aa1970706" exitCode=0
Jul 27 22:19:57.343048 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:57.342726    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-dntff" event={"ID":"76fb6faa-350a-413f-b335-0dd449c1eb6f","Type":"ContainerDied","Data":"e019cb2c518e435b9af88892c12cbe800da2240156c8e6d17f1d610aa1970706"}
Jul 27 22:19:57.454698 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:57.454676    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-dntff"
Jul 27 22:19:57.499939 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:57.499869    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ppkw\" (UniqueName: \"kubernetes.io/projected/76fb6faa-350a-413f-b335-0dd449c1eb6f-kube-api-access-9ppkw\") pod \"76fb6faa-350a-413f-b335-0dd449c1eb6f\" (UID: \"76fb6faa-350a-413f-b335-0dd449c1eb6f\") "
Jul 27 22:19:57.499939 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:57.499908    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76fb6faa-350a-413f-b335-0dd449c1eb6f-proxy-tls\") pod \"76fb6faa-350a-413f-b335-0dd449c1eb6f\" (UID: \"76fb6faa-350a-413f-b335-0dd449c1eb6f\") "
Jul 27 22:19:57.499939 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:57.499925    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/76fb6faa-350a-413f-b335-0dd449c1eb6f-kube-rbac-proxy-sar-config\") pod \"76fb6faa-350a-413f-b335-0dd449c1eb6f\" (UID: \"76fb6faa-350a-413f-b335-0dd449c1eb6f\") "
Jul 27 22:19:57.500213 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:57.499974    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76fb6faa-350a-413f-b335-0dd449c1eb6f-cabundle-cert\") pod \"76fb6faa-350a-413f-b335-0dd449c1eb6f\" (UID: \"76fb6faa-350a-413f-b335-0dd449c1eb6f\") "
Jul 27 22:19:57.500213 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:57.499993    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76fb6faa-350a-413f-b335-0dd449c1eb6f-kserve-provision-location\") pod \"76fb6faa-350a-413f-b335-0dd449c1eb6f\" (UID: \"76fb6faa-350a-413f-b335-0dd449c1eb6f\") "
Jul 27 22:19:57.500356 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:57.500330    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76fb6faa-350a-413f-b335-0dd449c1eb6f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "76fb6faa-350a-413f-b335-0dd449c1eb6f" (UID: "76fb6faa-350a-413f-b335-0dd449c1eb6f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:19:57.500412 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:57.500361    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76fb6faa-350a-413f-b335-0dd449c1eb6f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "76fb6faa-350a-413f-b335-0dd449c1eb6f" (UID: "76fb6faa-350a-413f-b335-0dd449c1eb6f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:19:57.500412 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:57.500362    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76fb6faa-350a-413f-b335-0dd449c1eb6f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "76fb6faa-350a-413f-b335-0dd449c1eb6f" (UID: "76fb6faa-350a-413f-b335-0dd449c1eb6f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:19:57.502034 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:57.502008    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76fb6faa-350a-413f-b335-0dd449c1eb6f-kube-api-access-9ppkw" (OuterVolumeSpecName: "kube-api-access-9ppkw") pod "76fb6faa-350a-413f-b335-0dd449c1eb6f" (UID: "76fb6faa-350a-413f-b335-0dd449c1eb6f"). InnerVolumeSpecName "kube-api-access-9ppkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:19:57.502034 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:57.502013    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76fb6faa-350a-413f-b335-0dd449c1eb6f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "76fb6faa-350a-413f-b335-0dd449c1eb6f" (UID: "76fb6faa-350a-413f-b335-0dd449c1eb6f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 22:19:57.601131 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:57.601092    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ppkw\" (UniqueName: \"kubernetes.io/projected/76fb6faa-350a-413f-b335-0dd449c1eb6f-kube-api-access-9ppkw\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:19:57.601131 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:57.601126    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76fb6faa-350a-413f-b335-0dd449c1eb6f-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:19:57.601131 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:57.601139    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/76fb6faa-350a-413f-b335-0dd449c1eb6f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:19:57.601365 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:57.601150    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76fb6faa-350a-413f-b335-0dd449c1eb6f-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:19:57.601365 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:57.601162    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76fb6faa-350a-413f-b335-0dd449c1eb6f-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:19:58.105430 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:58.105376    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:19:58.167226 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:58.167193    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-m4kxv"]
Jul 27 22:19:58.167518 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:58.167505    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76fb6faa-350a-413f-b335-0dd449c1eb6f" containerName="storage-initializer"
Jul 27 22:19:58.167578 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:58.167520    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="76fb6faa-350a-413f-b335-0dd449c1eb6f" containerName="storage-initializer"
Jul 27 22:19:58.167578 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:58.167531    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76fb6faa-350a-413f-b335-0dd449c1eb6f" containerName="kserve-container"
Jul 27 22:19:58.167578 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:58.167538    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="76fb6faa-350a-413f-b335-0dd449c1eb6f" containerName="kserve-container"
Jul 27 22:19:58.167578 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:58.167554    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76fb6faa-350a-413f-b335-0dd449c1eb6f" containerName="kube-rbac-proxy"
Jul 27 22:19:58.167578 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:58.167563    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="76fb6faa-350a-413f-b335-0dd449c1eb6f" containerName="kube-rbac-proxy"
Jul 27 22:19:58.167851 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:58.167613    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="76fb6faa-350a-413f-b335-0dd449c1eb6f" containerName="kserve-container"
Jul 27 22:19:58.167851 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:58.167625    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="76fb6faa-350a-413f-b335-0dd449c1eb6f" containerName="kube-rbac-proxy"
Jul 27 22:19:58.170903 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:58.170875    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-m4kxv"
Jul 27 22:19:58.173997 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:58.173976    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-predictor-serving-cert\""
Jul 27 22:19:58.174487 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:58.174456    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
Jul 27 22:19:58.182411 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:58.182391    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-m4kxv"]
Jul 27 22:19:58.203592 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:58.203562    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/59790390-7bdf-4b6f-9ef8-a9f572475caf-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-m4kxv\" (UID: \"59790390-7bdf-4b6f-9ef8-a9f572475caf\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-m4kxv"
Jul 27 22:19:58.203779 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:58.203621    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59790390-7bdf-4b6f-9ef8-a9f572475caf-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-m4kxv\" (UID: \"59790390-7bdf-4b6f-9ef8-a9f572475caf\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-m4kxv"
Jul 27 22:19:58.203779 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:58.203686    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59790390-7bdf-4b6f-9ef8-a9f572475caf-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-m4kxv\" (UID: \"59790390-7bdf-4b6f-9ef8-a9f572475caf\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-m4kxv"
Jul 27 22:19:58.203779 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:58.203727    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wr4q\" (UniqueName: \"kubernetes.io/projected/59790390-7bdf-4b6f-9ef8-a9f572475caf-kube-api-access-2wr4q\") pod \"isvc-tensorflow-predictor-547dfc5669-m4kxv\" (UID: \"59790390-7bdf-4b6f-9ef8-a9f572475caf\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-m4kxv"
Jul 27 22:19:58.203942 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:58.203802    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/59790390-7bdf-4b6f-9ef8-a9f572475caf-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-m4kxv\" (UID: \"59790390-7bdf-4b6f-9ef8-a9f572475caf\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-m4kxv"
Jul 27 22:19:58.304806 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:58.304752    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wr4q\" (UniqueName: \"kubernetes.io/projected/59790390-7bdf-4b6f-9ef8-a9f572475caf-kube-api-access-2wr4q\") pod \"isvc-tensorflow-predictor-547dfc5669-m4kxv\" (UID: \"59790390-7bdf-4b6f-9ef8-a9f572475caf\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-m4kxv"
Jul 27 22:19:58.304972 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:58.304817    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/59790390-7bdf-4b6f-9ef8-a9f572475caf-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-m4kxv\" (UID: \"59790390-7bdf-4b6f-9ef8-a9f572475caf\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-m4kxv"
Jul 27 22:19:58.304972 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:58.304857    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/59790390-7bdf-4b6f-9ef8-a9f572475caf-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-m4kxv\" (UID: \"59790390-7bdf-4b6f-9ef8-a9f572475caf\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-m4kxv"
Jul 27 22:19:58.304972 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:58.304872    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59790390-7bdf-4b6f-9ef8-a9f572475caf-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-m4kxv\" (UID: \"59790390-7bdf-4b6f-9ef8-a9f572475caf\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-m4kxv"
Jul 27 22:19:58.304972 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:58.304896    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59790390-7bdf-4b6f-9ef8-a9f572475caf-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-m4kxv\" (UID: \"59790390-7bdf-4b6f-9ef8-a9f572475caf\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-m4kxv"
Jul 27 22:19:58.305276 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:58.305259    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59790390-7bdf-4b6f-9ef8-a9f572475caf-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-m4kxv\" (UID: \"59790390-7bdf-4b6f-9ef8-a9f572475caf\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-m4kxv"
Jul 27 22:19:58.305564 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:58.305544    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/59790390-7bdf-4b6f-9ef8-a9f572475caf-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-m4kxv\" (UID: \"59790390-7bdf-4b6f-9ef8-a9f572475caf\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-m4kxv"
Jul 27 22:19:58.305600 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:58.305544    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/59790390-7bdf-4b6f-9ef8-a9f572475caf-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-m4kxv\" (UID: \"59790390-7bdf-4b6f-9ef8-a9f572475caf\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-m4kxv"
Jul 27 22:19:58.307289 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:58.307262    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59790390-7bdf-4b6f-9ef8-a9f572475caf-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-m4kxv\" (UID: \"59790390-7bdf-4b6f-9ef8-a9f572475caf\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-m4kxv"
Jul 27 22:19:58.313378 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:58.313357    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wr4q\" (UniqueName: \"kubernetes.io/projected/59790390-7bdf-4b6f-9ef8-a9f572475caf-kube-api-access-2wr4q\") pod \"isvc-tensorflow-predictor-547dfc5669-m4kxv\" (UID: \"59790390-7bdf-4b6f-9ef8-a9f572475caf\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-m4kxv"
Jul 27 22:19:58.319528 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:58.319507    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:19:58.349607 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:58.349583    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-dntff" event={"ID":"76fb6faa-350a-413f-b335-0dd449c1eb6f","Type":"ContainerDied","Data":"da6bf722f6f314901e4b3a538ad830a646ca83e39be88885089fee9f3626e15c"}
Jul 27 22:19:58.349982 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:58.349621    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-dntff"
Jul 27 22:19:58.349982 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:58.349653    2577 scope.go:117] "RemoveContainer" containerID="347a929ecc9c91527f25fa0b5eeb01668066f196355bc071685e06ad7e16a8f3"
Jul 27 22:19:58.358609 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:58.358344    2577 scope.go:117] "RemoveContainer" containerID="e019cb2c518e435b9af88892c12cbe800da2240156c8e6d17f1d610aa1970706"
Jul 27 22:19:58.368402 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:58.368384    2577 scope.go:117] "RemoveContainer" containerID="d6c4a4036e0d430717cf0da150da03963dfb2db9f9b0c298e701d236d6acddd1"
Jul 27 22:19:58.373071 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:58.373048    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-dntff"]
Jul 27 22:19:58.382034 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:58.382013    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-dntff"]
Jul 27 22:19:58.483121 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:58.483071    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-m4kxv"
Jul 27 22:19:58.608932 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:58.608909    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-m4kxv"]
Jul 27 22:19:58.611057 ip-10-0-142-219 kubenswrapper[2577]: W0727 22:19:58.611032    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59790390_7bdf_4b6f_9ef8_a9f572475caf.slice/crio-800cfdc097d0e33c371d275ac0b7e2af0d81c2aa3e94b5209a8004d71e4c581b WatchSource:0}: Error finding container 800cfdc097d0e33c371d275ac0b7e2af0d81c2aa3e94b5209a8004d71e4c581b: Status 404 returned error can't find the container with id 800cfdc097d0e33c371d275ac0b7e2af0d81c2aa3e94b5209a8004d71e4c581b
Jul 27 22:19:59.135457 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:59.135428    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76fb6faa-350a-413f-b335-0dd449c1eb6f" path="/var/lib/kubelet/pods/76fb6faa-350a-413f-b335-0dd449c1eb6f/volumes"
Jul 27 22:19:59.355232 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:59.355191    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-m4kxv" event={"ID":"59790390-7bdf-4b6f-9ef8-a9f572475caf","Type":"ContainerStarted","Data":"06dd0ab4434758daae8b990859f5c41039afcb1fab749a72843f3c02528513b3"}
Jul 27 22:19:59.355725 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:19:59.355266    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-m4kxv" event={"ID":"59790390-7bdf-4b6f-9ef8-a9f572475caf","Type":"ContainerStarted","Data":"800cfdc097d0e33c371d275ac0b7e2af0d81c2aa3e94b5209a8004d71e4c581b"}
Jul 27 22:20:03.371574 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:03.371541    2577 generic.go:356] "Generic (PLEG): container finished" podID="59790390-7bdf-4b6f-9ef8-a9f572475caf" containerID="06dd0ab4434758daae8b990859f5c41039afcb1fab749a72843f3c02528513b3" exitCode=0
Jul 27 22:20:03.371968 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:03.371608    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-m4kxv" event={"ID":"59790390-7bdf-4b6f-9ef8-a9f572475caf","Type":"ContainerDied","Data":"06dd0ab4434758daae8b990859f5c41039afcb1fab749a72843f3c02528513b3"}
Jul 27 22:20:03.372785 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:03.372751    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 22:20:08.397254 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:08.397220    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-m4kxv" event={"ID":"59790390-7bdf-4b6f-9ef8-a9f572475caf","Type":"ContainerStarted","Data":"12417044571cf05840bcc0c11d32a7ac4bd6ca13e6fa02ff1a7a822a0b3100ef"}
Jul 27 22:20:08.397667 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:08.397277    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-m4kxv" event={"ID":"59790390-7bdf-4b6f-9ef8-a9f572475caf","Type":"ContainerStarted","Data":"068a7d9565086805f6f99fbb009971659e5e4621ed6f03c30dd54a7ad75e1e7b"}
Jul 27 22:20:08.397667 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:08.397549    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-m4kxv"
Jul 27 22:20:08.397748 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:08.397681    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-m4kxv"
Jul 27 22:20:08.398961 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:08.398933    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-m4kxv" podUID="59790390-7bdf-4b6f-9ef8-a9f572475caf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.64:8080: connect: connection refused"
Jul 27 22:20:08.417657 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:08.417606    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-m4kxv" podStartSLOduration=1.782983083 podStartE2EDuration="10.417593524s" totalImagesPullingTime="4.154367851s" totalInitContainerRuntime="4.48024259s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:19:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:20:08.415938744 +0000 UTC m=+4531.783128611" watchObservedRunningTime="2026-07-27 22:20:08.417593524 +0000 UTC m=+4531.784783389"
Jul 27 22:20:09.401750 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:09.401711    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-m4kxv" podUID="59790390-7bdf-4b6f-9ef8-a9f572475caf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.64:8080: connect: connection refused"
Jul 27 22:20:14.406775 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:14.406731    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-m4kxv"
Jul 27 22:20:14.407311 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:14.407284    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-m4kxv" podUID="59790390-7bdf-4b6f-9ef8-a9f572475caf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.64:8080: connect: connection refused"
Jul 27 22:20:24.407887 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:24.407857    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-m4kxv"
Jul 27 22:20:38.896461 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:38.896424    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-m4kxv"]
Jul 27 22:20:38.896970 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:38.896777    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-m4kxv" podUID="59790390-7bdf-4b6f-9ef8-a9f572475caf" containerName="kserve-container" containerID="cri-o://068a7d9565086805f6f99fbb009971659e5e4621ed6f03c30dd54a7ad75e1e7b" gracePeriod=30
Jul 27 22:20:38.896970 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:38.896808    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-m4kxv" podUID="59790390-7bdf-4b6f-9ef8-a9f572475caf" containerName="kube-rbac-proxy" containerID="cri-o://12417044571cf05840bcc0c11d32a7ac4bd6ca13e6fa02ff1a7a822a0b3100ef" gracePeriod=30
Jul 27 22:20:38.918185 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:38.918157    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:20:39.401950 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:39.401913    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-m4kxv" podUID="59790390-7bdf-4b6f-9ef8-a9f572475caf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.64:8643/healthz\": dial tcp 10.133.0.64:8643: connect: connection refused"
Jul 27 22:20:39.530574 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:39.530538    2577 generic.go:356] "Generic (PLEG): container finished" podID="59790390-7bdf-4b6f-9ef8-a9f572475caf" containerID="12417044571cf05840bcc0c11d32a7ac4bd6ca13e6fa02ff1a7a822a0b3100ef" exitCode=2
Jul 27 22:20:39.530736 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:39.530604    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-m4kxv" event={"ID":"59790390-7bdf-4b6f-9ef8-a9f572475caf","Type":"ContainerDied","Data":"12417044571cf05840bcc0c11d32a7ac4bd6ca13e6fa02ff1a7a822a0b3100ef"}
Jul 27 22:20:44.006582 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:44.006548    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:20:44.045825 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:44.045792    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5"]
Jul 27 22:20:44.049154 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:44.049136    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5"
Jul 27 22:20:44.052059 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:44.052040    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
Jul 27 22:20:44.052185 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:44.052046    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
Jul 27 22:20:44.060672 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:44.060652    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5"]
Jul 27 22:20:44.191522 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:44.191490    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37680fa4-9f86-4647-8ada-43e9700b3f30-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5\" (UID: \"37680fa4-9f86-4647-8ada-43e9700b3f30\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5"
Jul 27 22:20:44.191694 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:44.191539    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37680fa4-9f86-4647-8ada-43e9700b3f30-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5\" (UID: \"37680fa4-9f86-4647-8ada-43e9700b3f30\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5"
Jul 27 22:20:44.191694 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:44.191592    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37680fa4-9f86-4647-8ada-43e9700b3f30-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5\" (UID: \"37680fa4-9f86-4647-8ada-43e9700b3f30\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5"
Jul 27 22:20:44.191694 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:44.191634    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37680fa4-9f86-4647-8ada-43e9700b3f30-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5\" (UID: \"37680fa4-9f86-4647-8ada-43e9700b3f30\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5"
Jul 27 22:20:44.191868 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:44.191718    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fskkn\" (UniqueName: \"kubernetes.io/projected/37680fa4-9f86-4647-8ada-43e9700b3f30-kube-api-access-fskkn\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5\" (UID: \"37680fa4-9f86-4647-8ada-43e9700b3f30\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5"
Jul 27 22:20:44.292964 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:44.292873    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37680fa4-9f86-4647-8ada-43e9700b3f30-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5\" (UID: \"37680fa4-9f86-4647-8ada-43e9700b3f30\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5"
Jul 27 22:20:44.292964 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:44.292939    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37680fa4-9f86-4647-8ada-43e9700b3f30-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5\" (UID: \"37680fa4-9f86-4647-8ada-43e9700b3f30\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5"
Jul 27 22:20:44.293194 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:44.292980    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37680fa4-9f86-4647-8ada-43e9700b3f30-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5\" (UID: \"37680fa4-9f86-4647-8ada-43e9700b3f30\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5"
Jul 27 22:20:44.293194 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:20:44.293048    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-serving-cert: secret "isvc-tensorflow-runtime-predictor-serving-cert" not found
Jul 27 22:20:44.293194 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:44.293058    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fskkn\" (UniqueName: \"kubernetes.io/projected/37680fa4-9f86-4647-8ada-43e9700b3f30-kube-api-access-fskkn\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5\" (UID: \"37680fa4-9f86-4647-8ada-43e9700b3f30\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5"
Jul 27 22:20:44.293194 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:20:44.293122    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37680fa4-9f86-4647-8ada-43e9700b3f30-proxy-tls podName:37680fa4-9f86-4647-8ada-43e9700b3f30 nodeName:}" failed. No retries permitted until 2026-07-27 22:20:44.79309789 +0000 UTC m=+4568.160287740 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/37680fa4-9f86-4647-8ada-43e9700b3f30-proxy-tls") pod "isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5" (UID: "37680fa4-9f86-4647-8ada-43e9700b3f30") : secret "isvc-tensorflow-runtime-predictor-serving-cert" not found
Jul 27 22:20:44.293194 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:44.293175    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37680fa4-9f86-4647-8ada-43e9700b3f30-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5\" (UID: \"37680fa4-9f86-4647-8ada-43e9700b3f30\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5"
Jul 27 22:20:44.293978 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:44.293926    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37680fa4-9f86-4647-8ada-43e9700b3f30-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5\" (UID: \"37680fa4-9f86-4647-8ada-43e9700b3f30\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5"
Jul 27 22:20:44.294112 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:44.293992    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37680fa4-9f86-4647-8ada-43e9700b3f30-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5\" (UID: \"37680fa4-9f86-4647-8ada-43e9700b3f30\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5"
Jul 27 22:20:44.294112 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:44.294041    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37680fa4-9f86-4647-8ada-43e9700b3f30-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5\" (UID: \"37680fa4-9f86-4647-8ada-43e9700b3f30\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5"
Jul 27 22:20:44.303706 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:44.303687    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fskkn\" (UniqueName: \"kubernetes.io/projected/37680fa4-9f86-4647-8ada-43e9700b3f30-kube-api-access-fskkn\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5\" (UID: \"37680fa4-9f86-4647-8ada-43e9700b3f30\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5"
Jul 27 22:20:44.402881 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:44.402825    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-m4kxv" podUID="59790390-7bdf-4b6f-9ef8-a9f572475caf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.64:8643/healthz\": dial tcp 10.133.0.64:8643: connect: connection refused"
Jul 27 22:20:44.492362 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:44.492326    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:20:44.797551 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:44.797519    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37680fa4-9f86-4647-8ada-43e9700b3f30-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5\" (UID: \"37680fa4-9f86-4647-8ada-43e9700b3f30\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5"
Jul 27 22:20:44.799961 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:44.799932    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37680fa4-9f86-4647-8ada-43e9700b3f30-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5\" (UID: \"37680fa4-9f86-4647-8ada-43e9700b3f30\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5"
Jul 27 22:20:44.959438 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:44.959404    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5"
Jul 27 22:20:45.086068 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:45.086017    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5"]
Jul 27 22:20:45.088363 ip-10-0-142-219 kubenswrapper[2577]: W0727 22:20:45.088338    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37680fa4_9f86_4647_8ada_43e9700b3f30.slice/crio-80dae239741bac5a22fdb374b2ed3997a78beb68d080fc818d93d65d2bec3cc8 WatchSource:0}: Error finding container 80dae239741bac5a22fdb374b2ed3997a78beb68d080fc818d93d65d2bec3cc8: Status 404 returned error can't find the container with id 80dae239741bac5a22fdb374b2ed3997a78beb68d080fc818d93d65d2bec3cc8
Jul 27 22:20:45.556211 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:45.556178    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5" event={"ID":"37680fa4-9f86-4647-8ada-43e9700b3f30","Type":"ContainerStarted","Data":"6c0a1b1909472a38b87520552549f16cecac084344bf86aedf78573539dcb4fb"}
Jul 27 22:20:45.556381 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:45.556236    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5" event={"ID":"37680fa4-9f86-4647-8ada-43e9700b3f30","Type":"ContainerStarted","Data":"80dae239741bac5a22fdb374b2ed3997a78beb68d080fc818d93d65d2bec3cc8"}
Jul 27 22:20:49.402007 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:49.401955    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-m4kxv" podUID="59790390-7bdf-4b6f-9ef8-a9f572475caf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.64:8643/healthz\": dial tcp 10.133.0.64:8643: connect: connection refused"
Jul 27 22:20:49.404774 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:49.402065    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-m4kxv"
Jul 27 22:20:50.577823 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:50.577782    2577 generic.go:356] "Generic (PLEG): container finished" podID="37680fa4-9f86-4647-8ada-43e9700b3f30" containerID="6c0a1b1909472a38b87520552549f16cecac084344bf86aedf78573539dcb4fb" exitCode=0
Jul 27 22:20:50.577823 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:50.577821    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5" event={"ID":"37680fa4-9f86-4647-8ada-43e9700b3f30","Type":"ContainerDied","Data":"6c0a1b1909472a38b87520552549f16cecac084344bf86aedf78573539dcb4fb"}
Jul 27 22:20:51.583529 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:51.583497    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5" event={"ID":"37680fa4-9f86-4647-8ada-43e9700b3f30","Type":"ContainerStarted","Data":"e17559b3a25155e429b64cd13873b0014df02d982fbef83846e0530d60d9b674"}
Jul 27 22:20:51.583904 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:51.583554    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5" event={"ID":"37680fa4-9f86-4647-8ada-43e9700b3f30","Type":"ContainerStarted","Data":"dcbbe247273ab9116b3945be6e796a603f8a143a3de77c58fa32f176ddaa0fca"}
Jul 27 22:20:51.583904 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:51.583798    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5"
Jul 27 22:20:51.602661 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:51.602618    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5" podStartSLOduration=3.119975806 podStartE2EDuration="7.602604573s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.482628767s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:20:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:20:51.601738562 +0000 UTC m=+4574.968928511" watchObservedRunningTime="2026-07-27 22:20:51.602604573 +0000 UTC m=+4574.969794439"
Jul 27 22:20:52.588078 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:52.588047    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5"
Jul 27 22:20:52.589135 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:52.589107    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5" podUID="37680fa4-9f86-4647-8ada-43e9700b3f30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.65:8080: connect: connection refused"
Jul 27 22:20:53.597128 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:53.597084    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5" podUID="37680fa4-9f86-4647-8ada-43e9700b3f30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.65:8080: connect: connection refused"
Jul 27 22:20:54.402066 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:54.402021    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-m4kxv" podUID="59790390-7bdf-4b6f-9ef8-a9f572475caf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.64:8643/healthz\": dial tcp 10.133.0.64:8643: connect: connection refused"
Jul 27 22:20:58.597784 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:58.597680    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5"
Jul 27 22:20:58.598235 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:58.598210    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5" podUID="37680fa4-9f86-4647-8ada-43e9700b3f30" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.65:8080: connect: connection refused"
Jul 27 22:20:59.402678 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:20:59.402639    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-m4kxv" podUID="59790390-7bdf-4b6f-9ef8-a9f572475caf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.64:8643/healthz\": dial tcp 10.133.0.64:8643: connect: connection refused"
Jul 27 22:21:04.402543 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:04.402499    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-m4kxv" podUID="59790390-7bdf-4b6f-9ef8-a9f572475caf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.64:8643/healthz\": dial tcp 10.133.0.64:8643: connect: connection refused"
Jul 27 22:21:08.602714 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:08.598714    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5"
Jul 27 22:21:09.402245 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:09.402203    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-m4kxv" podUID="59790390-7bdf-4b6f-9ef8-a9f572475caf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.64:8643/healthz\": dial tcp 10.133.0.64:8643: connect: connection refused"
Jul 27 22:21:09.540726 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:09.540702    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-m4kxv"
Jul 27 22:21:09.596643 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:09.596614    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/59790390-7bdf-4b6f-9ef8-a9f572475caf-kube-rbac-proxy-sar-config\") pod \"59790390-7bdf-4b6f-9ef8-a9f572475caf\" (UID: \"59790390-7bdf-4b6f-9ef8-a9f572475caf\") "
Jul 27 22:21:09.596805 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:09.596657    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59790390-7bdf-4b6f-9ef8-a9f572475caf-proxy-tls\") pod \"59790390-7bdf-4b6f-9ef8-a9f572475caf\" (UID: \"59790390-7bdf-4b6f-9ef8-a9f572475caf\") "
Jul 27 22:21:09.596805 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:09.596716    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/59790390-7bdf-4b6f-9ef8-a9f572475caf-cabundle-cert\") pod \"59790390-7bdf-4b6f-9ef8-a9f572475caf\" (UID: \"59790390-7bdf-4b6f-9ef8-a9f572475caf\") "
Jul 27 22:21:09.596805 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:09.596783    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59790390-7bdf-4b6f-9ef8-a9f572475caf-kserve-provision-location\") pod \"59790390-7bdf-4b6f-9ef8-a9f572475caf\" (UID: \"59790390-7bdf-4b6f-9ef8-a9f572475caf\") "
Jul 27 22:21:09.597034 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:09.596824    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2wr4q\" (UniqueName: \"kubernetes.io/projected/59790390-7bdf-4b6f-9ef8-a9f572475caf-kube-api-access-2wr4q\") pod \"59790390-7bdf-4b6f-9ef8-a9f572475caf\" (UID: \"59790390-7bdf-4b6f-9ef8-a9f572475caf\") "
Jul 27 22:21:09.597034 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:09.596909    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59790390-7bdf-4b6f-9ef8-a9f572475caf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "59790390-7bdf-4b6f-9ef8-a9f572475caf" (UID: "59790390-7bdf-4b6f-9ef8-a9f572475caf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:21:09.597125 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:09.597056    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/59790390-7bdf-4b6f-9ef8-a9f572475caf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:21:09.597167 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:09.597144    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59790390-7bdf-4b6f-9ef8-a9f572475caf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "59790390-7bdf-4b6f-9ef8-a9f572475caf" (UID: "59790390-7bdf-4b6f-9ef8-a9f572475caf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:21:09.598831 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:09.598810    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59790390-7bdf-4b6f-9ef8-a9f572475caf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "59790390-7bdf-4b6f-9ef8-a9f572475caf" (UID: "59790390-7bdf-4b6f-9ef8-a9f572475caf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 22:21:09.598922 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:09.598810    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59790390-7bdf-4b6f-9ef8-a9f572475caf-kube-api-access-2wr4q" (OuterVolumeSpecName: "kube-api-access-2wr4q") pod "59790390-7bdf-4b6f-9ef8-a9f572475caf" (UID: "59790390-7bdf-4b6f-9ef8-a9f572475caf"). InnerVolumeSpecName "kube-api-access-2wr4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:21:09.607370 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:09.607348    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59790390-7bdf-4b6f-9ef8-a9f572475caf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "59790390-7bdf-4b6f-9ef8-a9f572475caf" (UID: "59790390-7bdf-4b6f-9ef8-a9f572475caf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:21:09.657259 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:09.657198    2577 generic.go:356] "Generic (PLEG): container finished" podID="59790390-7bdf-4b6f-9ef8-a9f572475caf" containerID="068a7d9565086805f6f99fbb009971659e5e4621ed6f03c30dd54a7ad75e1e7b" exitCode=137
Jul 27 22:21:09.657259 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:09.657252    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-m4kxv"
Jul 27 22:21:09.657259 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:09.657255    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-m4kxv" event={"ID":"59790390-7bdf-4b6f-9ef8-a9f572475caf","Type":"ContainerDied","Data":"068a7d9565086805f6f99fbb009971659e5e4621ed6f03c30dd54a7ad75e1e7b"}
Jul 27 22:21:09.657412 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:09.657301    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-m4kxv" event={"ID":"59790390-7bdf-4b6f-9ef8-a9f572475caf","Type":"ContainerDied","Data":"800cfdc097d0e33c371d275ac0b7e2af0d81c2aa3e94b5209a8004d71e4c581b"}
Jul 27 22:21:09.657412 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:09.657329    2577 scope.go:117] "RemoveContainer" containerID="12417044571cf05840bcc0c11d32a7ac4bd6ca13e6fa02ff1a7a822a0b3100ef"
Jul 27 22:21:09.665416 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:09.665399    2577 scope.go:117] "RemoveContainer" containerID="068a7d9565086805f6f99fbb009971659e5e4621ed6f03c30dd54a7ad75e1e7b"
Jul 27 22:21:09.672286 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:09.672270    2577 scope.go:117] "RemoveContainer" containerID="06dd0ab4434758daae8b990859f5c41039afcb1fab749a72843f3c02528513b3"
Jul 27 22:21:09.679126 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:09.679109    2577 scope.go:117] "RemoveContainer" containerID="12417044571cf05840bcc0c11d32a7ac4bd6ca13e6fa02ff1a7a822a0b3100ef"
Jul 27 22:21:09.679352 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:09.679334    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-m4kxv"]
Jul 27 22:21:09.679406 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:21:09.679363    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12417044571cf05840bcc0c11d32a7ac4bd6ca13e6fa02ff1a7a822a0b3100ef\": container with ID starting with 12417044571cf05840bcc0c11d32a7ac4bd6ca13e6fa02ff1a7a822a0b3100ef not found: ID does not exist" containerID="12417044571cf05840bcc0c11d32a7ac4bd6ca13e6fa02ff1a7a822a0b3100ef"
Jul 27 22:21:09.679406 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:09.679386    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12417044571cf05840bcc0c11d32a7ac4bd6ca13e6fa02ff1a7a822a0b3100ef"} err="failed to get container status \"12417044571cf05840bcc0c11d32a7ac4bd6ca13e6fa02ff1a7a822a0b3100ef\": rpc error: code = NotFound desc = could not find container \"12417044571cf05840bcc0c11d32a7ac4bd6ca13e6fa02ff1a7a822a0b3100ef\": container with ID starting with 12417044571cf05840bcc0c11d32a7ac4bd6ca13e6fa02ff1a7a822a0b3100ef not found: ID does not exist"
Jul 27 22:21:09.679406 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:09.679402    2577 scope.go:117] "RemoveContainer" containerID="068a7d9565086805f6f99fbb009971659e5e4621ed6f03c30dd54a7ad75e1e7b"
Jul 27 22:21:09.679625 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:21:09.679608    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"068a7d9565086805f6f99fbb009971659e5e4621ed6f03c30dd54a7ad75e1e7b\": container with ID starting with 068a7d9565086805f6f99fbb009971659e5e4621ed6f03c30dd54a7ad75e1e7b not found: ID does not exist" containerID="068a7d9565086805f6f99fbb009971659e5e4621ed6f03c30dd54a7ad75e1e7b"
Jul 27 22:21:09.679665 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:09.679630    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"068a7d9565086805f6f99fbb009971659e5e4621ed6f03c30dd54a7ad75e1e7b"} err="failed to get container status \"068a7d9565086805f6f99fbb009971659e5e4621ed6f03c30dd54a7ad75e1e7b\": rpc error: code = NotFound desc = could not find container \"068a7d9565086805f6f99fbb009971659e5e4621ed6f03c30dd54a7ad75e1e7b\": container with ID starting with 068a7d9565086805f6f99fbb009971659e5e4621ed6f03c30dd54a7ad75e1e7b not found: ID does not exist"
Jul 27 22:21:09.679665 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:09.679646    2577 scope.go:117] "RemoveContainer" containerID="06dd0ab4434758daae8b990859f5c41039afcb1fab749a72843f3c02528513b3"
Jul 27 22:21:09.679894 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:21:09.679879    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06dd0ab4434758daae8b990859f5c41039afcb1fab749a72843f3c02528513b3\": container with ID starting with 06dd0ab4434758daae8b990859f5c41039afcb1fab749a72843f3c02528513b3 not found: ID does not exist" containerID="06dd0ab4434758daae8b990859f5c41039afcb1fab749a72843f3c02528513b3"
Jul 27 22:21:09.679945 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:09.679898    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06dd0ab4434758daae8b990859f5c41039afcb1fab749a72843f3c02528513b3"} err="failed to get container status \"06dd0ab4434758daae8b990859f5c41039afcb1fab749a72843f3c02528513b3\": rpc error: code = NotFound desc = could not find container \"06dd0ab4434758daae8b990859f5c41039afcb1fab749a72843f3c02528513b3\": container with ID starting with 06dd0ab4434758daae8b990859f5c41039afcb1fab749a72843f3c02528513b3 not found: ID does not exist"
Jul 27 22:21:09.684420 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:09.684403    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-m4kxv"]
Jul 27 22:21:09.698375 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:09.698360    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59790390-7bdf-4b6f-9ef8-a9f572475caf-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:21:09.698441 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:09.698378    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/59790390-7bdf-4b6f-9ef8-a9f572475caf-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:21:09.698441 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:09.698388    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59790390-7bdf-4b6f-9ef8-a9f572475caf-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:21:09.698441 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:09.698397    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wr4q\" (UniqueName: \"kubernetes.io/projected/59790390-7bdf-4b6f-9ef8-a9f572475caf-kube-api-access-2wr4q\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:21:11.135457 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:11.135421    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59790390-7bdf-4b6f-9ef8-a9f572475caf" path="/var/lib/kubelet/pods/59790390-7bdf-4b6f-9ef8-a9f572475caf/volumes"
Jul 27 22:21:24.965913 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:24.965883    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5"]
Jul 27 22:21:24.966415 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:24.966089    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5" podUID="37680fa4-9f86-4647-8ada-43e9700b3f30" containerName="kserve-container" containerID="cri-o://dcbbe247273ab9116b3945be6e796a603f8a143a3de77c58fa32f176ddaa0fca" gracePeriod=30
Jul 27 22:21:24.966415 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:24.966141    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5" podUID="37680fa4-9f86-4647-8ada-43e9700b3f30" containerName="kube-rbac-proxy" containerID="cri-o://e17559b3a25155e429b64cd13873b0014df02d982fbef83846e0530d60d9b674" gracePeriod=30
Jul 27 22:21:24.992468 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:24.992441    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:21:25.722211 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:25.722180    2577 generic.go:356] "Generic (PLEG): container finished" podID="37680fa4-9f86-4647-8ada-43e9700b3f30" containerID="e17559b3a25155e429b64cd13873b0014df02d982fbef83846e0530d60d9b674" exitCode=2
Jul 27 22:21:25.722385 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:25.722255    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5" event={"ID":"37680fa4-9f86-4647-8ada-43e9700b3f30","Type":"ContainerDied","Data":"e17559b3a25155e429b64cd13873b0014df02d982fbef83846e0530d60d9b674"}
Jul 27 22:21:28.594470 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:28.594427    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5" podUID="37680fa4-9f86-4647-8ada-43e9700b3f30" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.65:8643/healthz\": dial tcp 10.133.0.65:8643: connect: connection refused"
Jul 27 22:21:30.102547 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:30.102512    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:21:30.131468 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:30.131437    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bqwts"]
Jul 27 22:21:30.131817 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:30.131778    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59790390-7bdf-4b6f-9ef8-a9f572475caf" containerName="storage-initializer"
Jul 27 22:21:30.131817 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:30.131807    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="59790390-7bdf-4b6f-9ef8-a9f572475caf" containerName="storage-initializer"
Jul 27 22:21:30.131915 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:30.131819    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59790390-7bdf-4b6f-9ef8-a9f572475caf" containerName="kserve-container"
Jul 27 22:21:30.131915 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:30.131825    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="59790390-7bdf-4b6f-9ef8-a9f572475caf" containerName="kserve-container"
Jul 27 22:21:30.131915 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:30.131839    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59790390-7bdf-4b6f-9ef8-a9f572475caf" containerName="kube-rbac-proxy"
Jul 27 22:21:30.131915 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:30.131845    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="59790390-7bdf-4b6f-9ef8-a9f572475caf" containerName="kube-rbac-proxy"
Jul 27 22:21:30.131915 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:30.131892    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="59790390-7bdf-4b6f-9ef8-a9f572475caf" containerName="kube-rbac-proxy"
Jul 27 22:21:30.131915 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:30.131901    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="59790390-7bdf-4b6f-9ef8-a9f572475caf" containerName="kserve-container"
Jul 27 22:21:30.134977 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:30.134963    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bqwts"
Jul 27 22:21:30.137209 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:30.137176    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-predictor-serving-cert\""
Jul 27 22:21:30.137347 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:30.137248    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-kube-rbac-proxy-sar-config\""
Jul 27 22:21:30.145618 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:30.145587    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bqwts"]
Jul 27 22:21:30.250360 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:30.250329    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:21:30.261517 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:30.261490    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7bb09e6-fd46-4000-b024-5e4ac6dd94b1-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-bqwts\" (UID: \"a7bb09e6-fd46-4000-b024-5e4ac6dd94b1\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bqwts"
Jul 27 22:21:30.261626 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:30.261526    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7bb09e6-fd46-4000-b024-5e4ac6dd94b1-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-bqwts\" (UID: \"a7bb09e6-fd46-4000-b024-5e4ac6dd94b1\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bqwts"
Jul 27 22:21:30.261626 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:30.261548    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5zlw\" (UniqueName: \"kubernetes.io/projected/a7bb09e6-fd46-4000-b024-5e4ac6dd94b1-kube-api-access-h5zlw\") pod \"isvc-triton-predictor-5d7db58676-bqwts\" (UID: \"a7bb09e6-fd46-4000-b024-5e4ac6dd94b1\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bqwts"
Jul 27 22:21:30.261758 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:30.261647    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7bb09e6-fd46-4000-b024-5e4ac6dd94b1-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-bqwts\" (UID: \"a7bb09e6-fd46-4000-b024-5e4ac6dd94b1\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bqwts"
Jul 27 22:21:30.261758 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:30.261728    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7bb09e6-fd46-4000-b024-5e4ac6dd94b1-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-bqwts\" (UID: \"a7bb09e6-fd46-4000-b024-5e4ac6dd94b1\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bqwts"
Jul 27 22:21:30.362758 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:30.362663    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7bb09e6-fd46-4000-b024-5e4ac6dd94b1-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-bqwts\" (UID: \"a7bb09e6-fd46-4000-b024-5e4ac6dd94b1\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bqwts"
Jul 27 22:21:30.362758 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:30.362705    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7bb09e6-fd46-4000-b024-5e4ac6dd94b1-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-bqwts\" (UID: \"a7bb09e6-fd46-4000-b024-5e4ac6dd94b1\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bqwts"
Jul 27 22:21:30.362758 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:30.362732    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5zlw\" (UniqueName: \"kubernetes.io/projected/a7bb09e6-fd46-4000-b024-5e4ac6dd94b1-kube-api-access-h5zlw\") pod \"isvc-triton-predictor-5d7db58676-bqwts\" (UID: \"a7bb09e6-fd46-4000-b024-5e4ac6dd94b1\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bqwts"
Jul 27 22:21:30.363093 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:30.362796    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7bb09e6-fd46-4000-b024-5e4ac6dd94b1-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-bqwts\" (UID: \"a7bb09e6-fd46-4000-b024-5e4ac6dd94b1\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bqwts"
Jul 27 22:21:30.363093 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:30.362854    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7bb09e6-fd46-4000-b024-5e4ac6dd94b1-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-bqwts\" (UID: \"a7bb09e6-fd46-4000-b024-5e4ac6dd94b1\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bqwts"
Jul 27 22:21:30.363270 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:30.363247    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7bb09e6-fd46-4000-b024-5e4ac6dd94b1-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-bqwts\" (UID: \"a7bb09e6-fd46-4000-b024-5e4ac6dd94b1\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bqwts"
Jul 27 22:21:30.363502 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:30.363480    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7bb09e6-fd46-4000-b024-5e4ac6dd94b1-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-bqwts\" (UID: \"a7bb09e6-fd46-4000-b024-5e4ac6dd94b1\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bqwts"
Jul 27 22:21:30.363569 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:30.363488    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7bb09e6-fd46-4000-b024-5e4ac6dd94b1-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-bqwts\" (UID: \"a7bb09e6-fd46-4000-b024-5e4ac6dd94b1\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bqwts"
Jul 27 22:21:30.365317 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:30.365299    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7bb09e6-fd46-4000-b024-5e4ac6dd94b1-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-bqwts\" (UID: \"a7bb09e6-fd46-4000-b024-5e4ac6dd94b1\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bqwts"
Jul 27 22:21:30.370806 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:30.370786    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5zlw\" (UniqueName: \"kubernetes.io/projected/a7bb09e6-fd46-4000-b024-5e4ac6dd94b1-kube-api-access-h5zlw\") pod \"isvc-triton-predictor-5d7db58676-bqwts\" (UID: \"a7bb09e6-fd46-4000-b024-5e4ac6dd94b1\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bqwts"
Jul 27 22:21:30.445014 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:30.444979    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bqwts"
Jul 27 22:21:30.577911 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:30.577827    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bqwts"]
Jul 27 22:21:30.580115 ip-10-0-142-219 kubenswrapper[2577]: W0727 22:21:30.580087    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7bb09e6_fd46_4000_b024_5e4ac6dd94b1.slice/crio-34c92ca79181b9d491f1c9ff51f77de784e9e19fadab49f309a61c0c039b0701 WatchSource:0}: Error finding container 34c92ca79181b9d491f1c9ff51f77de784e9e19fadab49f309a61c0c039b0701: Status 404 returned error can't find the container with id 34c92ca79181b9d491f1c9ff51f77de784e9e19fadab49f309a61c0c039b0701
Jul 27 22:21:30.748780 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:30.748733    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bqwts" event={"ID":"a7bb09e6-fd46-4000-b024-5e4ac6dd94b1","Type":"ContainerStarted","Data":"48a82015915a5f77b47a9658a721ca31852fdd5f7610209cb91d18362fc8ef8b"}
Jul 27 22:21:30.748963 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:30.748814    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bqwts" event={"ID":"a7bb09e6-fd46-4000-b024-5e4ac6dd94b1","Type":"ContainerStarted","Data":"34c92ca79181b9d491f1c9ff51f77de784e9e19fadab49f309a61c0c039b0701"}
Jul 27 22:21:33.594449 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:33.594405    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5" podUID="37680fa4-9f86-4647-8ada-43e9700b3f30" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.65:8643/healthz\": dial tcp 10.133.0.65:8643: connect: connection refused"
Jul 27 22:21:34.765681 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:34.765586    2577 generic.go:356] "Generic (PLEG): container finished" podID="a7bb09e6-fd46-4000-b024-5e4ac6dd94b1" containerID="48a82015915a5f77b47a9658a721ca31852fdd5f7610209cb91d18362fc8ef8b" exitCode=0
Jul 27 22:21:34.766168 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:34.765674    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bqwts" event={"ID":"a7bb09e6-fd46-4000-b024-5e4ac6dd94b1","Type":"ContainerDied","Data":"48a82015915a5f77b47a9658a721ca31852fdd5f7610209cb91d18362fc8ef8b"}
Jul 27 22:21:38.594811 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:38.594256    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5" podUID="37680fa4-9f86-4647-8ada-43e9700b3f30" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.65:8643/healthz\": dial tcp 10.133.0.65:8643: connect: connection refused"
Jul 27 22:21:38.594811 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:38.594412    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5"
Jul 27 22:21:43.594796 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:43.594506    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5" podUID="37680fa4-9f86-4647-8ada-43e9700b3f30" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.65:8643/healthz\": dial tcp 10.133.0.65:8643: connect: connection refused"
Jul 27 22:21:48.593672 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:48.593607    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5" podUID="37680fa4-9f86-4647-8ada-43e9700b3f30" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.65:8643/healthz\": dial tcp 10.133.0.65:8643: connect: connection refused"
Jul 27 22:21:53.594353 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:53.594308    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5" podUID="37680fa4-9f86-4647-8ada-43e9700b3f30" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.65:8643/healthz\": dial tcp 10.133.0.65:8643: connect: connection refused"
Jul 27 22:21:55.670330 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:55.670298    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5"
Jul 27 22:21:55.707004 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:55.706971    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37680fa4-9f86-4647-8ada-43e9700b3f30-proxy-tls\") pod \"37680fa4-9f86-4647-8ada-43e9700b3f30\" (UID: \"37680fa4-9f86-4647-8ada-43e9700b3f30\") "
Jul 27 22:21:55.707189 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:55.707034    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37680fa4-9f86-4647-8ada-43e9700b3f30-cabundle-cert\") pod \"37680fa4-9f86-4647-8ada-43e9700b3f30\" (UID: \"37680fa4-9f86-4647-8ada-43e9700b3f30\") "
Jul 27 22:21:55.707189 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:55.707084    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37680fa4-9f86-4647-8ada-43e9700b3f30-kserve-provision-location\") pod \"37680fa4-9f86-4647-8ada-43e9700b3f30\" (UID: \"37680fa4-9f86-4647-8ada-43e9700b3f30\") "
Jul 27 22:21:55.707189 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:55.707117    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37680fa4-9f86-4647-8ada-43e9700b3f30-kube-rbac-proxy-sar-config\") pod \"37680fa4-9f86-4647-8ada-43e9700b3f30\" (UID: \"37680fa4-9f86-4647-8ada-43e9700b3f30\") "
Jul 27 22:21:55.707369 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:55.707206    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fskkn\" (UniqueName: \"kubernetes.io/projected/37680fa4-9f86-4647-8ada-43e9700b3f30-kube-api-access-fskkn\") pod \"37680fa4-9f86-4647-8ada-43e9700b3f30\" (UID: \"37680fa4-9f86-4647-8ada-43e9700b3f30\") "
Jul 27 22:21:55.708040 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:55.707788    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37680fa4-9f86-4647-8ada-43e9700b3f30-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "37680fa4-9f86-4647-8ada-43e9700b3f30" (UID: "37680fa4-9f86-4647-8ada-43e9700b3f30"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:21:55.708383 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:55.708356    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37680fa4-9f86-4647-8ada-43e9700b3f30-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "37680fa4-9f86-4647-8ada-43e9700b3f30" (UID: "37680fa4-9f86-4647-8ada-43e9700b3f30"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:21:55.712050 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:55.712012    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37680fa4-9f86-4647-8ada-43e9700b3f30-kube-api-access-fskkn" (OuterVolumeSpecName: "kube-api-access-fskkn") pod "37680fa4-9f86-4647-8ada-43e9700b3f30" (UID: "37680fa4-9f86-4647-8ada-43e9700b3f30"). InnerVolumeSpecName "kube-api-access-fskkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:21:55.712501 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:55.712469    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37680fa4-9f86-4647-8ada-43e9700b3f30-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "37680fa4-9f86-4647-8ada-43e9700b3f30" (UID: "37680fa4-9f86-4647-8ada-43e9700b3f30"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 22:21:55.717473 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:55.717446    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37680fa4-9f86-4647-8ada-43e9700b3f30-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "37680fa4-9f86-4647-8ada-43e9700b3f30" (UID: "37680fa4-9f86-4647-8ada-43e9700b3f30"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:21:55.808365 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:55.808282    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37680fa4-9f86-4647-8ada-43e9700b3f30-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:21:55.808365 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:55.808321    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37680fa4-9f86-4647-8ada-43e9700b3f30-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:21:55.808365 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:55.808336    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fskkn\" (UniqueName: \"kubernetes.io/projected/37680fa4-9f86-4647-8ada-43e9700b3f30-kube-api-access-fskkn\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:21:55.808365 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:55.808349    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37680fa4-9f86-4647-8ada-43e9700b3f30-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:21:55.808365 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:55.808362    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37680fa4-9f86-4647-8ada-43e9700b3f30-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:21:55.901614 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:55.901494    2577 generic.go:356] "Generic (PLEG): container finished" podID="37680fa4-9f86-4647-8ada-43e9700b3f30" containerID="dcbbe247273ab9116b3945be6e796a603f8a143a3de77c58fa32f176ddaa0fca" exitCode=137
Jul 27 22:21:55.901614 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:55.901552    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5" event={"ID":"37680fa4-9f86-4647-8ada-43e9700b3f30","Type":"ContainerDied","Data":"dcbbe247273ab9116b3945be6e796a603f8a143a3de77c58fa32f176ddaa0fca"}
Jul 27 22:21:55.901614 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:55.901575    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5"
Jul 27 22:21:55.901614 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:55.901620    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5" event={"ID":"37680fa4-9f86-4647-8ada-43e9700b3f30","Type":"ContainerDied","Data":"80dae239741bac5a22fdb374b2ed3997a78beb68d080fc818d93d65d2bec3cc8"}
Jul 27 22:21:55.902029 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:55.901668    2577 scope.go:117] "RemoveContainer" containerID="e17559b3a25155e429b64cd13873b0014df02d982fbef83846e0530d60d9b674"
Jul 27 22:21:55.914604 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:55.914509    2577 scope.go:117] "RemoveContainer" containerID="dcbbe247273ab9116b3945be6e796a603f8a143a3de77c58fa32f176ddaa0fca"
Jul 27 22:21:55.925093 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:55.925003    2577 scope.go:117] "RemoveContainer" containerID="6c0a1b1909472a38b87520552549f16cecac084344bf86aedf78573539dcb4fb"
Jul 27 22:21:55.929218 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:55.929194    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5"]
Jul 27 22:21:55.932945 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:55.932897    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-c6dh5"]
Jul 27 22:21:55.935898 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:55.935858    2577 scope.go:117] "RemoveContainer" containerID="e17559b3a25155e429b64cd13873b0014df02d982fbef83846e0530d60d9b674"
Jul 27 22:21:55.936309 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:21:55.936256    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e17559b3a25155e429b64cd13873b0014df02d982fbef83846e0530d60d9b674\": container with ID starting with e17559b3a25155e429b64cd13873b0014df02d982fbef83846e0530d60d9b674 not found: ID does not exist" containerID="e17559b3a25155e429b64cd13873b0014df02d982fbef83846e0530d60d9b674"
Jul 27 22:21:55.936309 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:55.936292    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e17559b3a25155e429b64cd13873b0014df02d982fbef83846e0530d60d9b674"} err="failed to get container status \"e17559b3a25155e429b64cd13873b0014df02d982fbef83846e0530d60d9b674\": rpc error: code = NotFound desc = could not find container \"e17559b3a25155e429b64cd13873b0014df02d982fbef83846e0530d60d9b674\": container with ID starting with e17559b3a25155e429b64cd13873b0014df02d982fbef83846e0530d60d9b674 not found: ID does not exist"
Jul 27 22:21:55.936485 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:55.936314    2577 scope.go:117] "RemoveContainer" containerID="dcbbe247273ab9116b3945be6e796a603f8a143a3de77c58fa32f176ddaa0fca"
Jul 27 22:21:55.936628 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:21:55.936601    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcbbe247273ab9116b3945be6e796a603f8a143a3de77c58fa32f176ddaa0fca\": container with ID starting with dcbbe247273ab9116b3945be6e796a603f8a143a3de77c58fa32f176ddaa0fca not found: ID does not exist" containerID="dcbbe247273ab9116b3945be6e796a603f8a143a3de77c58fa32f176ddaa0fca"
Jul 27 22:21:55.936710 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:55.936636    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcbbe247273ab9116b3945be6e796a603f8a143a3de77c58fa32f176ddaa0fca"} err="failed to get container status \"dcbbe247273ab9116b3945be6e796a603f8a143a3de77c58fa32f176ddaa0fca\": rpc error: code = NotFound desc = could not find container \"dcbbe247273ab9116b3945be6e796a603f8a143a3de77c58fa32f176ddaa0fca\": container with ID starting with dcbbe247273ab9116b3945be6e796a603f8a143a3de77c58fa32f176ddaa0fca not found: ID does not exist"
Jul 27 22:21:55.936710 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:55.936656    2577 scope.go:117] "RemoveContainer" containerID="6c0a1b1909472a38b87520552549f16cecac084344bf86aedf78573539dcb4fb"
Jul 27 22:21:55.936959 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:21:55.936928    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c0a1b1909472a38b87520552549f16cecac084344bf86aedf78573539dcb4fb\": container with ID starting with 6c0a1b1909472a38b87520552549f16cecac084344bf86aedf78573539dcb4fb not found: ID does not exist" containerID="6c0a1b1909472a38b87520552549f16cecac084344bf86aedf78573539dcb4fb"
Jul 27 22:21:55.937042 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:55.936960    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c0a1b1909472a38b87520552549f16cecac084344bf86aedf78573539dcb4fb"} err="failed to get container status \"6c0a1b1909472a38b87520552549f16cecac084344bf86aedf78573539dcb4fb\": rpc error: code = NotFound desc = could not find container \"6c0a1b1909472a38b87520552549f16cecac084344bf86aedf78573539dcb4fb\": container with ID starting with 6c0a1b1909472a38b87520552549f16cecac084344bf86aedf78573539dcb4fb not found: ID does not exist"
Jul 27 22:21:57.137358 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:21:57.136902    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37680fa4-9f86-4647-8ada-43e9700b3f30" path="/var/lib/kubelet/pods/37680fa4-9f86-4647-8ada-43e9700b3f30/volumes"
Jul 27 22:23:16.360901 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:16.360828    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:23:17.404746 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:17.404683    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:23:29.324823 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:29.324783    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bqwts" event={"ID":"a7bb09e6-fd46-4000-b024-5e4ac6dd94b1","Type":"ContainerStarted","Data":"44e075b62a2c639ad174be93dc5d1c410819cc51f100aecfa8e63ec979efe1ac"}
Jul 27 22:23:29.325343 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:29.324860    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bqwts" event={"ID":"a7bb09e6-fd46-4000-b024-5e4ac6dd94b1","Type":"ContainerStarted","Data":"0c590b916dd02fbaf677003baa342413aa8763e229406a9591f9ac1b4cc2437f"}
Jul 27 22:23:29.325343 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:29.324898    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bqwts"
Jul 27 22:23:29.358637 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:29.358586    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bqwts" podStartSLOduration=1.397612466 podStartE2EDuration="1m59.358555819s" totalImagesPullingTime="1m54.105800029s" totalInitContainerRuntime="3.855143324s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:21:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:23:29.357557255 +0000 UTC m=+4732.724747120" watchObservedRunningTime="2026-07-27 22:23:29.358555819 +0000 UTC m=+4732.725745667"
Jul 27 22:23:30.329400 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:30.329368    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bqwts"
Jul 27 22:23:30.330542 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:30.330517    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bqwts" podUID="a7bb09e6-fd46-4000-b024-5e4ac6dd94b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Jul 27 22:23:31.333284 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:31.333243    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bqwts" podUID="a7bb09e6-fd46-4000-b024-5e4ac6dd94b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Jul 27 22:23:36.337950 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:36.337921    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bqwts"
Jul 27 22:23:36.338744 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:36.338715    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bqwts"
Jul 27 22:23:41.592750 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:41.592717    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bqwts"]
Jul 27 22:23:41.593156 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:41.592953    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bqwts" podUID="a7bb09e6-fd46-4000-b024-5e4ac6dd94b1" containerName="kserve-container" containerID="cri-o://0c590b916dd02fbaf677003baa342413aa8763e229406a9591f9ac1b4cc2437f" gracePeriod=30
Jul 27 22:23:41.593156 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:41.593017    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bqwts" podUID="a7bb09e6-fd46-4000-b024-5e4ac6dd94b1" containerName="kube-rbac-proxy" containerID="cri-o://44e075b62a2c639ad174be93dc5d1c410819cc51f100aecfa8e63ec979efe1ac" gracePeriod=30
Jul 27 22:23:41.623947 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:41.623907    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:23:42.378506 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:42.378472    2577 generic.go:356] "Generic (PLEG): container finished" podID="a7bb09e6-fd46-4000-b024-5e4ac6dd94b1" containerID="44e075b62a2c639ad174be93dc5d1c410819cc51f100aecfa8e63ec979efe1ac" exitCode=2
Jul 27 22:23:42.378796 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:42.378554    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bqwts" event={"ID":"a7bb09e6-fd46-4000-b024-5e4ac6dd94b1","Type":"ContainerDied","Data":"44e075b62a2c639ad174be93dc5d1c410819cc51f100aecfa8e63ec979efe1ac"}
Jul 27 22:23:44.587523 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:44.587483    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bqwts"
Jul 27 22:23:44.643206 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:44.643163    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7bb09e6-fd46-4000-b024-5e4ac6dd94b1-proxy-tls\") pod \"a7bb09e6-fd46-4000-b024-5e4ac6dd94b1\" (UID: \"a7bb09e6-fd46-4000-b024-5e4ac6dd94b1\") "
Jul 27 22:23:44.643399 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:44.643227    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7bb09e6-fd46-4000-b024-5e4ac6dd94b1-cabundle-cert\") pod \"a7bb09e6-fd46-4000-b024-5e4ac6dd94b1\" (UID: \"a7bb09e6-fd46-4000-b024-5e4ac6dd94b1\") "
Jul 27 22:23:44.643399 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:44.643258    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5zlw\" (UniqueName: \"kubernetes.io/projected/a7bb09e6-fd46-4000-b024-5e4ac6dd94b1-kube-api-access-h5zlw\") pod \"a7bb09e6-fd46-4000-b024-5e4ac6dd94b1\" (UID: \"a7bb09e6-fd46-4000-b024-5e4ac6dd94b1\") "
Jul 27 22:23:44.643399 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:44.643292    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7bb09e6-fd46-4000-b024-5e4ac6dd94b1-kserve-provision-location\") pod \"a7bb09e6-fd46-4000-b024-5e4ac6dd94b1\" (UID: \"a7bb09e6-fd46-4000-b024-5e4ac6dd94b1\") "
Jul 27 22:23:44.643399 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:44.643317    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7bb09e6-fd46-4000-b024-5e4ac6dd94b1-kube-rbac-proxy-sar-config\") pod \"a7bb09e6-fd46-4000-b024-5e4ac6dd94b1\" (UID: \"a7bb09e6-fd46-4000-b024-5e4ac6dd94b1\") "
Jul 27 22:23:44.643718 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:44.643683    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7bb09e6-fd46-4000-b024-5e4ac6dd94b1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a7bb09e6-fd46-4000-b024-5e4ac6dd94b1" (UID: "a7bb09e6-fd46-4000-b024-5e4ac6dd94b1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:23:44.643718 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:44.643712    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7bb09e6-fd46-4000-b024-5e4ac6dd94b1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a7bb09e6-fd46-4000-b024-5e4ac6dd94b1" (UID: "a7bb09e6-fd46-4000-b024-5e4ac6dd94b1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:23:44.643892 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:44.643733    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7bb09e6-fd46-4000-b024-5e4ac6dd94b1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a7bb09e6-fd46-4000-b024-5e4ac6dd94b1" (UID: "a7bb09e6-fd46-4000-b024-5e4ac6dd94b1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:23:44.645343 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:44.645326    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7bb09e6-fd46-4000-b024-5e4ac6dd94b1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a7bb09e6-fd46-4000-b024-5e4ac6dd94b1" (UID: "a7bb09e6-fd46-4000-b024-5e4ac6dd94b1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 22:23:44.645691 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:44.645668    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7bb09e6-fd46-4000-b024-5e4ac6dd94b1-kube-api-access-h5zlw" (OuterVolumeSpecName: "kube-api-access-h5zlw") pod "a7bb09e6-fd46-4000-b024-5e4ac6dd94b1" (UID: "a7bb09e6-fd46-4000-b024-5e4ac6dd94b1"). InnerVolumeSpecName "kube-api-access-h5zlw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:23:44.744403 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:44.744357    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7bb09e6-fd46-4000-b024-5e4ac6dd94b1-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:23:44.744403 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:44.744395    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7bb09e6-fd46-4000-b024-5e4ac6dd94b1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:23:44.744403 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:44.744405    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7bb09e6-fd46-4000-b024-5e4ac6dd94b1-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:23:44.744403 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:44.744413    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7bb09e6-fd46-4000-b024-5e4ac6dd94b1-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:23:44.744665 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:44.744421    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5zlw\" (UniqueName: \"kubernetes.io/projected/a7bb09e6-fd46-4000-b024-5e4ac6dd94b1-kube-api-access-h5zlw\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:23:45.391851 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:45.391812    2577 generic.go:356] "Generic (PLEG): container finished" podID="a7bb09e6-fd46-4000-b024-5e4ac6dd94b1" containerID="0c590b916dd02fbaf677003baa342413aa8763e229406a9591f9ac1b4cc2437f" exitCode=0
Jul 27 22:23:45.392097 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:45.391866    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bqwts"
Jul 27 22:23:45.392097 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:45.391900    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bqwts" event={"ID":"a7bb09e6-fd46-4000-b024-5e4ac6dd94b1","Type":"ContainerDied","Data":"0c590b916dd02fbaf677003baa342413aa8763e229406a9591f9ac1b4cc2437f"}
Jul 27 22:23:45.392097 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:45.391964    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bqwts" event={"ID":"a7bb09e6-fd46-4000-b024-5e4ac6dd94b1","Type":"ContainerDied","Data":"34c92ca79181b9d491f1c9ff51f77de784e9e19fadab49f309a61c0c039b0701"}
Jul 27 22:23:45.392097 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:45.391993    2577 scope.go:117] "RemoveContainer" containerID="44e075b62a2c639ad174be93dc5d1c410819cc51f100aecfa8e63ec979efe1ac"
Jul 27 22:23:45.400166 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:45.400146    2577 scope.go:117] "RemoveContainer" containerID="0c590b916dd02fbaf677003baa342413aa8763e229406a9591f9ac1b4cc2437f"
Jul 27 22:23:45.407808 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:45.407787    2577 scope.go:117] "RemoveContainer" containerID="48a82015915a5f77b47a9658a721ca31852fdd5f7610209cb91d18362fc8ef8b"
Jul 27 22:23:45.409497 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:45.409475    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bqwts"]
Jul 27 22:23:45.412912 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:45.412893    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-bqwts"]
Jul 27 22:23:45.415551 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:45.415537    2577 scope.go:117] "RemoveContainer" containerID="44e075b62a2c639ad174be93dc5d1c410819cc51f100aecfa8e63ec979efe1ac"
Jul 27 22:23:45.415842 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:23:45.415823    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44e075b62a2c639ad174be93dc5d1c410819cc51f100aecfa8e63ec979efe1ac\": container with ID starting with 44e075b62a2c639ad174be93dc5d1c410819cc51f100aecfa8e63ec979efe1ac not found: ID does not exist" containerID="44e075b62a2c639ad174be93dc5d1c410819cc51f100aecfa8e63ec979efe1ac"
Jul 27 22:23:45.415913 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:45.415848    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44e075b62a2c639ad174be93dc5d1c410819cc51f100aecfa8e63ec979efe1ac"} err="failed to get container status \"44e075b62a2c639ad174be93dc5d1c410819cc51f100aecfa8e63ec979efe1ac\": rpc error: code = NotFound desc = could not find container \"44e075b62a2c639ad174be93dc5d1c410819cc51f100aecfa8e63ec979efe1ac\": container with ID starting with 44e075b62a2c639ad174be93dc5d1c410819cc51f100aecfa8e63ec979efe1ac not found: ID does not exist"
Jul 27 22:23:45.415913 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:45.415864    2577 scope.go:117] "RemoveContainer" containerID="0c590b916dd02fbaf677003baa342413aa8763e229406a9591f9ac1b4cc2437f"
Jul 27 22:23:45.416113 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:23:45.416095    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c590b916dd02fbaf677003baa342413aa8763e229406a9591f9ac1b4cc2437f\": container with ID starting with 0c590b916dd02fbaf677003baa342413aa8763e229406a9591f9ac1b4cc2437f not found: ID does not exist" containerID="0c590b916dd02fbaf677003baa342413aa8763e229406a9591f9ac1b4cc2437f"
Jul 27 22:23:45.416152 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:45.416120    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c590b916dd02fbaf677003baa342413aa8763e229406a9591f9ac1b4cc2437f"} err="failed to get container status \"0c590b916dd02fbaf677003baa342413aa8763e229406a9591f9ac1b4cc2437f\": rpc error: code = NotFound desc = could not find container \"0c590b916dd02fbaf677003baa342413aa8763e229406a9591f9ac1b4cc2437f\": container with ID starting with 0c590b916dd02fbaf677003baa342413aa8763e229406a9591f9ac1b4cc2437f not found: ID does not exist"
Jul 27 22:23:45.416152 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:45.416135    2577 scope.go:117] "RemoveContainer" containerID="48a82015915a5f77b47a9658a721ca31852fdd5f7610209cb91d18362fc8ef8b"
Jul 27 22:23:45.416376 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:23:45.416358    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48a82015915a5f77b47a9658a721ca31852fdd5f7610209cb91d18362fc8ef8b\": container with ID starting with 48a82015915a5f77b47a9658a721ca31852fdd5f7610209cb91d18362fc8ef8b not found: ID does not exist" containerID="48a82015915a5f77b47a9658a721ca31852fdd5f7610209cb91d18362fc8ef8b"
Jul 27 22:23:45.416428 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:45.416380    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48a82015915a5f77b47a9658a721ca31852fdd5f7610209cb91d18362fc8ef8b"} err="failed to get container status \"48a82015915a5f77b47a9658a721ca31852fdd5f7610209cb91d18362fc8ef8b\": rpc error: code = NotFound desc = could not find container \"48a82015915a5f77b47a9658a721ca31852fdd5f7610209cb91d18362fc8ef8b\": container with ID starting with 48a82015915a5f77b47a9658a721ca31852fdd5f7610209cb91d18362fc8ef8b not found: ID does not exist"
Jul 27 22:23:46.700949 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:46.700917    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:23:46.764613 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:46.764579    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-qpdw8"]
Jul 27 22:23:46.764883 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:46.764870    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7bb09e6-fd46-4000-b024-5e4ac6dd94b1" containerName="kserve-container"
Jul 27 22:23:46.764932 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:46.764884    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7bb09e6-fd46-4000-b024-5e4ac6dd94b1" containerName="kserve-container"
Jul 27 22:23:46.764932 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:46.764908    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37680fa4-9f86-4647-8ada-43e9700b3f30" containerName="kserve-container"
Jul 27 22:23:46.764932 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:46.764913    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="37680fa4-9f86-4647-8ada-43e9700b3f30" containerName="kserve-container"
Jul 27 22:23:46.764932 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:46.764921    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37680fa4-9f86-4647-8ada-43e9700b3f30" containerName="storage-initializer"
Jul 27 22:23:46.764932 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:46.764927    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="37680fa4-9f86-4647-8ada-43e9700b3f30" containerName="storage-initializer"
Jul 27 22:23:46.765078 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:46.764934    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7bb09e6-fd46-4000-b024-5e4ac6dd94b1" containerName="kube-rbac-proxy"
Jul 27 22:23:46.765078 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:46.764940    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7bb09e6-fd46-4000-b024-5e4ac6dd94b1" containerName="kube-rbac-proxy"
Jul 27 22:23:46.765078 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:46.764947    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7bb09e6-fd46-4000-b024-5e4ac6dd94b1" containerName="storage-initializer"
Jul 27 22:23:46.765078 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:46.764952    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7bb09e6-fd46-4000-b024-5e4ac6dd94b1" containerName="storage-initializer"
Jul 27 22:23:46.765078 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:46.764959    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37680fa4-9f86-4647-8ada-43e9700b3f30" containerName="kube-rbac-proxy"
Jul 27 22:23:46.765078 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:46.764964    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="37680fa4-9f86-4647-8ada-43e9700b3f30" containerName="kube-rbac-proxy"
Jul 27 22:23:46.765078 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:46.765024    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="37680fa4-9f86-4647-8ada-43e9700b3f30" containerName="kserve-container"
Jul 27 22:23:46.765078 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:46.765031    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="37680fa4-9f86-4647-8ada-43e9700b3f30" containerName="kube-rbac-proxy"
Jul 27 22:23:46.765078 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:46.765039    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7bb09e6-fd46-4000-b024-5e4ac6dd94b1" containerName="kserve-container"
Jul 27 22:23:46.765078 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:46.765044    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7bb09e6-fd46-4000-b024-5e4ac6dd94b1" containerName="kube-rbac-proxy"
Jul 27 22:23:46.795027 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:46.794997    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-qpdw8"]
Jul 27 22:23:46.795210 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:46.795125    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-qpdw8"
Jul 27 22:23:46.797653 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:46.797616    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\""
Jul 27 22:23:46.797854 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:46.797809    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 22:23:46.797854 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:46.797834    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v7c66\""
Jul 27 22:23:46.797968 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:46.797854    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 22:23:46.797968 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:46.797914    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 22:23:46.798442 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:46.798427    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
Jul 27 22:23:46.798505 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:46.798430    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 22:23:46.864017 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:46.863979    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dabb304a-e144-4790-a84b-dc6b365be53e-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-qpdw8\" (UID: \"dabb304a-e144-4790-a84b-dc6b365be53e\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-qpdw8"
Jul 27 22:23:46.864017 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:46.864015    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zl7g\" (UniqueName: \"kubernetes.io/projected/dabb304a-e144-4790-a84b-dc6b365be53e-kube-api-access-6zl7g\") pod \"isvc-xgboost-predictor-7566d65bf7-qpdw8\" (UID: \"dabb304a-e144-4790-a84b-dc6b365be53e\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-qpdw8"
Jul 27 22:23:46.864257 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:46.864047    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dabb304a-e144-4790-a84b-dc6b365be53e-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-qpdw8\" (UID: \"dabb304a-e144-4790-a84b-dc6b365be53e\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-qpdw8"
Jul 27 22:23:46.864257 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:46.864090    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dabb304a-e144-4790-a84b-dc6b365be53e-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-qpdw8\" (UID: \"dabb304a-e144-4790-a84b-dc6b365be53e\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-qpdw8"
Jul 27 22:23:46.864257 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:46.864128    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dabb304a-e144-4790-a84b-dc6b365be53e-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-qpdw8\" (UID: \"dabb304a-e144-4790-a84b-dc6b365be53e\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-qpdw8"
Jul 27 22:23:46.913535 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:46.913502    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:23:46.964783 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:46.964686    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zl7g\" (UniqueName: \"kubernetes.io/projected/dabb304a-e144-4790-a84b-dc6b365be53e-kube-api-access-6zl7g\") pod \"isvc-xgboost-predictor-7566d65bf7-qpdw8\" (UID: \"dabb304a-e144-4790-a84b-dc6b365be53e\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-qpdw8"
Jul 27 22:23:46.964783 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:46.964725    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dabb304a-e144-4790-a84b-dc6b365be53e-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-qpdw8\" (UID: \"dabb304a-e144-4790-a84b-dc6b365be53e\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-qpdw8"
Jul 27 22:23:46.964783 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:46.964746    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dabb304a-e144-4790-a84b-dc6b365be53e-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-qpdw8\" (UID: \"dabb304a-e144-4790-a84b-dc6b365be53e\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-qpdw8"
Jul 27 22:23:46.965058 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:46.964789    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dabb304a-e144-4790-a84b-dc6b365be53e-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-qpdw8\" (UID: \"dabb304a-e144-4790-a84b-dc6b365be53e\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-qpdw8"
Jul 27 22:23:46.965058 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:46.964869    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dabb304a-e144-4790-a84b-dc6b365be53e-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-qpdw8\" (UID: \"dabb304a-e144-4790-a84b-dc6b365be53e\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-qpdw8"
Jul 27 22:23:46.965279 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:46.965256    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dabb304a-e144-4790-a84b-dc6b365be53e-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-qpdw8\" (UID: \"dabb304a-e144-4790-a84b-dc6b365be53e\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-qpdw8"
Jul 27 22:23:46.965435 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:46.965415    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dabb304a-e144-4790-a84b-dc6b365be53e-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-qpdw8\" (UID: \"dabb304a-e144-4790-a84b-dc6b365be53e\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-qpdw8"
Jul 27 22:23:46.965531 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:46.965512    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dabb304a-e144-4790-a84b-dc6b365be53e-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-qpdw8\" (UID: \"dabb304a-e144-4790-a84b-dc6b365be53e\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-qpdw8"
Jul 27 22:23:46.967317 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:46.967300    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dabb304a-e144-4790-a84b-dc6b365be53e-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-qpdw8\" (UID: \"dabb304a-e144-4790-a84b-dc6b365be53e\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-qpdw8"
Jul 27 22:23:46.973223 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:46.973194    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zl7g\" (UniqueName: \"kubernetes.io/projected/dabb304a-e144-4790-a84b-dc6b365be53e-kube-api-access-6zl7g\") pod \"isvc-xgboost-predictor-7566d65bf7-qpdw8\" (UID: \"dabb304a-e144-4790-a84b-dc6b365be53e\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-qpdw8"
Jul 27 22:23:47.105996 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:47.105953    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-qpdw8"
Jul 27 22:23:47.136274 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:47.136229    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7bb09e6-fd46-4000-b024-5e4ac6dd94b1" path="/var/lib/kubelet/pods/a7bb09e6-fd46-4000-b024-5e4ac6dd94b1/volumes"
Jul 27 22:23:47.301240 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:47.301030    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-qpdw8"]
Jul 27 22:23:47.304020 ip-10-0-142-219 kubenswrapper[2577]: W0727 22:23:47.303996    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddabb304a_e144_4790_a84b_dc6b365be53e.slice/crio-8e943310e21cdfeaa510c50342a8979be50446f3b1c543aa6155d03d4485ab02 WatchSource:0}: Error finding container 8e943310e21cdfeaa510c50342a8979be50446f3b1c543aa6155d03d4485ab02: Status 404 returned error can't find the container with id 8e943310e21cdfeaa510c50342a8979be50446f3b1c543aa6155d03d4485ab02
Jul 27 22:23:47.402732 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:47.402700    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-qpdw8" event={"ID":"dabb304a-e144-4790-a84b-dc6b365be53e","Type":"ContainerStarted","Data":"0a6b2c892b56f5ce06e2196262d270752043079e025a8de7f13a474e42995126"}
Jul 27 22:23:47.402905 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:47.402778    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-qpdw8" event={"ID":"dabb304a-e144-4790-a84b-dc6b365be53e","Type":"ContainerStarted","Data":"8e943310e21cdfeaa510c50342a8979be50446f3b1c543aa6155d03d4485ab02"}
Jul 27 22:23:50.829105 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:50.829048    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:23:51.419512 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:51.419429    2577 generic.go:356] "Generic (PLEG): container finished" podID="dabb304a-e144-4790-a84b-dc6b365be53e" containerID="0a6b2c892b56f5ce06e2196262d270752043079e025a8de7f13a474e42995126" exitCode=0
Jul 27 22:23:51.419655 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:51.419505    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-qpdw8" event={"ID":"dabb304a-e144-4790-a84b-dc6b365be53e","Type":"ContainerDied","Data":"0a6b2c892b56f5ce06e2196262d270752043079e025a8de7f13a474e42995126"}
Jul 27 22:23:52.353558 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:23:52.353525    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:24:11.504504 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:24:11.504464    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-qpdw8" event={"ID":"dabb304a-e144-4790-a84b-dc6b365be53e","Type":"ContainerStarted","Data":"b2f2557675f55f5824b2e1ccf1badabea443756deb2df3f0f4d463d8eb628b65"}
Jul 27 22:24:11.505037 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:24:11.504547    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-qpdw8" event={"ID":"dabb304a-e144-4790-a84b-dc6b365be53e","Type":"ContainerStarted","Data":"2ad9d09e28718d1b0cc44b81f737246b6256f9a4ba3601cf3af883317c17e142"}
Jul 27 22:24:11.505037 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:24:11.504781    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-qpdw8"
Jul 27 22:24:11.524326 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:24:11.524268    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-qpdw8" podStartSLOduration=2.6459847720000003 podStartE2EDuration="25.52425405s" totalImagesPullingTime="19.134177607s" totalInitContainerRuntime="3.744091671s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:23:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:24:11.522165015 +0000 UTC m=+4774.889354880" watchObservedRunningTime="2026-07-27 22:24:11.52425405 +0000 UTC m=+4774.891443916"
Jul 27 22:24:12.509304 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:24:12.509270    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-qpdw8"
Jul 27 22:24:12.510706 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:24:12.510677    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-qpdw8" podUID="dabb304a-e144-4790-a84b-dc6b365be53e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Jul 27 22:24:13.512948 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:24:13.512907    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-qpdw8" podUID="dabb304a-e144-4790-a84b-dc6b365be53e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Jul 27 22:24:18.517339 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:24:18.517309    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-qpdw8"
Jul 27 22:24:18.517981 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:24:18.517913    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-qpdw8" podUID="dabb304a-e144-4790-a84b-dc6b365be53e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Jul 27 22:24:28.518300 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:24:28.518253    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-qpdw8" podUID="dabb304a-e144-4790-a84b-dc6b365be53e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Jul 27 22:24:38.518856 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:24:38.518809    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-qpdw8" podUID="dabb304a-e144-4790-a84b-dc6b365be53e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Jul 27 22:24:48.517893 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:24:48.517856    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-qpdw8" podUID="dabb304a-e144-4790-a84b-dc6b365be53e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Jul 27 22:24:58.518346 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:24:58.518310    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-qpdw8" podUID="dabb304a-e144-4790-a84b-dc6b365be53e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Jul 27 22:25:08.517945 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:08.517896    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-qpdw8" podUID="dabb304a-e144-4790-a84b-dc6b365be53e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Jul 27 22:25:18.518928 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:18.518900    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-qpdw8"
Jul 27 22:25:26.872344 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:26.872296    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-qpdw8"]
Jul 27 22:25:26.872815 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:26.872599    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-qpdw8" podUID="dabb304a-e144-4790-a84b-dc6b365be53e" containerName="kserve-container" containerID="cri-o://2ad9d09e28718d1b0cc44b81f737246b6256f9a4ba3601cf3af883317c17e142" gracePeriod=30
Jul 27 22:25:26.872815 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:26.872647    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-qpdw8" podUID="dabb304a-e144-4790-a84b-dc6b365be53e" containerName="kube-rbac-proxy" containerID="cri-o://b2f2557675f55f5824b2e1ccf1badabea443756deb2df3f0f4d463d8eb628b65" gracePeriod=30
Jul 27 22:25:26.922161 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:26.922127    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:25:27.797294 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:27.797257    2577 generic.go:356] "Generic (PLEG): container finished" podID="dabb304a-e144-4790-a84b-dc6b365be53e" containerID="b2f2557675f55f5824b2e1ccf1badabea443756deb2df3f0f4d463d8eb628b65" exitCode=2
Jul 27 22:25:27.797463 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:27.797327    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-qpdw8" event={"ID":"dabb304a-e144-4790-a84b-dc6b365be53e","Type":"ContainerDied","Data":"b2f2557675f55f5824b2e1ccf1badabea443756deb2df3f0f4d463d8eb628b65"}
Jul 27 22:25:28.513528 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:28.513485    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-qpdw8" podUID="dabb304a-e144-4790-a84b-dc6b365be53e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.67:8643/healthz\": dial tcp 10.133.0.67:8643: connect: connection refused"
Jul 27 22:25:28.517789 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:28.517751    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-qpdw8" podUID="dabb304a-e144-4790-a84b-dc6b365be53e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.67:8080: connect: connection refused"
Jul 27 22:25:30.411001 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:30.410980    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-qpdw8"
Jul 27 22:25:30.505330 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:30.505253    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dabb304a-e144-4790-a84b-dc6b365be53e-cabundle-cert\") pod \"dabb304a-e144-4790-a84b-dc6b365be53e\" (UID: \"dabb304a-e144-4790-a84b-dc6b365be53e\") "
Jul 27 22:25:30.505330 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:30.505307    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6zl7g\" (UniqueName: \"kubernetes.io/projected/dabb304a-e144-4790-a84b-dc6b365be53e-kube-api-access-6zl7g\") pod \"dabb304a-e144-4790-a84b-dc6b365be53e\" (UID: \"dabb304a-e144-4790-a84b-dc6b365be53e\") "
Jul 27 22:25:30.505330 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:30.505331    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dabb304a-e144-4790-a84b-dc6b365be53e-kube-rbac-proxy-sar-config\") pod \"dabb304a-e144-4790-a84b-dc6b365be53e\" (UID: \"dabb304a-e144-4790-a84b-dc6b365be53e\") "
Jul 27 22:25:30.505620 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:30.505362    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dabb304a-e144-4790-a84b-dc6b365be53e-kserve-provision-location\") pod \"dabb304a-e144-4790-a84b-dc6b365be53e\" (UID: \"dabb304a-e144-4790-a84b-dc6b365be53e\") "
Jul 27 22:25:30.505620 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:30.505378    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dabb304a-e144-4790-a84b-dc6b365be53e-proxy-tls\") pod \"dabb304a-e144-4790-a84b-dc6b365be53e\" (UID: \"dabb304a-e144-4790-a84b-dc6b365be53e\") "
Jul 27 22:25:30.505745 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:30.505677    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dabb304a-e144-4790-a84b-dc6b365be53e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dabb304a-e144-4790-a84b-dc6b365be53e" (UID: "dabb304a-e144-4790-a84b-dc6b365be53e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:25:30.505745 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:30.505708    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dabb304a-e144-4790-a84b-dc6b365be53e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dabb304a-e144-4790-a84b-dc6b365be53e" (UID: "dabb304a-e144-4790-a84b-dc6b365be53e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:25:30.505745 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:30.505720    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dabb304a-e144-4790-a84b-dc6b365be53e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dabb304a-e144-4790-a84b-dc6b365be53e" (UID: "dabb304a-e144-4790-a84b-dc6b365be53e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:25:30.507491 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:30.507469    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dabb304a-e144-4790-a84b-dc6b365be53e-kube-api-access-6zl7g" (OuterVolumeSpecName: "kube-api-access-6zl7g") pod "dabb304a-e144-4790-a84b-dc6b365be53e" (UID: "dabb304a-e144-4790-a84b-dc6b365be53e"). InnerVolumeSpecName "kube-api-access-6zl7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:25:30.507565 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:30.507537    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dabb304a-e144-4790-a84b-dc6b365be53e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dabb304a-e144-4790-a84b-dc6b365be53e" (UID: "dabb304a-e144-4790-a84b-dc6b365be53e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 22:25:30.606338 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:30.606301    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dabb304a-e144-4790-a84b-dc6b365be53e-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:25:30.606338 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:30.606330    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dabb304a-e144-4790-a84b-dc6b365be53e-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:25:30.606338 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:30.606340    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dabb304a-e144-4790-a84b-dc6b365be53e-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:25:30.606338 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:30.606349    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zl7g\" (UniqueName: \"kubernetes.io/projected/dabb304a-e144-4790-a84b-dc6b365be53e-kube-api-access-6zl7g\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:25:30.606610 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:30.606358    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dabb304a-e144-4790-a84b-dc6b365be53e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:25:30.811054 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:30.810966    2577 generic.go:356] "Generic (PLEG): container finished" podID="dabb304a-e144-4790-a84b-dc6b365be53e" containerID="2ad9d09e28718d1b0cc44b81f737246b6256f9a4ba3601cf3af883317c17e142" exitCode=0
Jul 27 22:25:30.811054 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:30.811016    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-qpdw8" event={"ID":"dabb304a-e144-4790-a84b-dc6b365be53e","Type":"ContainerDied","Data":"2ad9d09e28718d1b0cc44b81f737246b6256f9a4ba3601cf3af883317c17e142"}
Jul 27 22:25:30.811279 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:30.811082    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-qpdw8" event={"ID":"dabb304a-e144-4790-a84b-dc6b365be53e","Type":"ContainerDied","Data":"8e943310e21cdfeaa510c50342a8979be50446f3b1c543aa6155d03d4485ab02"}
Jul 27 22:25:30.811279 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:30.811027    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-qpdw8"
Jul 27 22:25:30.811279 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:30.811112    2577 scope.go:117] "RemoveContainer" containerID="b2f2557675f55f5824b2e1ccf1badabea443756deb2df3f0f4d463d8eb628b65"
Jul 27 22:25:30.820685 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:30.820671    2577 scope.go:117] "RemoveContainer" containerID="2ad9d09e28718d1b0cc44b81f737246b6256f9a4ba3601cf3af883317c17e142"
Jul 27 22:25:30.828051 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:30.828034    2577 scope.go:117] "RemoveContainer" containerID="0a6b2c892b56f5ce06e2196262d270752043079e025a8de7f13a474e42995126"
Jul 27 22:25:30.835266 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:30.835243    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-qpdw8"]
Jul 27 22:25:30.835733 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:30.835717    2577 scope.go:117] "RemoveContainer" containerID="b2f2557675f55f5824b2e1ccf1badabea443756deb2df3f0f4d463d8eb628b65"
Jul 27 22:25:30.836032 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:25:30.835999    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2f2557675f55f5824b2e1ccf1badabea443756deb2df3f0f4d463d8eb628b65\": container with ID starting with b2f2557675f55f5824b2e1ccf1badabea443756deb2df3f0f4d463d8eb628b65 not found: ID does not exist" containerID="b2f2557675f55f5824b2e1ccf1badabea443756deb2df3f0f4d463d8eb628b65"
Jul 27 22:25:30.836126 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:30.836047    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2f2557675f55f5824b2e1ccf1badabea443756deb2df3f0f4d463d8eb628b65"} err="failed to get container status \"b2f2557675f55f5824b2e1ccf1badabea443756deb2df3f0f4d463d8eb628b65\": rpc error: code = NotFound desc = could not find container \"b2f2557675f55f5824b2e1ccf1badabea443756deb2df3f0f4d463d8eb628b65\": container with ID starting with b2f2557675f55f5824b2e1ccf1badabea443756deb2df3f0f4d463d8eb628b65 not found: ID does not exist"
Jul 27 22:25:30.836126 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:30.836072    2577 scope.go:117] "RemoveContainer" containerID="2ad9d09e28718d1b0cc44b81f737246b6256f9a4ba3601cf3af883317c17e142"
Jul 27 22:25:30.836319 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:25:30.836302    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ad9d09e28718d1b0cc44b81f737246b6256f9a4ba3601cf3af883317c17e142\": container with ID starting with 2ad9d09e28718d1b0cc44b81f737246b6256f9a4ba3601cf3af883317c17e142 not found: ID does not exist" containerID="2ad9d09e28718d1b0cc44b81f737246b6256f9a4ba3601cf3af883317c17e142"
Jul 27 22:25:30.836360 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:30.836325    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ad9d09e28718d1b0cc44b81f737246b6256f9a4ba3601cf3af883317c17e142"} err="failed to get container status \"2ad9d09e28718d1b0cc44b81f737246b6256f9a4ba3601cf3af883317c17e142\": rpc error: code = NotFound desc = could not find container \"2ad9d09e28718d1b0cc44b81f737246b6256f9a4ba3601cf3af883317c17e142\": container with ID starting with 2ad9d09e28718d1b0cc44b81f737246b6256f9a4ba3601cf3af883317c17e142 not found: ID does not exist"
Jul 27 22:25:30.836360 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:30.836340    2577 scope.go:117] "RemoveContainer" containerID="0a6b2c892b56f5ce06e2196262d270752043079e025a8de7f13a474e42995126"
Jul 27 22:25:30.836559 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:25:30.836540    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a6b2c892b56f5ce06e2196262d270752043079e025a8de7f13a474e42995126\": container with ID starting with 0a6b2c892b56f5ce06e2196262d270752043079e025a8de7f13a474e42995126 not found: ID does not exist" containerID="0a6b2c892b56f5ce06e2196262d270752043079e025a8de7f13a474e42995126"
Jul 27 22:25:30.836605 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:30.836564    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a6b2c892b56f5ce06e2196262d270752043079e025a8de7f13a474e42995126"} err="failed to get container status \"0a6b2c892b56f5ce06e2196262d270752043079e025a8de7f13a474e42995126\": rpc error: code = NotFound desc = could not find container \"0a6b2c892b56f5ce06e2196262d270752043079e025a8de7f13a474e42995126\": container with ID starting with 0a6b2c892b56f5ce06e2196262d270752043079e025a8de7f13a474e42995126 not found: ID does not exist"
Jul 27 22:25:30.841159 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:30.841138    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-qpdw8"]
Jul 27 22:25:31.135358 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:31.135266    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dabb304a-e144-4790-a84b-dc6b365be53e" path="/var/lib/kubelet/pods/dabb304a-e144-4790-a84b-dc6b365be53e/volumes"
Jul 27 22:25:31.982258 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:31.982222    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:25:32.048255 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:32.048220    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6mrk7"]
Jul 27 22:25:32.048540 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:32.048527    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dabb304a-e144-4790-a84b-dc6b365be53e" containerName="storage-initializer"
Jul 27 22:25:32.048581 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:32.048541    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="dabb304a-e144-4790-a84b-dc6b365be53e" containerName="storage-initializer"
Jul 27 22:25:32.048581 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:32.048551    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dabb304a-e144-4790-a84b-dc6b365be53e" containerName="kube-rbac-proxy"
Jul 27 22:25:32.048581 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:32.048558    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="dabb304a-e144-4790-a84b-dc6b365be53e" containerName="kube-rbac-proxy"
Jul 27 22:25:32.048581 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:32.048573    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dabb304a-e144-4790-a84b-dc6b365be53e" containerName="kserve-container"
Jul 27 22:25:32.048581 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:32.048579    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="dabb304a-e144-4790-a84b-dc6b365be53e" containerName="kserve-container"
Jul 27 22:25:32.048746 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:32.048622    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="dabb304a-e144-4790-a84b-dc6b365be53e" containerName="kube-rbac-proxy"
Jul 27 22:25:32.048746 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:32.048632    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="dabb304a-e144-4790-a84b-dc6b365be53e" containerName="kserve-container"
Jul 27 22:25:32.051013 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:32.050993    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6mrk7"
Jul 27 22:25:32.053793 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:32.053755    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
Jul 27 22:25:32.053980 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:32.053827    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 22:25:32.053980 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:32.053844    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 22:25:32.053980 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:32.053859    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 22:25:32.053980 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:32.053852    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 22:25:32.054470 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:32.054451    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Jul 27 22:25:32.054470 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:32.054460    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v7c66\""
Jul 27 22:25:32.062993 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:32.062964    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6mrk7"]
Jul 27 22:25:32.114851 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:32.114804    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/400fdac5-21e2-4f10-a1ec-a51e61b86869-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6mrk7\" (UID: \"400fdac5-21e2-4f10-a1ec-a51e61b86869\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6mrk7"
Jul 27 22:25:32.114851 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:32.114852    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlgdz\" (UniqueName: \"kubernetes.io/projected/400fdac5-21e2-4f10-a1ec-a51e61b86869-kube-api-access-jlgdz\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6mrk7\" (UID: \"400fdac5-21e2-4f10-a1ec-a51e61b86869\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6mrk7"
Jul 27 22:25:32.115119 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:32.114877    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/400fdac5-21e2-4f10-a1ec-a51e61b86869-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6mrk7\" (UID: \"400fdac5-21e2-4f10-a1ec-a51e61b86869\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6mrk7"
Jul 27 22:25:32.115119 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:32.114908    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/400fdac5-21e2-4f10-a1ec-a51e61b86869-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6mrk7\" (UID: \"400fdac5-21e2-4f10-a1ec-a51e61b86869\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6mrk7"
Jul 27 22:25:32.115119 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:32.115010    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/400fdac5-21e2-4f10-a1ec-a51e61b86869-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6mrk7\" (UID: \"400fdac5-21e2-4f10-a1ec-a51e61b86869\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6mrk7"
Jul 27 22:25:32.130208 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:32.130175    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:25:32.215398 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:32.215360    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/400fdac5-21e2-4f10-a1ec-a51e61b86869-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6mrk7\" (UID: \"400fdac5-21e2-4f10-a1ec-a51e61b86869\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6mrk7"
Jul 27 22:25:32.215398 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:32.215400    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlgdz\" (UniqueName: \"kubernetes.io/projected/400fdac5-21e2-4f10-a1ec-a51e61b86869-kube-api-access-jlgdz\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6mrk7\" (UID: \"400fdac5-21e2-4f10-a1ec-a51e61b86869\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6mrk7"
Jul 27 22:25:32.215657 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:32.215513    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/400fdac5-21e2-4f10-a1ec-a51e61b86869-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6mrk7\" (UID: \"400fdac5-21e2-4f10-a1ec-a51e61b86869\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6mrk7"
Jul 27 22:25:32.215657 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:32.215576    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/400fdac5-21e2-4f10-a1ec-a51e61b86869-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6mrk7\" (UID: \"400fdac5-21e2-4f10-a1ec-a51e61b86869\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6mrk7"
Jul 27 22:25:32.215657 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:32.215612    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/400fdac5-21e2-4f10-a1ec-a51e61b86869-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6mrk7\" (UID: \"400fdac5-21e2-4f10-a1ec-a51e61b86869\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6mrk7"
Jul 27 22:25:32.215997 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:32.215976    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/400fdac5-21e2-4f10-a1ec-a51e61b86869-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6mrk7\" (UID: \"400fdac5-21e2-4f10-a1ec-a51e61b86869\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6mrk7"
Jul 27 22:25:32.216124 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:32.216106    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/400fdac5-21e2-4f10-a1ec-a51e61b86869-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6mrk7\" (UID: \"400fdac5-21e2-4f10-a1ec-a51e61b86869\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6mrk7"
Jul 27 22:25:32.216192 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:32.216178    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/400fdac5-21e2-4f10-a1ec-a51e61b86869-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6mrk7\" (UID: \"400fdac5-21e2-4f10-a1ec-a51e61b86869\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6mrk7"
Jul 27 22:25:32.218019 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:32.217995    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/400fdac5-21e2-4f10-a1ec-a51e61b86869-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6mrk7\" (UID: \"400fdac5-21e2-4f10-a1ec-a51e61b86869\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6mrk7"
Jul 27 22:25:32.224985 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:32.224966    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlgdz\" (UniqueName: \"kubernetes.io/projected/400fdac5-21e2-4f10-a1ec-a51e61b86869-kube-api-access-jlgdz\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6mrk7\" (UID: \"400fdac5-21e2-4f10-a1ec-a51e61b86869\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6mrk7"
Jul 27 22:25:32.361281 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:32.361194    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6mrk7"
Jul 27 22:25:32.493927 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:32.493899    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6mrk7"]
Jul 27 22:25:32.496503 ip-10-0-142-219 kubenswrapper[2577]: W0727 22:25:32.496469    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod400fdac5_21e2_4f10_a1ec_a51e61b86869.slice/crio-6a835ff90373e66f3b9b06e85f105bf5959e0722c1c46f6d081910c1c350c57e WatchSource:0}: Error finding container 6a835ff90373e66f3b9b06e85f105bf5959e0722c1c46f6d081910c1c350c57e: Status 404 returned error can't find the container with id 6a835ff90373e66f3b9b06e85f105bf5959e0722c1c46f6d081910c1c350c57e
Jul 27 22:25:32.825383 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:32.825345    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6mrk7" event={"ID":"400fdac5-21e2-4f10-a1ec-a51e61b86869","Type":"ContainerStarted","Data":"56a450ac5822a6357aa6812ee67df2f591c8f21cf05637a6c3b896283f4f9d37"}
Jul 27 22:25:32.825583 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:32.825405    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6mrk7" event={"ID":"400fdac5-21e2-4f10-a1ec-a51e61b86869","Type":"ContainerStarted","Data":"6a835ff90373e66f3b9b06e85f105bf5959e0722c1c46f6d081910c1c350c57e"}
Jul 27 22:25:36.843289 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:36.843258    2577 generic.go:356] "Generic (PLEG): container finished" podID="400fdac5-21e2-4f10-a1ec-a51e61b86869" containerID="56a450ac5822a6357aa6812ee67df2f591c8f21cf05637a6c3b896283f4f9d37" exitCode=0
Jul 27 22:25:36.843674 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:36.843341    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6mrk7" event={"ID":"400fdac5-21e2-4f10-a1ec-a51e61b86869","Type":"ContainerDied","Data":"56a450ac5822a6357aa6812ee67df2f591c8f21cf05637a6c3b896283f4f9d37"}
Jul 27 22:25:37.849073 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:37.849037    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6mrk7" event={"ID":"400fdac5-21e2-4f10-a1ec-a51e61b86869","Type":"ContainerStarted","Data":"53b07b634936ccfea16c240cc9eaed2674c26471ea1ec02935d8667f21c6a6c5"}
Jul 27 22:25:37.849543 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:37.849109    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6mrk7" event={"ID":"400fdac5-21e2-4f10-a1ec-a51e61b86869","Type":"ContainerStarted","Data":"14b5c3057f5e7c4d5aa5e4de074b4b27e280c1581a69bf71a77206137bbc8b90"}
Jul 27 22:25:37.849543 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:37.849324    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6mrk7"
Jul 27 22:25:37.873424 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:37.873384    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6mrk7" podStartSLOduration=2.1832173360000002 podStartE2EDuration="5.873363753s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.690146417s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:25:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:25:37.869402024 +0000 UTC m=+4861.236591887" watchObservedRunningTime="2026-07-27 22:25:37.873363753 +0000 UTC m=+4861.240553617"
Jul 27 22:25:38.854011 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:38.853971    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6mrk7"
Jul 27 22:25:44.863256 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:25:44.863228    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6mrk7"
Jul 27 22:26:14.867601 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:14.867573    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6mrk7"
Jul 27 22:26:22.080030 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:22.079993    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6mrk7"]
Jul 27 22:26:22.080578 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:22.080222    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6mrk7" podUID="400fdac5-21e2-4f10-a1ec-a51e61b86869" containerName="kserve-container" containerID="cri-o://14b5c3057f5e7c4d5aa5e4de074b4b27e280c1581a69bf71a77206137bbc8b90" gracePeriod=30
Jul 27 22:26:22.080578 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:22.080265    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6mrk7" podUID="400fdac5-21e2-4f10-a1ec-a51e61b86869" containerName="kube-rbac-proxy" containerID="cri-o://53b07b634936ccfea16c240cc9eaed2674c26471ea1ec02935d8667f21c6a6c5" gracePeriod=30
Jul 27 22:26:22.100291 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:22.100260    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:26:23.030428 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:23.030397    2577 generic.go:356] "Generic (PLEG): container finished" podID="400fdac5-21e2-4f10-a1ec-a51e61b86869" containerID="53b07b634936ccfea16c240cc9eaed2674c26471ea1ec02935d8667f21c6a6c5" exitCode=2
Jul 27 22:26:23.030599 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:23.030469    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6mrk7" event={"ID":"400fdac5-21e2-4f10-a1ec-a51e61b86869","Type":"ContainerDied","Data":"53b07b634936ccfea16c240cc9eaed2674c26471ea1ec02935d8667f21c6a6c5"}
Jul 27 22:26:24.858313 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:24.858274    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6mrk7" podUID="400fdac5-21e2-4f10-a1ec-a51e61b86869" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.68:8643/healthz\": dial tcp 10.133.0.68:8643: connect: connection refused"
Jul 27 22:26:24.863758 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:24.863729    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6mrk7" podUID="400fdac5-21e2-4f10-a1ec-a51e61b86869" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.68:8080/v2/models/isvc-xgboost-v2-mlserver/ready\": dial tcp 10.133.0.68:8080: connect: connection refused"
Jul 27 22:26:27.234185 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:27.234152    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:26:27.266020 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:27.265987    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9drb4"]
Jul 27 22:26:27.268801 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:27.268783    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9drb4"
Jul 27 22:26:27.270070 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:27.270047    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee02286a-40e4-414d-8b03-da6fef2d8e0a-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-9drb4\" (UID: \"ee02286a-40e4-414d-8b03-da6fef2d8e0a\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9drb4"
Jul 27 22:26:27.270190 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:27.270113    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee02286a-40e4-414d-8b03-da6fef2d8e0a-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-9drb4\" (UID: \"ee02286a-40e4-414d-8b03-da6fef2d8e0a\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9drb4"
Jul 27 22:26:27.270190 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:27.270138    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84wkb\" (UniqueName: \"kubernetes.io/projected/ee02286a-40e4-414d-8b03-da6fef2d8e0a-kube-api-access-84wkb\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-9drb4\" (UID: \"ee02286a-40e4-414d-8b03-da6fef2d8e0a\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9drb4"
Jul 27 22:26:27.270275 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:27.270162    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee02286a-40e4-414d-8b03-da6fef2d8e0a-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-9drb4\" (UID: \"ee02286a-40e4-414d-8b03-da6fef2d8e0a\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9drb4"
Jul 27 22:26:27.270275 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:27.270213    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee02286a-40e4-414d-8b03-da6fef2d8e0a-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-9drb4\" (UID: \"ee02286a-40e4-414d-8b03-da6fef2d8e0a\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9drb4"
Jul 27 22:26:27.271125 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:27.271106    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
Jul 27 22:26:27.271220 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:27.271132    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Jul 27 22:26:27.280301 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:27.280281    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9drb4"]
Jul 27 22:26:27.371337 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:27.371309    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee02286a-40e4-414d-8b03-da6fef2d8e0a-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-9drb4\" (UID: \"ee02286a-40e4-414d-8b03-da6fef2d8e0a\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9drb4"
Jul 27 22:26:27.371487 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:27.371377    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee02286a-40e4-414d-8b03-da6fef2d8e0a-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-9drb4\" (UID: \"ee02286a-40e4-414d-8b03-da6fef2d8e0a\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9drb4"
Jul 27 22:26:27.371487 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:27.371421    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84wkb\" (UniqueName: \"kubernetes.io/projected/ee02286a-40e4-414d-8b03-da6fef2d8e0a-kube-api-access-84wkb\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-9drb4\" (UID: \"ee02286a-40e4-414d-8b03-da6fef2d8e0a\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9drb4"
Jul 27 22:26:27.371487 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:27.371457    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee02286a-40e4-414d-8b03-da6fef2d8e0a-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-9drb4\" (UID: \"ee02286a-40e4-414d-8b03-da6fef2d8e0a\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9drb4"
Jul 27 22:26:27.371487 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:27.371481    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee02286a-40e4-414d-8b03-da6fef2d8e0a-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-9drb4\" (UID: \"ee02286a-40e4-414d-8b03-da6fef2d8e0a\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9drb4"
Jul 27 22:26:27.371705 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:26:27.371646    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-serving-cert: secret "xgboost-v2-mlserver-predictor-serving-cert" not found
Jul 27 22:26:27.371749 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:26:27.371712    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee02286a-40e4-414d-8b03-da6fef2d8e0a-proxy-tls podName:ee02286a-40e4-414d-8b03-da6fef2d8e0a nodeName:}" failed. No retries permitted until 2026-07-27 22:26:27.87169243 +0000 UTC m=+4911.238882276 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ee02286a-40e4-414d-8b03-da6fef2d8e0a-proxy-tls") pod "xgboost-v2-mlserver-predictor-59944bcf4c-9drb4" (UID: "ee02286a-40e4-414d-8b03-da6fef2d8e0a") : secret "xgboost-v2-mlserver-predictor-serving-cert" not found
Jul 27 22:26:27.371886 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:27.371864    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee02286a-40e4-414d-8b03-da6fef2d8e0a-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-9drb4\" (UID: \"ee02286a-40e4-414d-8b03-da6fef2d8e0a\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9drb4"
Jul 27 22:26:27.372067 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:27.372052    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee02286a-40e4-414d-8b03-da6fef2d8e0a-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-9drb4\" (UID: \"ee02286a-40e4-414d-8b03-da6fef2d8e0a\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9drb4"
Jul 27 22:26:27.372178 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:27.372160    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee02286a-40e4-414d-8b03-da6fef2d8e0a-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-9drb4\" (UID: \"ee02286a-40e4-414d-8b03-da6fef2d8e0a\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9drb4"
Jul 27 22:26:27.380224 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:27.380199    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84wkb\" (UniqueName: \"kubernetes.io/projected/ee02286a-40e4-414d-8b03-da6fef2d8e0a-kube-api-access-84wkb\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-9drb4\" (UID: \"ee02286a-40e4-414d-8b03-da6fef2d8e0a\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9drb4"
Jul 27 22:26:27.875611 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:27.875580    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee02286a-40e4-414d-8b03-da6fef2d8e0a-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-9drb4\" (UID: \"ee02286a-40e4-414d-8b03-da6fef2d8e0a\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9drb4"
Jul 27 22:26:27.877984 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:27.877955    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee02286a-40e4-414d-8b03-da6fef2d8e0a-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-9drb4\" (UID: \"ee02286a-40e4-414d-8b03-da6fef2d8e0a\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9drb4"
Jul 27 22:26:27.881348 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:27.881324    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9drb4"
Jul 27 22:26:27.891914 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:27.891883    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:26:28.013259 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:28.013235    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9drb4"]
Jul 27 22:26:28.015395 ip-10-0-142-219 kubenswrapper[2577]: W0727 22:26:28.015367    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee02286a_40e4_414d_8b03_da6fef2d8e0a.slice/crio-251c0d034cc563b114b219e971c17bd19d3c3c889973644b852869c6fb12ee77 WatchSource:0}: Error finding container 251c0d034cc563b114b219e971c17bd19d3c3c889973644b852869c6fb12ee77: Status 404 returned error can't find the container with id 251c0d034cc563b114b219e971c17bd19d3c3c889973644b852869c6fb12ee77
Jul 27 22:26:28.057258 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:28.057231    2577 generic.go:356] "Generic (PLEG): container finished" podID="400fdac5-21e2-4f10-a1ec-a51e61b86869" containerID="14b5c3057f5e7c4d5aa5e4de074b4b27e280c1581a69bf71a77206137bbc8b90" exitCode=0
Jul 27 22:26:28.057382 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:28.057294    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6mrk7" event={"ID":"400fdac5-21e2-4f10-a1ec-a51e61b86869","Type":"ContainerDied","Data":"14b5c3057f5e7c4d5aa5e4de074b4b27e280c1581a69bf71a77206137bbc8b90"}
Jul 27 22:26:28.058973 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:28.058953    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9drb4" event={"ID":"ee02286a-40e4-414d-8b03-da6fef2d8e0a","Type":"ContainerStarted","Data":"251c0d034cc563b114b219e971c17bd19d3c3c889973644b852869c6fb12ee77"}
Jul 27 22:26:28.145293 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:28.145270    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6mrk7"
Jul 27 22:26:28.177589 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:28.177559    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/400fdac5-21e2-4f10-a1ec-a51e61b86869-kserve-provision-location\") pod \"400fdac5-21e2-4f10-a1ec-a51e61b86869\" (UID: \"400fdac5-21e2-4f10-a1ec-a51e61b86869\") "
Jul 27 22:26:28.177788 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:28.177614    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/400fdac5-21e2-4f10-a1ec-a51e61b86869-kube-rbac-proxy-sar-config\") pod \"400fdac5-21e2-4f10-a1ec-a51e61b86869\" (UID: \"400fdac5-21e2-4f10-a1ec-a51e61b86869\") "
Jul 27 22:26:28.177788 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:28.177676    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/400fdac5-21e2-4f10-a1ec-a51e61b86869-cabundle-cert\") pod \"400fdac5-21e2-4f10-a1ec-a51e61b86869\" (UID: \"400fdac5-21e2-4f10-a1ec-a51e61b86869\") "
Jul 27 22:26:28.177788 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:28.177715    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/400fdac5-21e2-4f10-a1ec-a51e61b86869-proxy-tls\") pod \"400fdac5-21e2-4f10-a1ec-a51e61b86869\" (UID: \"400fdac5-21e2-4f10-a1ec-a51e61b86869\") "
Jul 27 22:26:28.177788 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:28.177743    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jlgdz\" (UniqueName: \"kubernetes.io/projected/400fdac5-21e2-4f10-a1ec-a51e61b86869-kube-api-access-jlgdz\") pod \"400fdac5-21e2-4f10-a1ec-a51e61b86869\" (UID: \"400fdac5-21e2-4f10-a1ec-a51e61b86869\") "
Jul 27 22:26:28.178039 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:28.177965    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/400fdac5-21e2-4f10-a1ec-a51e61b86869-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "400fdac5-21e2-4f10-a1ec-a51e61b86869" (UID: "400fdac5-21e2-4f10-a1ec-a51e61b86869"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:26:28.178039 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:28.177986    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/400fdac5-21e2-4f10-a1ec-a51e61b86869-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "400fdac5-21e2-4f10-a1ec-a51e61b86869" (UID: "400fdac5-21e2-4f10-a1ec-a51e61b86869"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:26:28.178318 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:28.178287    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/400fdac5-21e2-4f10-a1ec-a51e61b86869-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "400fdac5-21e2-4f10-a1ec-a51e61b86869" (UID: "400fdac5-21e2-4f10-a1ec-a51e61b86869"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:26:28.179991 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:28.179961    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/400fdac5-21e2-4f10-a1ec-a51e61b86869-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "400fdac5-21e2-4f10-a1ec-a51e61b86869" (UID: "400fdac5-21e2-4f10-a1ec-a51e61b86869"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 22:26:28.179991 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:28.179985    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/400fdac5-21e2-4f10-a1ec-a51e61b86869-kube-api-access-jlgdz" (OuterVolumeSpecName: "kube-api-access-jlgdz") pod "400fdac5-21e2-4f10-a1ec-a51e61b86869" (UID: "400fdac5-21e2-4f10-a1ec-a51e61b86869"). InnerVolumeSpecName "kube-api-access-jlgdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:26:28.279219 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:28.279185    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/400fdac5-21e2-4f10-a1ec-a51e61b86869-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:26:28.279219 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:28.279212    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/400fdac5-21e2-4f10-a1ec-a51e61b86869-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:26:28.279219 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:28.279227    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/400fdac5-21e2-4f10-a1ec-a51e61b86869-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:26:28.279628 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:28.279241    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/400fdac5-21e2-4f10-a1ec-a51e61b86869-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:26:28.279628 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:28.279253    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlgdz\" (UniqueName: \"kubernetes.io/projected/400fdac5-21e2-4f10-a1ec-a51e61b86869-kube-api-access-jlgdz\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:26:29.064412 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:29.064373    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9drb4" event={"ID":"ee02286a-40e4-414d-8b03-da6fef2d8e0a","Type":"ContainerStarted","Data":"8b92bddfa67a5fbf2d28c8c62e2567cbe52cbbf024ba2ec3ef38f464a1bcc1d2"}
Jul 27 22:26:29.066111 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:29.066078    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6mrk7" event={"ID":"400fdac5-21e2-4f10-a1ec-a51e61b86869","Type":"ContainerDied","Data":"6a835ff90373e66f3b9b06e85f105bf5959e0722c1c46f6d081910c1c350c57e"}
Jul 27 22:26:29.066249 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:29.066131    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6mrk7"
Jul 27 22:26:29.066249 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:29.066155    2577 scope.go:117] "RemoveContainer" containerID="53b07b634936ccfea16c240cc9eaed2674c26471ea1ec02935d8667f21c6a6c5"
Jul 27 22:26:29.074533 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:29.074518    2577 scope.go:117] "RemoveContainer" containerID="14b5c3057f5e7c4d5aa5e4de074b4b27e280c1581a69bf71a77206137bbc8b90"
Jul 27 22:26:29.081476 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:29.081455    2577 scope.go:117] "RemoveContainer" containerID="56a450ac5822a6357aa6812ee67df2f591c8f21cf05637a6c3b896283f4f9d37"
Jul 27 22:26:29.096680 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:29.096659    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6mrk7"]
Jul 27 22:26:29.100384 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:29.100365    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-6mrk7"]
Jul 27 22:26:29.135053 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:29.135030    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="400fdac5-21e2-4f10-a1ec-a51e61b86869" path="/var/lib/kubelet/pods/400fdac5-21e2-4f10-a1ec-a51e61b86869/volumes"
Jul 27 22:26:32.080858 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:32.080758    2577 generic.go:356] "Generic (PLEG): container finished" podID="ee02286a-40e4-414d-8b03-da6fef2d8e0a" containerID="8b92bddfa67a5fbf2d28c8c62e2567cbe52cbbf024ba2ec3ef38f464a1bcc1d2" exitCode=0
Jul 27 22:26:32.080858 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:32.080832    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9drb4" event={"ID":"ee02286a-40e4-414d-8b03-da6fef2d8e0a","Type":"ContainerDied","Data":"8b92bddfa67a5fbf2d28c8c62e2567cbe52cbbf024ba2ec3ef38f464a1bcc1d2"}
Jul 27 22:26:33.092541 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:33.092506    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9drb4" event={"ID":"ee02286a-40e4-414d-8b03-da6fef2d8e0a","Type":"ContainerStarted","Data":"78dd2cf593cc7f11217be17996f680682a7e1e86fd0b39b6e1090ca6721edfd7"}
Jul 27 22:26:33.092961 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:33.092589    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9drb4" event={"ID":"ee02286a-40e4-414d-8b03-da6fef2d8e0a","Type":"ContainerStarted","Data":"c843dade7c585a6bbbf8bf4d7a91321ca3edb56c1af1069edf41decee1748dae"}
Jul 27 22:26:33.092961 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:33.092810    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9drb4"
Jul 27 22:26:33.093096 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:33.092961    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9drb4"
Jul 27 22:26:33.114287 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:33.114235    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9drb4" podStartSLOduration=2.423961261 podStartE2EDuration="6.11422219s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.690260929s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:26:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:26:33.112501677 +0000 UTC m=+4916.479691545" watchObservedRunningTime="2026-07-27 22:26:33.11422219 +0000 UTC m=+4916.481412056"
Jul 27 22:26:39.101625 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:26:39.101587    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9drb4"
Jul 27 22:27:09.105603 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:27:09.105524    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9drb4"
Jul 27 22:27:17.347422 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:27:17.347388    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9drb4"]
Jul 27 22:27:17.348004 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:27:17.347676    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9drb4" podUID="ee02286a-40e4-414d-8b03-da6fef2d8e0a" containerName="kserve-container" containerID="cri-o://c843dade7c585a6bbbf8bf4d7a91321ca3edb56c1af1069edf41decee1748dae" gracePeriod=30
Jul 27 22:27:17.348004 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:27:17.347736    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9drb4" podUID="ee02286a-40e4-414d-8b03-da6fef2d8e0a" containerName="kube-rbac-proxy" containerID="cri-o://78dd2cf593cc7f11217be17996f680682a7e1e86fd0b39b6e1090ca6721edfd7" gracePeriod=30
Jul 27 22:27:17.365816 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:27:17.365790    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:27:18.266227 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:27:18.266191    2577 generic.go:356] "Generic (PLEG): container finished" podID="ee02286a-40e4-414d-8b03-da6fef2d8e0a" containerID="78dd2cf593cc7f11217be17996f680682a7e1e86fd0b39b6e1090ca6721edfd7" exitCode=2
Jul 27 22:27:18.266429 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:27:18.266271    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9drb4" event={"ID":"ee02286a-40e4-414d-8b03-da6fef2d8e0a","Type":"ContainerDied","Data":"78dd2cf593cc7f11217be17996f680682a7e1e86fd0b39b6e1090ca6721edfd7"}
Jul 27 22:27:19.097513 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:27:19.097470    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9drb4" podUID="ee02286a-40e4-414d-8b03-da6fef2d8e0a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.69:8643/healthz\": dial tcp 10.133.0.69:8643: connect: connection refused"
Jul 27 22:27:22.574485 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:27:22.574453    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:27:22.785443 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:27:22.785407    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:27:23.682670 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:27:23.682648    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9drb4"
Jul 27 22:27:23.724968 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:27:23.724937    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee02286a-40e4-414d-8b03-da6fef2d8e0a-kserve-provision-location\") pod \"ee02286a-40e4-414d-8b03-da6fef2d8e0a\" (UID: \"ee02286a-40e4-414d-8b03-da6fef2d8e0a\") "
Jul 27 22:27:23.725097 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:27:23.724994    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee02286a-40e4-414d-8b03-da6fef2d8e0a-kube-rbac-proxy-sar-config\") pod \"ee02286a-40e4-414d-8b03-da6fef2d8e0a\" (UID: \"ee02286a-40e4-414d-8b03-da6fef2d8e0a\") "
Jul 27 22:27:23.725097 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:27:23.725026    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-84wkb\" (UniqueName: \"kubernetes.io/projected/ee02286a-40e4-414d-8b03-da6fef2d8e0a-kube-api-access-84wkb\") pod \"ee02286a-40e4-414d-8b03-da6fef2d8e0a\" (UID: \"ee02286a-40e4-414d-8b03-da6fef2d8e0a\") "
Jul 27 22:27:23.725097 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:27:23.725053    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee02286a-40e4-414d-8b03-da6fef2d8e0a-cabundle-cert\") pod \"ee02286a-40e4-414d-8b03-da6fef2d8e0a\" (UID: \"ee02286a-40e4-414d-8b03-da6fef2d8e0a\") "
Jul 27 22:27:23.725097 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:27:23.725082    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee02286a-40e4-414d-8b03-da6fef2d8e0a-proxy-tls\") pod \"ee02286a-40e4-414d-8b03-da6fef2d8e0a\" (UID: \"ee02286a-40e4-414d-8b03-da6fef2d8e0a\") "
Jul 27 22:27:23.725355 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:27:23.725247    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee02286a-40e4-414d-8b03-da6fef2d8e0a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ee02286a-40e4-414d-8b03-da6fef2d8e0a" (UID: "ee02286a-40e4-414d-8b03-da6fef2d8e0a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:27:23.725355 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:27:23.725350    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee02286a-40e4-414d-8b03-da6fef2d8e0a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ee02286a-40e4-414d-8b03-da6fef2d8e0a" (UID: "ee02286a-40e4-414d-8b03-da6fef2d8e0a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:27:23.725466 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:27:23.725433    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee02286a-40e4-414d-8b03-da6fef2d8e0a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ee02286a-40e4-414d-8b03-da6fef2d8e0a" (UID: "ee02286a-40e4-414d-8b03-da6fef2d8e0a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:27:23.727174 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:27:23.727144    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee02286a-40e4-414d-8b03-da6fef2d8e0a-kube-api-access-84wkb" (OuterVolumeSpecName: "kube-api-access-84wkb") pod "ee02286a-40e4-414d-8b03-da6fef2d8e0a" (UID: "ee02286a-40e4-414d-8b03-da6fef2d8e0a"). InnerVolumeSpecName "kube-api-access-84wkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:27:23.727267 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:27:23.727148    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee02286a-40e4-414d-8b03-da6fef2d8e0a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ee02286a-40e4-414d-8b03-da6fef2d8e0a" (UID: "ee02286a-40e4-414d-8b03-da6fef2d8e0a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 22:27:23.826604 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:27:23.826533    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee02286a-40e4-414d-8b03-da6fef2d8e0a-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:27:23.826604 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:27:23.826557    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee02286a-40e4-414d-8b03-da6fef2d8e0a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:27:23.826604 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:27:23.826566    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84wkb\" (UniqueName: \"kubernetes.io/projected/ee02286a-40e4-414d-8b03-da6fef2d8e0a-kube-api-access-84wkb\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:27:23.826604 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:27:23.826575    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee02286a-40e4-414d-8b03-da6fef2d8e0a-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:27:23.826604 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:27:23.826584    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee02286a-40e4-414d-8b03-da6fef2d8e0a-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:27:24.294571 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:27:24.294537    2577 generic.go:356] "Generic (PLEG): container finished" podID="ee02286a-40e4-414d-8b03-da6fef2d8e0a" containerID="c843dade7c585a6bbbf8bf4d7a91321ca3edb56c1af1069edf41decee1748dae" exitCode=0
Jul 27 22:27:24.294898 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:27:24.294603    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9drb4"
Jul 27 22:27:24.294898 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:27:24.294626    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9drb4" event={"ID":"ee02286a-40e4-414d-8b03-da6fef2d8e0a","Type":"ContainerDied","Data":"c843dade7c585a6bbbf8bf4d7a91321ca3edb56c1af1069edf41decee1748dae"}
Jul 27 22:27:24.294898 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:27:24.294691    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9drb4" event={"ID":"ee02286a-40e4-414d-8b03-da6fef2d8e0a","Type":"ContainerDied","Data":"251c0d034cc563b114b219e971c17bd19d3c3c889973644b852869c6fb12ee77"}
Jul 27 22:27:24.294898 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:27:24.294750    2577 scope.go:117] "RemoveContainer" containerID="78dd2cf593cc7f11217be17996f680682a7e1e86fd0b39b6e1090ca6721edfd7"
Jul 27 22:27:24.303684 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:27:24.303667    2577 scope.go:117] "RemoveContainer" containerID="c843dade7c585a6bbbf8bf4d7a91321ca3edb56c1af1069edf41decee1748dae"
Jul 27 22:27:24.310835 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:27:24.310817    2577 scope.go:117] "RemoveContainer" containerID="8b92bddfa67a5fbf2d28c8c62e2567cbe52cbbf024ba2ec3ef38f464a1bcc1d2"
Jul 27 22:27:24.318265 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:27:24.318245    2577 scope.go:117] "RemoveContainer" containerID="78dd2cf593cc7f11217be17996f680682a7e1e86fd0b39b6e1090ca6721edfd7"
Jul 27 22:27:24.318544 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:27:24.318519    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78dd2cf593cc7f11217be17996f680682a7e1e86fd0b39b6e1090ca6721edfd7\": container with ID starting with 78dd2cf593cc7f11217be17996f680682a7e1e86fd0b39b6e1090ca6721edfd7 not found: ID does not exist" containerID="78dd2cf593cc7f11217be17996f680682a7e1e86fd0b39b6e1090ca6721edfd7"
Jul 27 22:27:24.318618 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:27:24.318557    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78dd2cf593cc7f11217be17996f680682a7e1e86fd0b39b6e1090ca6721edfd7"} err="failed to get container status \"78dd2cf593cc7f11217be17996f680682a7e1e86fd0b39b6e1090ca6721edfd7\": rpc error: code = NotFound desc = could not find container \"78dd2cf593cc7f11217be17996f680682a7e1e86fd0b39b6e1090ca6721edfd7\": container with ID starting with 78dd2cf593cc7f11217be17996f680682a7e1e86fd0b39b6e1090ca6721edfd7 not found: ID does not exist"
Jul 27 22:27:24.318618 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:27:24.318575    2577 scope.go:117] "RemoveContainer" containerID="c843dade7c585a6bbbf8bf4d7a91321ca3edb56c1af1069edf41decee1748dae"
Jul 27 22:27:24.318841 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:27:24.318811    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c843dade7c585a6bbbf8bf4d7a91321ca3edb56c1af1069edf41decee1748dae\": container with ID starting with c843dade7c585a6bbbf8bf4d7a91321ca3edb56c1af1069edf41decee1748dae not found: ID does not exist" containerID="c843dade7c585a6bbbf8bf4d7a91321ca3edb56c1af1069edf41decee1748dae"
Jul 27 22:27:24.318973 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:27:24.318842    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c843dade7c585a6bbbf8bf4d7a91321ca3edb56c1af1069edf41decee1748dae"} err="failed to get container status \"c843dade7c585a6bbbf8bf4d7a91321ca3edb56c1af1069edf41decee1748dae\": rpc error: code = NotFound desc = could not find container \"c843dade7c585a6bbbf8bf4d7a91321ca3edb56c1af1069edf41decee1748dae\": container with ID starting with c843dade7c585a6bbbf8bf4d7a91321ca3edb56c1af1069edf41decee1748dae not found: ID does not exist"
Jul 27 22:27:24.318973 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:27:24.318874    2577 scope.go:117] "RemoveContainer" containerID="8b92bddfa67a5fbf2d28c8c62e2567cbe52cbbf024ba2ec3ef38f464a1bcc1d2"
Jul 27 22:27:24.319136 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:27:24.319118    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b92bddfa67a5fbf2d28c8c62e2567cbe52cbbf024ba2ec3ef38f464a1bcc1d2\": container with ID starting with 8b92bddfa67a5fbf2d28c8c62e2567cbe52cbbf024ba2ec3ef38f464a1bcc1d2 not found: ID does not exist" containerID="8b92bddfa67a5fbf2d28c8c62e2567cbe52cbbf024ba2ec3ef38f464a1bcc1d2"
Jul 27 22:27:24.319191 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:27:24.319145    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b92bddfa67a5fbf2d28c8c62e2567cbe52cbbf024ba2ec3ef38f464a1bcc1d2"} err="failed to get container status \"8b92bddfa67a5fbf2d28c8c62e2567cbe52cbbf024ba2ec3ef38f464a1bcc1d2\": rpc error: code = NotFound desc = could not find container \"8b92bddfa67a5fbf2d28c8c62e2567cbe52cbbf024ba2ec3ef38f464a1bcc1d2\": container with ID starting with 8b92bddfa67a5fbf2d28c8c62e2567cbe52cbbf024ba2ec3ef38f464a1bcc1d2 not found: ID does not exist"
Jul 27 22:27:24.319469 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:27:24.319451    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9drb4"]
Jul 27 22:27:24.324254 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:27:24.324230    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9drb4"]
Jul 27 22:27:25.134881 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:27:25.134830    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee02286a-40e4-414d-8b03-da6fef2d8e0a" path="/var/lib/kubelet/pods/ee02286a-40e4-414d-8b03-da6fef2d8e0a/volumes"
Jul 27 22:29:02.700091 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:29:02.700055    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:29:07.790071 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:29:07.790037    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:29:07.849822 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:29:07.849786    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86"]
Jul 27 22:29:07.850228 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:29:07.850209    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee02286a-40e4-414d-8b03-da6fef2d8e0a" containerName="kube-rbac-proxy"
Jul 27 22:29:07.850275 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:29:07.850231    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee02286a-40e4-414d-8b03-da6fef2d8e0a" containerName="kube-rbac-proxy"
Jul 27 22:29:07.850275 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:29:07.850241    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee02286a-40e4-414d-8b03-da6fef2d8e0a" containerName="storage-initializer"
Jul 27 22:29:07.850275 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:29:07.850246    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee02286a-40e4-414d-8b03-da6fef2d8e0a" containerName="storage-initializer"
Jul 27 22:29:07.850275 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:29:07.850266    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="400fdac5-21e2-4f10-a1ec-a51e61b86869" containerName="storage-initializer"
Jul 27 22:29:07.850275 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:29:07.850272    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="400fdac5-21e2-4f10-a1ec-a51e61b86869" containerName="storage-initializer"
Jul 27 22:29:07.850431 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:29:07.850281    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="400fdac5-21e2-4f10-a1ec-a51e61b86869" containerName="kserve-container"
Jul 27 22:29:07.850431 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:29:07.850286    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="400fdac5-21e2-4f10-a1ec-a51e61b86869" containerName="kserve-container"
Jul 27 22:29:07.850431 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:29:07.850292    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee02286a-40e4-414d-8b03-da6fef2d8e0a" containerName="kserve-container"
Jul 27 22:29:07.850431 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:29:07.850298    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee02286a-40e4-414d-8b03-da6fef2d8e0a" containerName="kserve-container"
Jul 27 22:29:07.850431 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:29:07.850305    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="400fdac5-21e2-4f10-a1ec-a51e61b86869" containerName="kube-rbac-proxy"
Jul 27 22:29:07.850431 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:29:07.850310    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="400fdac5-21e2-4f10-a1ec-a51e61b86869" containerName="kube-rbac-proxy"
Jul 27 22:29:07.850431 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:29:07.850356    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="400fdac5-21e2-4f10-a1ec-a51e61b86869" containerName="kube-rbac-proxy"
Jul 27 22:29:07.850431 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:29:07.850362    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee02286a-40e4-414d-8b03-da6fef2d8e0a" containerName="kube-rbac-proxy"
Jul 27 22:29:07.850431 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:29:07.850371    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee02286a-40e4-414d-8b03-da6fef2d8e0a" containerName="kserve-container"
Jul 27 22:29:07.850431 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:29:07.850377    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="400fdac5-21e2-4f10-a1ec-a51e61b86869" containerName="kserve-container"
Jul 27 22:29:07.853381 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:29:07.853365    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86"
Jul 27 22:29:07.855982 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:29:07.855937    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
Jul 27 22:29:07.855982 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:29:07.855973    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 22:29:07.856181 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:29:07.855955    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v7c66\""
Jul 27 22:29:07.856181 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:29:07.855944    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 22:29:07.856814 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:29:07.856797    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 27 22:29:07.856919 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:29:07.856826    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 22:29:07.856919 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:29:07.856836    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 22:29:07.863477 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:29:07.863457    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86"]
Jul 27 22:29:07.925228 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:29:07.925199    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86\" (UID: \"2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86"
Jul 27 22:29:07.925353 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:29:07.925245    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff4n9\" (UniqueName: \"kubernetes.io/projected/2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b-kube-api-access-ff4n9\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86\" (UID: \"2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86"
Jul 27 22:29:07.925353 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:29:07.925320    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86\" (UID: \"2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86"
Jul 27 22:29:07.925439 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:29:07.925354    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86\" (UID: \"2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86"
Jul 27 22:29:07.925439 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:29:07.925383    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86\" (UID: \"2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86"
Jul 27 22:29:07.936281 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:29:07.936246    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:29:08.026638 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:29:08.026603    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86\" (UID: \"2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86"
Jul 27 22:29:08.026800 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:29:08.026650    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86\" (UID: \"2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86"
Jul 27 22:29:08.026865 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:29:08.026792    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86\" (UID: \"2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86"
Jul 27 22:29:08.026865 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:29:08.026851    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86\" (UID: \"2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86"
Jul 27 22:29:08.026946 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:29:08.026901    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff4n9\" (UniqueName: \"kubernetes.io/projected/2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b-kube-api-access-ff4n9\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86\" (UID: \"2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86"
Jul 27 22:29:08.027063 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:29:08.027046    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86\" (UID: \"2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86"
Jul 27 22:29:08.027402 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:29:08.027376    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86\" (UID: \"2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86"
Jul 27 22:29:08.027540 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:29:08.027475    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86\" (UID: \"2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86"
Jul 27 22:29:08.029349 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:29:08.029329    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86\" (UID: \"2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86"
Jul 27 22:29:08.035828 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:29:08.035808    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff4n9\" (UniqueName: \"kubernetes.io/projected/2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b-kube-api-access-ff4n9\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86\" (UID: \"2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86"
Jul 27 22:29:08.164290 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:29:08.164217    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86"
Jul 27 22:29:08.294382 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:29:08.294359    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86"]
Jul 27 22:29:08.296418 ip-10-0-142-219 kubenswrapper[2577]: W0727 22:29:08.296394    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fd08e6c_8600_41fd_9ea2_bac7b8b9f33b.slice/crio-2f5586421344b633a8521022211dad133ee46a1998571ee4e0e0d415b7d79475 WatchSource:0}: Error finding container 2f5586421344b633a8521022211dad133ee46a1998571ee4e0e0d415b7d79475: Status 404 returned error can't find the container with id 2f5586421344b633a8521022211dad133ee46a1998571ee4e0e0d415b7d79475
Jul 27 22:29:08.693348 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:29:08.693310    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86" event={"ID":"2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b","Type":"ContainerStarted","Data":"03a9a9b41972e10ee1abce1b2f1c1a85904b9c9ba0a29c2d4368ab1ba0419370"}
Jul 27 22:29:08.693540 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:29:08.693377    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86" event={"ID":"2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b","Type":"ContainerStarted","Data":"2f5586421344b633a8521022211dad133ee46a1998571ee4e0e0d415b7d79475"}
Jul 27 22:29:12.715814 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:29:12.715755    2577 generic.go:356] "Generic (PLEG): container finished" podID="2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b" containerID="03a9a9b41972e10ee1abce1b2f1c1a85904b9c9ba0a29c2d4368ab1ba0419370" exitCode=0
Jul 27 22:29:12.715814 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:29:12.715816    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86" event={"ID":"2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b","Type":"ContainerDied","Data":"03a9a9b41972e10ee1abce1b2f1c1a85904b9c9ba0a29c2d4368ab1ba0419370"}
Jul 27 22:29:13.721604 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:29:13.721569    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86" event={"ID":"2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b","Type":"ContainerStarted","Data":"64b6fd67ce29b9dce0115cbf824c7635e08e3e50f28b9a8cd8712cd29db5e93d"}
Jul 27 22:29:13.722062 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:29:13.721639    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86" event={"ID":"2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b","Type":"ContainerStarted","Data":"b69d31fc0433e1a2e56022634f9fcdbfe6256a0b275dfbe9c370e603171fc6ff"}
Jul 27 22:29:13.722062 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:29:13.721862    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86"
Jul 27 22:29:13.722062 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:29:13.721906    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86"
Jul 27 22:29:13.742224 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:29:13.742181    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86" podStartSLOduration=2.983520652 podStartE2EDuration="6.742167663s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.758647011s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:29:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:29:13.740670047 +0000 UTC m=+5077.107859912" watchObservedRunningTime="2026-07-27 22:29:13.742167663 +0000 UTC m=+5077.109357529"
Jul 27 22:29:19.733156 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:29:19.733127    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86"
Jul 27 22:29:49.810200 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:29:49.810148    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86" podUID="2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Jul 27 22:29:59.736047 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:29:59.735970    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86"
Jul 27 22:30:07.889725 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:30:07.889690    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86"]
Jul 27 22:30:07.890153 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:30:07.889995    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86" podUID="2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b" containerName="kserve-container" containerID="cri-o://b69d31fc0433e1a2e56022634f9fcdbfe6256a0b275dfbe9c370e603171fc6ff" gracePeriod=30
Jul 27 22:30:07.890153 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:30:07.890037    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86" podUID="2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b" containerName="kube-rbac-proxy" containerID="cri-o://64b6fd67ce29b9dce0115cbf824c7635e08e3e50f28b9a8cd8712cd29db5e93d" gracePeriod=30
Jul 27 22:30:08.666333 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:30:08.666299    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:30:08.936937 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:30:08.936905    2577 generic.go:356] "Generic (PLEG): container finished" podID="2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b" containerID="64b6fd67ce29b9dce0115cbf824c7635e08e3e50f28b9a8cd8712cd29db5e93d" exitCode=2
Jul 27 22:30:08.937296 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:30:08.936944    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86" event={"ID":"2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b","Type":"ContainerDied","Data":"64b6fd67ce29b9dce0115cbf824c7635e08e3e50f28b9a8cd8712cd29db5e93d"}
Jul 27 22:30:09.728640 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:30:09.728596    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86" podUID="2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.70:8643/healthz\": dial tcp 10.133.0.70:8643: connect: connection refused"
Jul 27 22:30:10.775940 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:30:10.775890    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86" podUID="2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.70:8080/v2/models/isvc-xgboost-v2-runtime/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 27 22:30:12.993879 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:30:12.993841    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:30:13.336468 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:30:13.336385    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:30:14.728439 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:30:14.728404    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86" podUID="2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.70:8643/healthz\": dial tcp 10.133.0.70:8643: connect: connection refused"
Jul 27 22:30:14.932944 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:30:14.932922    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86"
Jul 27 22:30:14.962885 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:30:14.962821    2577 generic.go:356] "Generic (PLEG): container finished" podID="2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b" containerID="b69d31fc0433e1a2e56022634f9fcdbfe6256a0b275dfbe9c370e603171fc6ff" exitCode=0
Jul 27 22:30:14.962885 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:30:14.962865    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86" event={"ID":"2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b","Type":"ContainerDied","Data":"b69d31fc0433e1a2e56022634f9fcdbfe6256a0b275dfbe9c370e603171fc6ff"}
Jul 27 22:30:14.962885 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:30:14.962871    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86"
Jul 27 22:30:14.963086 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:30:14.962912    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86" event={"ID":"2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b","Type":"ContainerDied","Data":"2f5586421344b633a8521022211dad133ee46a1998571ee4e0e0d415b7d79475"}
Jul 27 22:30:14.963086 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:30:14.962938    2577 scope.go:117] "RemoveContainer" containerID="64b6fd67ce29b9dce0115cbf824c7635e08e3e50f28b9a8cd8712cd29db5e93d"
Jul 27 22:30:14.970707 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:30:14.970688    2577 scope.go:117] "RemoveContainer" containerID="b69d31fc0433e1a2e56022634f9fcdbfe6256a0b275dfbe9c370e603171fc6ff"
Jul 27 22:30:14.977604 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:30:14.977588    2577 scope.go:117] "RemoveContainer" containerID="03a9a9b41972e10ee1abce1b2f1c1a85904b9c9ba0a29c2d4368ab1ba0419370"
Jul 27 22:30:14.984158 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:30:14.984143    2577 scope.go:117] "RemoveContainer" containerID="64b6fd67ce29b9dce0115cbf824c7635e08e3e50f28b9a8cd8712cd29db5e93d"
Jul 27 22:30:14.984385 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:30:14.984369    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64b6fd67ce29b9dce0115cbf824c7635e08e3e50f28b9a8cd8712cd29db5e93d\": container with ID starting with 64b6fd67ce29b9dce0115cbf824c7635e08e3e50f28b9a8cd8712cd29db5e93d not found: ID does not exist" containerID="64b6fd67ce29b9dce0115cbf824c7635e08e3e50f28b9a8cd8712cd29db5e93d"
Jul 27 22:30:14.984436 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:30:14.984396    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64b6fd67ce29b9dce0115cbf824c7635e08e3e50f28b9a8cd8712cd29db5e93d"} err="failed to get container status \"64b6fd67ce29b9dce0115cbf824c7635e08e3e50f28b9a8cd8712cd29db5e93d\": rpc error: code = NotFound desc = could not find container \"64b6fd67ce29b9dce0115cbf824c7635e08e3e50f28b9a8cd8712cd29db5e93d\": container with ID starting with 64b6fd67ce29b9dce0115cbf824c7635e08e3e50f28b9a8cd8712cd29db5e93d not found: ID does not exist"
Jul 27 22:30:14.984436 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:30:14.984411    2577 scope.go:117] "RemoveContainer" containerID="b69d31fc0433e1a2e56022634f9fcdbfe6256a0b275dfbe9c370e603171fc6ff"
Jul 27 22:30:14.984591 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:30:14.984577    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b69d31fc0433e1a2e56022634f9fcdbfe6256a0b275dfbe9c370e603171fc6ff\": container with ID starting with b69d31fc0433e1a2e56022634f9fcdbfe6256a0b275dfbe9c370e603171fc6ff not found: ID does not exist" containerID="b69d31fc0433e1a2e56022634f9fcdbfe6256a0b275dfbe9c370e603171fc6ff"
Jul 27 22:30:14.984629 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:30:14.984596    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b69d31fc0433e1a2e56022634f9fcdbfe6256a0b275dfbe9c370e603171fc6ff"} err="failed to get container status \"b69d31fc0433e1a2e56022634f9fcdbfe6256a0b275dfbe9c370e603171fc6ff\": rpc error: code = NotFound desc = could not find container \"b69d31fc0433e1a2e56022634f9fcdbfe6256a0b275dfbe9c370e603171fc6ff\": container with ID starting with b69d31fc0433e1a2e56022634f9fcdbfe6256a0b275dfbe9c370e603171fc6ff not found: ID does not exist"
Jul 27 22:30:14.984629 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:30:14.984609    2577 scope.go:117] "RemoveContainer" containerID="03a9a9b41972e10ee1abce1b2f1c1a85904b9c9ba0a29c2d4368ab1ba0419370"
Jul 27 22:30:14.984822 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:30:14.984802    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03a9a9b41972e10ee1abce1b2f1c1a85904b9c9ba0a29c2d4368ab1ba0419370\": container with ID starting with 03a9a9b41972e10ee1abce1b2f1c1a85904b9c9ba0a29c2d4368ab1ba0419370 not found: ID does not exist" containerID="03a9a9b41972e10ee1abce1b2f1c1a85904b9c9ba0a29c2d4368ab1ba0419370"
Jul 27 22:30:14.984870 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:30:14.984830    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03a9a9b41972e10ee1abce1b2f1c1a85904b9c9ba0a29c2d4368ab1ba0419370"} err="failed to get container status \"03a9a9b41972e10ee1abce1b2f1c1a85904b9c9ba0a29c2d4368ab1ba0419370\": rpc error: code = NotFound desc = could not find container \"03a9a9b41972e10ee1abce1b2f1c1a85904b9c9ba0a29c2d4368ab1ba0419370\": container with ID starting with 03a9a9b41972e10ee1abce1b2f1c1a85904b9c9ba0a29c2d4368ab1ba0419370 not found: ID does not exist"
Jul 27 22:30:15.072399 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:30:15.072365    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b-cabundle-cert\") pod \"2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b\" (UID: \"2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b\") "
Jul 27 22:30:15.072523 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:30:15.072409    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b-kserve-provision-location\") pod \"2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b\" (UID: \"2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b\") "
Jul 27 22:30:15.072523 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:30:15.072436    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b-kube-rbac-proxy-sar-config\") pod \"2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b\" (UID: \"2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b\") "
Jul 27 22:30:15.072523 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:30:15.072461    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b-proxy-tls\") pod \"2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b\" (UID: \"2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b\") "
Jul 27 22:30:15.072523 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:30:15.072501    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ff4n9\" (UniqueName: \"kubernetes.io/projected/2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b-kube-api-access-ff4n9\") pod \"2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b\" (UID: \"2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b\") "
Jul 27 22:30:15.072753 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:30:15.072729    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b" (UID: "2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:30:15.072820 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:30:15.072795    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b" (UID: "2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:30:15.072862 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:30:15.072799    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b" (UID: "2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:30:15.074561 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:30:15.074536    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b-kube-api-access-ff4n9" (OuterVolumeSpecName: "kube-api-access-ff4n9") pod "2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b" (UID: "2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b"). InnerVolumeSpecName "kube-api-access-ff4n9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:30:15.074659 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:30:15.074569    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b" (UID: "2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 22:30:15.173459 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:30:15.173438    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ff4n9\" (UniqueName: \"kubernetes.io/projected/2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b-kube-api-access-ff4n9\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:30:15.173459 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:30:15.173460    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:30:15.173601 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:30:15.173469    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:30:15.173601 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:30:15.173477    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:30:15.173601 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:30:15.173486    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:30:15.279111 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:30:15.279046    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86"]
Jul 27 22:30:15.282336 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:30:15.282312    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-ztx86"]
Jul 27 22:30:17.134820 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:30:17.134756    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b" path="/var/lib/kubelet/pods/2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b/volumes"
Jul 27 22:30:21.157192 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:30:21.157162    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:30:32.297825 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:30:32.297785    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:31:33.145484 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:31:33.145405    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:31:38.312232 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:31:38.312196    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:31:38.526087 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:31:38.526051    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:32:58.422258 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:32:58.422220    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:32:58.529167 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:32:58.529126    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:32:58.810472 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:32:58.810367    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:34:18.631534 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:34:18.631493    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:34:19.901240 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:34:19.901202    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:34:20.114290 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:34:20.114262    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:34:29.680639 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:34:29.680515    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:34:30.730282 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:34:30.730241    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:34:30.750679 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:34:30.750652    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt"]
Jul 27 22:34:30.751012 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:34:30.750996    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b" containerName="storage-initializer"
Jul 27 22:34:30.751106 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:34:30.751015    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b" containerName="storage-initializer"
Jul 27 22:34:30.751106 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:34:30.751030    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b" containerName="kserve-container"
Jul 27 22:34:30.751106 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:34:30.751038    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b" containerName="kserve-container"
Jul 27 22:34:30.751106 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:34:30.751053    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b" containerName="kube-rbac-proxy"
Jul 27 22:34:30.751106 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:34:30.751061    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b" containerName="kube-rbac-proxy"
Jul 27 22:34:30.751367 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:34:30.751153    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b" containerName="kserve-container"
Jul 27 22:34:30.751367 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:34:30.751167    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fd08e6c-8600-41fd-9ea2-bac7b8b9f33b" containerName="kube-rbac-proxy"
Jul 27 22:34:30.754271 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:34:30.754252    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt"
Jul 27 22:34:30.756610 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:34:30.756586    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v7c66\""
Jul 27 22:34:30.756610 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:34:30.756595    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Jul 27 22:34:30.756834 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:34:30.756820    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 22:34:30.757085 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:34:30.757062    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 22:34:30.757151 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:34:30.757123    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
Jul 27 22:34:30.757151 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:34:30.757131    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 22:34:30.757260 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:34:30.757161    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
Jul 27 22:34:30.764170 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:34:30.764149    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt"]
Jul 27 22:34:30.830625 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:34:30.830597    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21347091-b8fa-49e4-9bcd-413f7ef30d57-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt\" (UID: \"21347091-b8fa-49e4-9bcd-413f7ef30d57\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt"
Jul 27 22:34:30.830754 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:34:30.830650    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21347091-b8fa-49e4-9bcd-413f7ef30d57-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt\" (UID: \"21347091-b8fa-49e4-9bcd-413f7ef30d57\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt"
Jul 27 22:34:30.830754 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:34:30.830667    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq2ff\" (UniqueName: \"kubernetes.io/projected/21347091-b8fa-49e4-9bcd-413f7ef30d57-kube-api-access-fq2ff\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt\" (UID: \"21347091-b8fa-49e4-9bcd-413f7ef30d57\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt"
Jul 27 22:34:30.830856 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:34:30.830787    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21347091-b8fa-49e4-9bcd-413f7ef30d57-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt\" (UID: \"21347091-b8fa-49e4-9bcd-413f7ef30d57\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt"
Jul 27 22:34:30.830856 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:34:30.830812    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21347091-b8fa-49e4-9bcd-413f7ef30d57-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt\" (UID: \"21347091-b8fa-49e4-9bcd-413f7ef30d57\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt"
Jul 27 22:34:30.931346 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:34:30.931318    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21347091-b8fa-49e4-9bcd-413f7ef30d57-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt\" (UID: \"21347091-b8fa-49e4-9bcd-413f7ef30d57\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt"
Jul 27 22:34:30.931541 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:34:30.931371    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21347091-b8fa-49e4-9bcd-413f7ef30d57-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt\" (UID: \"21347091-b8fa-49e4-9bcd-413f7ef30d57\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt"
Jul 27 22:34:30.931541 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:34:30.931389    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq2ff\" (UniqueName: \"kubernetes.io/projected/21347091-b8fa-49e4-9bcd-413f7ef30d57-kube-api-access-fq2ff\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt\" (UID: \"21347091-b8fa-49e4-9bcd-413f7ef30d57\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt"
Jul 27 22:34:30.931541 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:34:30.931424    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21347091-b8fa-49e4-9bcd-413f7ef30d57-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt\" (UID: \"21347091-b8fa-49e4-9bcd-413f7ef30d57\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt"
Jul 27 22:34:30.931541 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:34:30.931443    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21347091-b8fa-49e4-9bcd-413f7ef30d57-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt\" (UID: \"21347091-b8fa-49e4-9bcd-413f7ef30d57\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt"
Jul 27 22:34:30.931541 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:34:30.931484    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found
Jul 27 22:34:30.931864 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:34:30.931567    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21347091-b8fa-49e4-9bcd-413f7ef30d57-proxy-tls podName:21347091-b8fa-49e4-9bcd-413f7ef30d57 nodeName:}" failed. No retries permitted until 2026-07-27 22:34:31.43154621 +0000 UTC m=+5394.798736068 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/21347091-b8fa-49e4-9bcd-413f7ef30d57-proxy-tls") pod "isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt" (UID: "21347091-b8fa-49e4-9bcd-413f7ef30d57") : secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found
Jul 27 22:34:30.931933 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:34:30.931905    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21347091-b8fa-49e4-9bcd-413f7ef30d57-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt\" (UID: \"21347091-b8fa-49e4-9bcd-413f7ef30d57\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt"
Jul 27 22:34:30.932113 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:34:30.932095    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21347091-b8fa-49e4-9bcd-413f7ef30d57-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt\" (UID: \"21347091-b8fa-49e4-9bcd-413f7ef30d57\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt"
Jul 27 22:34:30.932113 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:34:30.932102    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21347091-b8fa-49e4-9bcd-413f7ef30d57-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt\" (UID: \"21347091-b8fa-49e4-9bcd-413f7ef30d57\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt"
Jul 27 22:34:30.942401 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:34:30.942380    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq2ff\" (UniqueName: \"kubernetes.io/projected/21347091-b8fa-49e4-9bcd-413f7ef30d57-kube-api-access-fq2ff\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt\" (UID: \"21347091-b8fa-49e4-9bcd-413f7ef30d57\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt"
Jul 27 22:34:31.296424 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:34:31.296388    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:34:31.436052 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:34:31.436019    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21347091-b8fa-49e4-9bcd-413f7ef30d57-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt\" (UID: \"21347091-b8fa-49e4-9bcd-413f7ef30d57\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt"
Jul 27 22:34:31.438386 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:34:31.438363    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21347091-b8fa-49e4-9bcd-413f7ef30d57-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt\" (UID: \"21347091-b8fa-49e4-9bcd-413f7ef30d57\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt"
Jul 27 22:34:31.665588 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:34:31.665503    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt"
Jul 27 22:34:31.797886 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:34:31.797862    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt"]
Jul 27 22:34:31.799723 ip-10-0-142-219 kubenswrapper[2577]: W0727 22:34:31.799691    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21347091_b8fa_49e4_9bcd_413f7ef30d57.slice/crio-e22516b600f13612141ad4c7e7cea28212292603afd3220ceab5196dfa1f58ca WatchSource:0}: Error finding container e22516b600f13612141ad4c7e7cea28212292603afd3220ceab5196dfa1f58ca: Status 404 returned error can't find the container with id e22516b600f13612141ad4c7e7cea28212292603afd3220ceab5196dfa1f58ca
Jul 27 22:34:31.911638 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:34:31.911600    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt" event={"ID":"21347091-b8fa-49e4-9bcd-413f7ef30d57","Type":"ContainerStarted","Data":"efbc2df420dcbd96f47c7dc1f86586b426a8ca90a81743ac7ad4c791b002f2a9"}
Jul 27 22:34:31.911838 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:34:31.911663    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt" event={"ID":"21347091-b8fa-49e4-9bcd-413f7ef30d57","Type":"ContainerStarted","Data":"e22516b600f13612141ad4c7e7cea28212292603afd3220ceab5196dfa1f58ca"}
Jul 27 22:34:32.916282 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:34:32.916246    2577 generic.go:356] "Generic (PLEG): container finished" podID="21347091-b8fa-49e4-9bcd-413f7ef30d57" containerID="efbc2df420dcbd96f47c7dc1f86586b426a8ca90a81743ac7ad4c791b002f2a9" exitCode=0
Jul 27 22:34:32.916659 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:34:32.916329    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt" event={"ID":"21347091-b8fa-49e4-9bcd-413f7ef30d57","Type":"ContainerDied","Data":"efbc2df420dcbd96f47c7dc1f86586b426a8ca90a81743ac7ad4c791b002f2a9"}
Jul 27 22:34:33.921844 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:34:33.921805    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt" event={"ID":"21347091-b8fa-49e4-9bcd-413f7ef30d57","Type":"ContainerStarted","Data":"359a201876690cac285d52a62f20ae4274a93713e261170beccecca121d88d91"}
Jul 27 22:34:33.922272 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:34:33.921862    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt" event={"ID":"21347091-b8fa-49e4-9bcd-413f7ef30d57","Type":"ContainerStarted","Data":"81f80712ff285f3aa6569eaf7419b70e76d7cfb84b99f61cc9da0eb47b3bdb0b"}
Jul 27 22:34:33.922272 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:34:33.921955    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt"
Jul 27 22:34:33.922272 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:34:33.921996    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt"
Jul 27 22:34:33.923184 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:34:33.923157    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt" podUID="21347091-b8fa-49e4-9bcd-413f7ef30d57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.71:8080: connect: connection refused"
Jul 27 22:34:33.946564 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:34:33.946521    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt" podStartSLOduration=2.930853753 podStartE2EDuration="3.946507311s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.015653558s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:34:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:34:33.945407156 +0000 UTC m=+5397.312597020" watchObservedRunningTime="2026-07-27 22:34:33.946507311 +0000 UTC m=+5397.313697177"
Jul 27 22:34:34.926942 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:34:34.926903    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt" podUID="21347091-b8fa-49e4-9bcd-413f7ef30d57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.71:8080: connect: connection refused"
Jul 27 22:34:39.932297 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:34:39.932267    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt"
Jul 27 22:34:39.932832 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:34:39.932801    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt" podUID="21347091-b8fa-49e4-9bcd-413f7ef30d57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.71:8080: connect: connection refused"
Jul 27 22:34:49.933697 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:34:49.933654    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt" podUID="21347091-b8fa-49e4-9bcd-413f7ef30d57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.71:8080: connect: connection refused"
Jul 27 22:34:59.932870 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:34:59.932826    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt" podUID="21347091-b8fa-49e4-9bcd-413f7ef30d57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.71:8080: connect: connection refused"
Jul 27 22:35:09.933608 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:35:09.933559    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt" podUID="21347091-b8fa-49e4-9bcd-413f7ef30d57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.71:8080: connect: connection refused"
Jul 27 22:35:19.932989 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:35:19.932952    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt" podUID="21347091-b8fa-49e4-9bcd-413f7ef30d57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.71:8080: connect: connection refused"
Jul 27 22:35:29.932794 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:35:29.932739    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt" podUID="21347091-b8fa-49e4-9bcd-413f7ef30d57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.71:8080: connect: connection refused"
Jul 27 22:35:39.933754 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:35:39.933716    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt" podUID="21347091-b8fa-49e4-9bcd-413f7ef30d57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.71:8080: connect: connection refused"
Jul 27 22:35:49.933700 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:35:49.933669    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt"
Jul 27 22:35:50.799597 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:35:50.799562    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt"]
Jul 27 22:35:50.799895 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:35:50.799856    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt" podUID="21347091-b8fa-49e4-9bcd-413f7ef30d57" containerName="kserve-container" containerID="cri-o://81f80712ff285f3aa6569eaf7419b70e76d7cfb84b99f61cc9da0eb47b3bdb0b" gracePeriod=30
Jul 27 22:35:50.799985 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:35:50.799960    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt" podUID="21347091-b8fa-49e4-9bcd-413f7ef30d57" containerName="kube-rbac-proxy" containerID="cri-o://359a201876690cac285d52a62f20ae4274a93713e261170beccecca121d88d91" gracePeriod=30
Jul 27 22:35:50.823638 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:35:50.823611    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:35:51.211025 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:35:51.210990    2577 generic.go:356] "Generic (PLEG): container finished" podID="21347091-b8fa-49e4-9bcd-413f7ef30d57" containerID="359a201876690cac285d52a62f20ae4274a93713e261170beccecca121d88d91" exitCode=2
Jul 27 22:35:51.211393 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:35:51.211066    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt" event={"ID":"21347091-b8fa-49e4-9bcd-413f7ef30d57","Type":"ContainerDied","Data":"359a201876690cac285d52a62f20ae4274a93713e261170beccecca121d88d91"}
Jul 27 22:35:51.853023 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:35:51.852988    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:35:51.919034 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:35:51.919004    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:35:54.927607 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:35:54.927568    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt" podUID="21347091-b8fa-49e4-9bcd-413f7ef30d57" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.71:8643/healthz\": dial tcp 10.133.0.71:8643: connect: connection refused"
Jul 27 22:35:55.137657 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:35:55.137632    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt"
Jul 27 22:35:55.186333 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:35:55.186309    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fq2ff\" (UniqueName: \"kubernetes.io/projected/21347091-b8fa-49e4-9bcd-413f7ef30d57-kube-api-access-fq2ff\") pod \"21347091-b8fa-49e4-9bcd-413f7ef30d57\" (UID: \"21347091-b8fa-49e4-9bcd-413f7ef30d57\") "
Jul 27 22:35:55.186474 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:35:55.186364    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21347091-b8fa-49e4-9bcd-413f7ef30d57-kserve-provision-location\") pod \"21347091-b8fa-49e4-9bcd-413f7ef30d57\" (UID: \"21347091-b8fa-49e4-9bcd-413f7ef30d57\") "
Jul 27 22:35:55.186474 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:35:55.186387    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21347091-b8fa-49e4-9bcd-413f7ef30d57-kube-rbac-proxy-sar-config\") pod \"21347091-b8fa-49e4-9bcd-413f7ef30d57\" (UID: \"21347091-b8fa-49e4-9bcd-413f7ef30d57\") "
Jul 27 22:35:55.186474 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:35:55.186415    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21347091-b8fa-49e4-9bcd-413f7ef30d57-proxy-tls\") pod \"21347091-b8fa-49e4-9bcd-413f7ef30d57\" (UID: \"21347091-b8fa-49e4-9bcd-413f7ef30d57\") "
Jul 27 22:35:55.186474 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:35:55.186430    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21347091-b8fa-49e4-9bcd-413f7ef30d57-cabundle-cert\") pod \"21347091-b8fa-49e4-9bcd-413f7ef30d57\" (UID: \"21347091-b8fa-49e4-9bcd-413f7ef30d57\") "
Jul 27 22:35:55.186745 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:35:55.186703    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21347091-b8fa-49e4-9bcd-413f7ef30d57-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "21347091-b8fa-49e4-9bcd-413f7ef30d57" (UID: "21347091-b8fa-49e4-9bcd-413f7ef30d57"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:35:55.186857 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:35:55.186830    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21347091-b8fa-49e4-9bcd-413f7ef30d57-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "21347091-b8fa-49e4-9bcd-413f7ef30d57" (UID: "21347091-b8fa-49e4-9bcd-413f7ef30d57"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:35:55.186900 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:35:55.186851    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21347091-b8fa-49e4-9bcd-413f7ef30d57-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "21347091-b8fa-49e4-9bcd-413f7ef30d57" (UID: "21347091-b8fa-49e4-9bcd-413f7ef30d57"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:35:55.188446 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:35:55.188425    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21347091-b8fa-49e4-9bcd-413f7ef30d57-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "21347091-b8fa-49e4-9bcd-413f7ef30d57" (UID: "21347091-b8fa-49e4-9bcd-413f7ef30d57"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 22:35:55.188528 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:35:55.188467    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21347091-b8fa-49e4-9bcd-413f7ef30d57-kube-api-access-fq2ff" (OuterVolumeSpecName: "kube-api-access-fq2ff") pod "21347091-b8fa-49e4-9bcd-413f7ef30d57" (UID: "21347091-b8fa-49e4-9bcd-413f7ef30d57"). InnerVolumeSpecName "kube-api-access-fq2ff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:35:55.227713 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:35:55.227692    2577 generic.go:356] "Generic (PLEG): container finished" podID="21347091-b8fa-49e4-9bcd-413f7ef30d57" containerID="81f80712ff285f3aa6569eaf7419b70e76d7cfb84b99f61cc9da0eb47b3bdb0b" exitCode=0
Jul 27 22:35:55.227818 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:35:55.227741    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt" event={"ID":"21347091-b8fa-49e4-9bcd-413f7ef30d57","Type":"ContainerDied","Data":"81f80712ff285f3aa6569eaf7419b70e76d7cfb84b99f61cc9da0eb47b3bdb0b"}
Jul 27 22:35:55.227818 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:35:55.227753    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt"
Jul 27 22:35:55.227818 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:35:55.227809    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt" event={"ID":"21347091-b8fa-49e4-9bcd-413f7ef30d57","Type":"ContainerDied","Data":"e22516b600f13612141ad4c7e7cea28212292603afd3220ceab5196dfa1f58ca"}
Jul 27 22:35:55.227927 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:35:55.227836    2577 scope.go:117] "RemoveContainer" containerID="359a201876690cac285d52a62f20ae4274a93713e261170beccecca121d88d91"
Jul 27 22:35:55.236098 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:35:55.236083    2577 scope.go:117] "RemoveContainer" containerID="81f80712ff285f3aa6569eaf7419b70e76d7cfb84b99f61cc9da0eb47b3bdb0b"
Jul 27 22:35:55.242820 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:35:55.242806    2577 scope.go:117] "RemoveContainer" containerID="efbc2df420dcbd96f47c7dc1f86586b426a8ca90a81743ac7ad4c791b002f2a9"
Jul 27 22:35:55.248456 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:35:55.248431    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt"]
Jul 27 22:35:55.250818 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:35:55.250796    2577 scope.go:117] "RemoveContainer" containerID="359a201876690cac285d52a62f20ae4274a93713e261170beccecca121d88d91"
Jul 27 22:35:55.251077 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:35:55.251060    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"359a201876690cac285d52a62f20ae4274a93713e261170beccecca121d88d91\": container with ID starting with 359a201876690cac285d52a62f20ae4274a93713e261170beccecca121d88d91 not found: ID does not exist" containerID="359a201876690cac285d52a62f20ae4274a93713e261170beccecca121d88d91"
Jul 27 22:35:55.251141 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:35:55.251085    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"359a201876690cac285d52a62f20ae4274a93713e261170beccecca121d88d91"} err="failed to get container status \"359a201876690cac285d52a62f20ae4274a93713e261170beccecca121d88d91\": rpc error: code = NotFound desc = could not find container \"359a201876690cac285d52a62f20ae4274a93713e261170beccecca121d88d91\": container with ID starting with 359a201876690cac285d52a62f20ae4274a93713e261170beccecca121d88d91 not found: ID does not exist"
Jul 27 22:35:55.251141 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:35:55.251100    2577 scope.go:117] "RemoveContainer" containerID="81f80712ff285f3aa6569eaf7419b70e76d7cfb84b99f61cc9da0eb47b3bdb0b"
Jul 27 22:35:55.251301 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:35:55.251279    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81f80712ff285f3aa6569eaf7419b70e76d7cfb84b99f61cc9da0eb47b3bdb0b\": container with ID starting with 81f80712ff285f3aa6569eaf7419b70e76d7cfb84b99f61cc9da0eb47b3bdb0b not found: ID does not exist" containerID="81f80712ff285f3aa6569eaf7419b70e76d7cfb84b99f61cc9da0eb47b3bdb0b"
Jul 27 22:35:55.251349 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:35:55.251311    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81f80712ff285f3aa6569eaf7419b70e76d7cfb84b99f61cc9da0eb47b3bdb0b"} err="failed to get container status \"81f80712ff285f3aa6569eaf7419b70e76d7cfb84b99f61cc9da0eb47b3bdb0b\": rpc error: code = NotFound desc = could not find container \"81f80712ff285f3aa6569eaf7419b70e76d7cfb84b99f61cc9da0eb47b3bdb0b\": container with ID starting with 81f80712ff285f3aa6569eaf7419b70e76d7cfb84b99f61cc9da0eb47b3bdb0b not found: ID does not exist"
Jul 27 22:35:55.251349 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:35:55.251326    2577 scope.go:117] "RemoveContainer" containerID="efbc2df420dcbd96f47c7dc1f86586b426a8ca90a81743ac7ad4c791b002f2a9"
Jul 27 22:35:55.251569 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:35:55.251539    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efbc2df420dcbd96f47c7dc1f86586b426a8ca90a81743ac7ad4c791b002f2a9\": container with ID starting with efbc2df420dcbd96f47c7dc1f86586b426a8ca90a81743ac7ad4c791b002f2a9 not found: ID does not exist" containerID="efbc2df420dcbd96f47c7dc1f86586b426a8ca90a81743ac7ad4c791b002f2a9"
Jul 27 22:35:55.251615 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:35:55.251578    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efbc2df420dcbd96f47c7dc1f86586b426a8ca90a81743ac7ad4c791b002f2a9"} err="failed to get container status \"efbc2df420dcbd96f47c7dc1f86586b426a8ca90a81743ac7ad4c791b002f2a9\": rpc error: code = NotFound desc = could not find container \"efbc2df420dcbd96f47c7dc1f86586b426a8ca90a81743ac7ad4c791b002f2a9\": container with ID starting with efbc2df420dcbd96f47c7dc1f86586b426a8ca90a81743ac7ad4c791b002f2a9 not found: ID does not exist"
Jul 27 22:35:55.252185 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:35:55.252170    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-756d9c6b57-zzrtt"]
Jul 27 22:35:55.286938 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:35:55.286918    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq2ff\" (UniqueName: \"kubernetes.io/projected/21347091-b8fa-49e4-9bcd-413f7ef30d57-kube-api-access-fq2ff\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:35:55.286938 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:35:55.286936    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21347091-b8fa-49e4-9bcd-413f7ef30d57-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:35:55.287049 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:35:55.286946    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21347091-b8fa-49e4-9bcd-413f7ef30d57-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:35:55.287049 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:35:55.286956    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21347091-b8fa-49e4-9bcd-413f7ef30d57-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:35:55.287049 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:35:55.286965    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21347091-b8fa-49e4-9bcd-413f7ef30d57-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:35:57.135530 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:35:57.135495    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21347091-b8fa-49e4-9bcd-413f7ef30d57" path="/var/lib/kubelet/pods/21347091-b8fa-49e4-9bcd-413f7ef30d57/volumes"
Jul 27 22:36:02.233150 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:36:02.233066    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:36:02.969215 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:36:02.969183    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:36:03.149042 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:36:03.149011    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:36:58.000184 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:36:58.000144    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:37:23.066387 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:37:23.066336    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:37:24.160088 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:37:24.160050    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:37:24.384476 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:37:24.384441    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:37:34.140132 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:37:34.140101    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:37:36.233230 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:37:36.233196    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-clf4c/must-gather-2454s"]
Jul 27 22:37:36.233594 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:37:36.233461    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21347091-b8fa-49e4-9bcd-413f7ef30d57" containerName="kube-rbac-proxy"
Jul 27 22:37:36.233594 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:37:36.233473    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="21347091-b8fa-49e4-9bcd-413f7ef30d57" containerName="kube-rbac-proxy"
Jul 27 22:37:36.233594 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:37:36.233484    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21347091-b8fa-49e4-9bcd-413f7ef30d57" containerName="kserve-container"
Jul 27 22:37:36.233594 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:37:36.233489    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="21347091-b8fa-49e4-9bcd-413f7ef30d57" containerName="kserve-container"
Jul 27 22:37:36.233594 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:37:36.233500    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21347091-b8fa-49e4-9bcd-413f7ef30d57" containerName="storage-initializer"
Jul 27 22:37:36.233594 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:37:36.233507    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="21347091-b8fa-49e4-9bcd-413f7ef30d57" containerName="storage-initializer"
Jul 27 22:37:36.233594 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:37:36.233553    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="21347091-b8fa-49e4-9bcd-413f7ef30d57" containerName="kube-rbac-proxy"
Jul 27 22:37:36.233594 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:37:36.233561    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="21347091-b8fa-49e4-9bcd-413f7ef30d57" containerName="kserve-container"
Jul 27 22:37:36.236555 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:37:36.236535    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-clf4c/must-gather-2454s"
Jul 27 22:37:36.239031 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:37:36.239010    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-clf4c\"/\"openshift-service-ca.crt\""
Jul 27 22:37:36.239142 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:37:36.239090    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-clf4c\"/\"kube-root-ca.crt\""
Jul 27 22:37:36.246361 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:37:36.246341    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-clf4c/must-gather-2454s"]
Jul 27 22:37:36.251019 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:37:36.250970    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:37:36.310042 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:37:36.310017    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:37:36.373335 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:37:36.373300    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9f5f8dbd-f0b8-4464-a438-eddd06969443-must-gather-output\") pod \"must-gather-2454s\" (UID: \"9f5f8dbd-f0b8-4464-a438-eddd06969443\") " pod="openshift-must-gather-clf4c/must-gather-2454s"
Jul 27 22:37:36.373439 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:37:36.373393    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dll8k\" (UniqueName: \"kubernetes.io/projected/9f5f8dbd-f0b8-4464-a438-eddd06969443-kube-api-access-dll8k\") pod \"must-gather-2454s\" (UID: \"9f5f8dbd-f0b8-4464-a438-eddd06969443\") " pod="openshift-must-gather-clf4c/must-gather-2454s"
Jul 27 22:37:36.473858 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:37:36.473834    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dll8k\" (UniqueName: \"kubernetes.io/projected/9f5f8dbd-f0b8-4464-a438-eddd06969443-kube-api-access-dll8k\") pod \"must-gather-2454s\" (UID: \"9f5f8dbd-f0b8-4464-a438-eddd06969443\") " pod="openshift-must-gather-clf4c/must-gather-2454s"
Jul 27 22:37:36.473946 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:37:36.473872    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9f5f8dbd-f0b8-4464-a438-eddd06969443-must-gather-output\") pod \"must-gather-2454s\" (UID: \"9f5f8dbd-f0b8-4464-a438-eddd06969443\") " pod="openshift-must-gather-clf4c/must-gather-2454s"
Jul 27 22:37:36.474231 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:37:36.474215    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9f5f8dbd-f0b8-4464-a438-eddd06969443-must-gather-output\") pod \"must-gather-2454s\" (UID: \"9f5f8dbd-f0b8-4464-a438-eddd06969443\") " pod="openshift-must-gather-clf4c/must-gather-2454s"
Jul 27 22:37:36.481159 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:37:36.481141    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dll8k\" (UniqueName: \"kubernetes.io/projected/9f5f8dbd-f0b8-4464-a438-eddd06969443-kube-api-access-dll8k\") pod \"must-gather-2454s\" (UID: \"9f5f8dbd-f0b8-4464-a438-eddd06969443\") " pod="openshift-must-gather-clf4c/must-gather-2454s"
Jul 27 22:37:36.545034 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:37:36.544981    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-clf4c/must-gather-2454s"
Jul 27 22:37:36.669126 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:37:36.669101    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-clf4c/must-gather-2454s"]
Jul 27 22:37:36.670883 ip-10-0-142-219 kubenswrapper[2577]: W0727 22:37:36.670854    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f5f8dbd_f0b8_4464_a438_eddd06969443.slice/crio-14d4e2650cc3ef9a14e925afe33f30bc32a51486831af43459f3fbb7023ecfda WatchSource:0}: Error finding container 14d4e2650cc3ef9a14e925afe33f30bc32a51486831af43459f3fbb7023ecfda: Status 404 returned error can't find the container with id 14d4e2650cc3ef9a14e925afe33f30bc32a51486831af43459f3fbb7023ecfda
Jul 27 22:37:36.672628 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:37:36.672611    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 22:37:37.625058 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:37:37.624987    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-clf4c/must-gather-2454s" event={"ID":"9f5f8dbd-f0b8-4464-a438-eddd06969443","Type":"ContainerStarted","Data":"14d4e2650cc3ef9a14e925afe33f30bc32a51486831af43459f3fbb7023ecfda"}
Jul 27 22:37:43.654807 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:37:43.654742    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-clf4c/must-gather-2454s" event={"ID":"9f5f8dbd-f0b8-4464-a438-eddd06969443","Type":"ContainerStarted","Data":"00f97aacae6f9f15967a6e917fab80b63e33a1088163d6f24339d7d0be86a270"}
Jul 27 22:37:43.655292 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:37:43.654820    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-clf4c/must-gather-2454s" event={"ID":"9f5f8dbd-f0b8-4464-a438-eddd06969443","Type":"ContainerStarted","Data":"5b4b0af534ce3dbbd6d834e75f53fdb7fbe3c95a4b0566ff41b7e33c1266fe2f"}
Jul 27 22:37:43.672821 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:37:43.672753    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-clf4c/must-gather-2454s" podStartSLOduration=1.482561281 podStartE2EDuration="7.672737682s" totalImagesPullingTime="6.190176401s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:37:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:37:43.671672394 +0000 UTC m=+5587.038862258" watchObservedRunningTime="2026-07-27 22:37:43.672737682 +0000 UTC m=+5587.039927547"
Jul 27 22:37:57.624532 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:37:57.624490    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:38:09.764511 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:09.764473    2577 generic.go:356] "Generic (PLEG): container finished" podID="9f5f8dbd-f0b8-4464-a438-eddd06969443" containerID="5b4b0af534ce3dbbd6d834e75f53fdb7fbe3c95a4b0566ff41b7e33c1266fe2f" exitCode=0
Jul 27 22:38:09.764963 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:09.764543    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-clf4c/must-gather-2454s" event={"ID":"9f5f8dbd-f0b8-4464-a438-eddd06969443","Type":"ContainerDied","Data":"5b4b0af534ce3dbbd6d834e75f53fdb7fbe3c95a4b0566ff41b7e33c1266fe2f"}
Jul 27 22:38:09.764963 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:09.764833    2577 scope.go:117] "RemoveContainer" containerID="5b4b0af534ce3dbbd6d834e75f53fdb7fbe3c95a4b0566ff41b7e33c1266fe2f"
Jul 27 22:38:09.888588 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:09.888534    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-clf4c_must-gather-2454s_9f5f8dbd-f0b8-4464-a438-eddd06969443/gather/0.log"
Jul 27 22:38:10.409176 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:10.409098    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:38:10.557140 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:10.557109    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:38:10.619911 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:10.619877    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:38:13.140036 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:13.140006    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5d7vf_a4af1379-e40a-471d-a011-d50ab76b7f51/global-pull-secret-syncer/0.log"
Jul 27 22:38:13.313993 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:13.313950    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log"
Jul 27 22:38:13.452534 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:13.452479    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-219.ec2.internal_7fe2fc32304c33ef13ea6c3a49bb44af/haproxy/0.log"
Jul 27 22:38:15.394661 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:15.394626    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-clf4c/must-gather-2454s"]
Jul 27 22:38:15.395115 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:15.394798    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-clf4c/must-gather-2454s" podUID="9f5f8dbd-f0b8-4464-a438-eddd06969443" containerName="copy" containerID="cri-o://00f97aacae6f9f15967a6e917fab80b63e33a1088163d6f24339d7d0be86a270" gracePeriod=2
Jul 27 22:38:15.396951 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:38:15.396919    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-2454s\" is forbidden: User \"system:node:ip-10-0-142-219.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-clf4c\": no relationship found between node 'ip-10-0-142-219.ec2.internal' and this object" podUID="9f5f8dbd-f0b8-4464-a438-eddd06969443" pod="openshift-must-gather-clf4c/must-gather-2454s"
Jul 27 22:38:15.397139 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:15.397078    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-clf4c/must-gather-2454s"]
Jul 27 22:38:15.622749 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:15.622726    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-clf4c_must-gather-2454s_9f5f8dbd-f0b8-4464-a438-eddd06969443/copy/0.log"
Jul 27 22:38:15.624483 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:15.624463    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-clf4c/must-gather-2454s"
Jul 27 22:38:15.626401 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:38:15.626379    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-2454s\" is forbidden: User \"system:node:ip-10-0-142-219.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-clf4c\": no relationship found between node 'ip-10-0-142-219.ec2.internal' and this object" podUID="9f5f8dbd-f0b8-4464-a438-eddd06969443" pod="openshift-must-gather-clf4c/must-gather-2454s"
Jul 27 22:38:15.696857 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:15.696831    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dll8k\" (UniqueName: \"kubernetes.io/projected/9f5f8dbd-f0b8-4464-a438-eddd06969443-kube-api-access-dll8k\") pod \"9f5f8dbd-f0b8-4464-a438-eddd06969443\" (UID: \"9f5f8dbd-f0b8-4464-a438-eddd06969443\") "
Jul 27 22:38:15.696957 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:15.696895    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9f5f8dbd-f0b8-4464-a438-eddd06969443-must-gather-output\") pod \"9f5f8dbd-f0b8-4464-a438-eddd06969443\" (UID: \"9f5f8dbd-f0b8-4464-a438-eddd06969443\") "
Jul 27 22:38:15.698480 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:15.698453    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f5f8dbd-f0b8-4464-a438-eddd06969443-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "9f5f8dbd-f0b8-4464-a438-eddd06969443" (UID: "9f5f8dbd-f0b8-4464-a438-eddd06969443"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:38:15.698989 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:15.698969    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f5f8dbd-f0b8-4464-a438-eddd06969443-kube-api-access-dll8k" (OuterVolumeSpecName: "kube-api-access-dll8k") pod "9f5f8dbd-f0b8-4464-a438-eddd06969443" (UID: "9f5f8dbd-f0b8-4464-a438-eddd06969443"). InnerVolumeSpecName "kube-api-access-dll8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:38:15.787482 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:15.787455    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-clf4c_must-gather-2454s_9f5f8dbd-f0b8-4464-a438-eddd06969443/copy/0.log"
Jul 27 22:38:15.787842 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:15.787818    2577 generic.go:356] "Generic (PLEG): container finished" podID="9f5f8dbd-f0b8-4464-a438-eddd06969443" containerID="00f97aacae6f9f15967a6e917fab80b63e33a1088163d6f24339d7d0be86a270" exitCode=143
Jul 27 22:38:15.787956 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:15.787861    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-clf4c/must-gather-2454s"
Jul 27 22:38:15.787956 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:15.787901    2577 scope.go:117] "RemoveContainer" containerID="00f97aacae6f9f15967a6e917fab80b63e33a1088163d6f24339d7d0be86a270"
Jul 27 22:38:15.790181 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:38:15.790153    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-2454s\" is forbidden: User \"system:node:ip-10-0-142-219.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-clf4c\": no relationship found between node 'ip-10-0-142-219.ec2.internal' and this object" podUID="9f5f8dbd-f0b8-4464-a438-eddd06969443" pod="openshift-must-gather-clf4c/must-gather-2454s"
Jul 27 22:38:15.795779 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:15.795748    2577 scope.go:117] "RemoveContainer" containerID="5b4b0af534ce3dbbd6d834e75f53fdb7fbe3c95a4b0566ff41b7e33c1266fe2f"
Jul 27 22:38:15.798105 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:38:15.798075    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-2454s\" is forbidden: User \"system:node:ip-10-0-142-219.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-clf4c\": no relationship found between node 'ip-10-0-142-219.ec2.internal' and this object" podUID="9f5f8dbd-f0b8-4464-a438-eddd06969443" pod="openshift-must-gather-clf4c/must-gather-2454s"
Jul 27 22:38:15.798197 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:15.798093    2577 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9f5f8dbd-f0b8-4464-a438-eddd06969443-must-gather-output\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:38:15.798197 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:15.798132    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dll8k\" (UniqueName: \"kubernetes.io/projected/9f5f8dbd-f0b8-4464-a438-eddd06969443-kube-api-access-dll8k\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 27 22:38:15.807446 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:15.807416    2577 scope.go:117] "RemoveContainer" containerID="00f97aacae6f9f15967a6e917fab80b63e33a1088163d6f24339d7d0be86a270"
Jul 27 22:38:15.807705 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:38:15.807686    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00f97aacae6f9f15967a6e917fab80b63e33a1088163d6f24339d7d0be86a270\": container with ID starting with 00f97aacae6f9f15967a6e917fab80b63e33a1088163d6f24339d7d0be86a270 not found: ID does not exist" containerID="00f97aacae6f9f15967a6e917fab80b63e33a1088163d6f24339d7d0be86a270"
Jul 27 22:38:15.807774 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:15.807720    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00f97aacae6f9f15967a6e917fab80b63e33a1088163d6f24339d7d0be86a270"} err="failed to get container status \"00f97aacae6f9f15967a6e917fab80b63e33a1088163d6f24339d7d0be86a270\": rpc error: code = NotFound desc = could not find container \"00f97aacae6f9f15967a6e917fab80b63e33a1088163d6f24339d7d0be86a270\": container with ID starting with 00f97aacae6f9f15967a6e917fab80b63e33a1088163d6f24339d7d0be86a270 not found: ID does not exist"
Jul 27 22:38:15.807774 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:15.807738    2577 scope.go:117] "RemoveContainer" containerID="5b4b0af534ce3dbbd6d834e75f53fdb7fbe3c95a4b0566ff41b7e33c1266fe2f"
Jul 27 22:38:15.807952 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:38:15.807937    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b4b0af534ce3dbbd6d834e75f53fdb7fbe3c95a4b0566ff41b7e33c1266fe2f\": container with ID starting with 5b4b0af534ce3dbbd6d834e75f53fdb7fbe3c95a4b0566ff41b7e33c1266fe2f not found: ID does not exist" containerID="5b4b0af534ce3dbbd6d834e75f53fdb7fbe3c95a4b0566ff41b7e33c1266fe2f"
Jul 27 22:38:15.807995 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:15.807955    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b4b0af534ce3dbbd6d834e75f53fdb7fbe3c95a4b0566ff41b7e33c1266fe2f"} err="failed to get container status \"5b4b0af534ce3dbbd6d834e75f53fdb7fbe3c95a4b0566ff41b7e33c1266fe2f\": rpc error: code = NotFound desc = could not find container \"5b4b0af534ce3dbbd6d834e75f53fdb7fbe3c95a4b0566ff41b7e33c1266fe2f\": container with ID starting with 5b4b0af534ce3dbbd6d834e75f53fdb7fbe3c95a4b0566ff41b7e33c1266fe2f not found: ID does not exist"
Jul 27 22:38:15.941189 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:15.941154    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:38:16.001960 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:16.001889    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b9b6s_0b7c9f3b-8cf6-44dd-8378-3b5af6b644b6/konnectivity-agent/0.log" limit=1024
Jul 27 22:38:17.133871 ip-10-0-142-219 kubenswrapper[2577]: E0727 22:38:17.133833    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-2454s\" is forbidden: User \"system:node:ip-10-0-142-219.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-clf4c\": no relationship found between node 'ip-10-0-142-219.ec2.internal' and this object" podUID="9f5f8dbd-f0b8-4464-a438-eddd06969443" pod="openshift-must-gather-clf4c/must-gather-2454s"
Jul 27 22:38:17.134429 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:17.134408    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f5f8dbd-f0b8-4464-a438-eddd06969443" path="/var/lib/kubelet/pods/9f5f8dbd-f0b8-4464-a438-eddd06969443/volumes"
Jul 27 22:38:18.145506 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:18.145481    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-glj4s_b5d1ed58-e125-4d48-beba-c93d6cb7d5b7/node-exporter/0.log"
Jul 27 22:38:18.179200 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:18.179176    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-glj4s_b5d1ed58-e125-4d48-beba-c93d6cb7d5b7/kube-rbac-proxy/0.log"
Jul 27 22:38:18.206491 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:18.206467    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-glj4s_b5d1ed58-e125-4d48-beba-c93d6cb7d5b7/init-textfile/0.log"
Jul 27 22:38:18.313070 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:18.313045    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-dmpfr_8c65aca0-1763-49ac-a497-5a9176651765/kube-rbac-proxy-main/0.log"
Jul 27 22:38:18.339495 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:18.339471    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-dmpfr_8c65aca0-1763-49ac-a497-5a9176651765/kube-rbac-proxy-self/0.log"
Jul 27 22:38:18.365005 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:18.364982    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-dmpfr_8c65aca0-1763-49ac-a497-5a9176651765/openshift-state-metrics/0.log"
Jul 27 22:38:18.411800 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:18.411712    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0f14c1b0-8963-4e4a-8f39-1501689b67d6/prometheus/0.log"
Jul 27 22:38:18.438391 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:18.438356    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0f14c1b0-8963-4e4a-8f39-1501689b67d6/config-reloader/0.log"
Jul 27 22:38:18.462967 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:18.462943    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0f14c1b0-8963-4e4a-8f39-1501689b67d6/thanos-sidecar/0.log"
Jul 27 22:38:18.491357 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:18.491331    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0f14c1b0-8963-4e4a-8f39-1501689b67d6/kube-rbac-proxy-web/0.log"
Jul 27 22:38:18.517854 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:18.517834    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0f14c1b0-8963-4e4a-8f39-1501689b67d6/kube-rbac-proxy/0.log"
Jul 27 22:38:18.542602 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:18.542585    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0f14c1b0-8963-4e4a-8f39-1501689b67d6/kube-rbac-proxy-thanos/0.log"
Jul 27 22:38:18.570387 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:18.570369    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0f14c1b0-8963-4e4a-8f39-1501689b67d6/init-config-reloader/0.log"
Jul 27 22:38:18.603808 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:18.603788    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-qt8p9_4a316693-388f-41e7-bc94-2dce39838f9a/prometheus-operator/0.log"
Jul 27 22:38:18.631489 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:18.631467    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-qt8p9_4a316693-388f-41e7-bc94-2dce39838f9a/kube-rbac-proxy/0.log"
Jul 27 22:38:18.658882 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:18.658863    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-2959c_f87d5ad2-d8c3-4bd5-8ac4-722d19ce0ef0/prometheus-operator-admission-webhook/0.log"
Jul 27 22:38:21.275039 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:21.275010    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-t7jph/perf-node-gather-daemonset-2mmr4"]
Jul 27 22:38:21.275442 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:21.275272    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f5f8dbd-f0b8-4464-a438-eddd06969443" containerName="gather"
Jul 27 22:38:21.275442 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:21.275282    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f5f8dbd-f0b8-4464-a438-eddd06969443" containerName="gather"
Jul 27 22:38:21.275442 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:21.275301    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f5f8dbd-f0b8-4464-a438-eddd06969443" containerName="copy"
Jul 27 22:38:21.275442 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:21.275306    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f5f8dbd-f0b8-4464-a438-eddd06969443" containerName="copy"
Jul 27 22:38:21.275442 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:21.275345    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f5f8dbd-f0b8-4464-a438-eddd06969443" containerName="gather"
Jul 27 22:38:21.275442 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:21.275358    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f5f8dbd-f0b8-4464-a438-eddd06969443" containerName="copy"
Jul 27 22:38:21.278935 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:21.278920    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t7jph/perf-node-gather-daemonset-2mmr4"
Jul 27 22:38:21.281483 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:21.281463    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t7jph\"/\"kube-root-ca.crt\""
Jul 27 22:38:21.281601 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:21.281533    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t7jph\"/\"openshift-service-ca.crt\""
Jul 27 22:38:21.282305 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:21.282293    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-t7jph\"/\"default-dockercfg-wwbv9\""
Jul 27 22:38:21.288306 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:21.288285    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t7jph/perf-node-gather-daemonset-2mmr4"]
Jul 27 22:38:21.338919 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:21.338895    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1788568b-f83e-45b9-ba74-083ed033b3d7-sys\") pod \"perf-node-gather-daemonset-2mmr4\" (UID: \"1788568b-f83e-45b9-ba74-083ed033b3d7\") " pod="openshift-must-gather-t7jph/perf-node-gather-daemonset-2mmr4"
Jul 27 22:38:21.339017 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:21.338968    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1788568b-f83e-45b9-ba74-083ed033b3d7-podres\") pod \"perf-node-gather-daemonset-2mmr4\" (UID: \"1788568b-f83e-45b9-ba74-083ed033b3d7\") " pod="openshift-must-gather-t7jph/perf-node-gather-daemonset-2mmr4"
Jul 27 22:38:21.339065 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:21.339017    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1788568b-f83e-45b9-ba74-083ed033b3d7-proc\") pod \"perf-node-gather-daemonset-2mmr4\" (UID: \"1788568b-f83e-45b9-ba74-083ed033b3d7\") " pod="openshift-must-gather-t7jph/perf-node-gather-daemonset-2mmr4"
Jul 27 22:38:21.339065 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:21.339040    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1788568b-f83e-45b9-ba74-083ed033b3d7-lib-modules\") pod \"perf-node-gather-daemonset-2mmr4\" (UID: \"1788568b-f83e-45b9-ba74-083ed033b3d7\") " pod="openshift-must-gather-t7jph/perf-node-gather-daemonset-2mmr4"
Jul 27 22:38:21.339132 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:21.339063    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zsx7\" (UniqueName: \"kubernetes.io/projected/1788568b-f83e-45b9-ba74-083ed033b3d7-kube-api-access-9zsx7\") pod \"perf-node-gather-daemonset-2mmr4\" (UID: \"1788568b-f83e-45b9-ba74-083ed033b3d7\") " pod="openshift-must-gather-t7jph/perf-node-gather-daemonset-2mmr4"
Jul 27 22:38:21.439941 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:21.439914    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1788568b-f83e-45b9-ba74-083ed033b3d7-podres\") pod \"perf-node-gather-daemonset-2mmr4\" (UID: \"1788568b-f83e-45b9-ba74-083ed033b3d7\") " pod="openshift-must-gather-t7jph/perf-node-gather-daemonset-2mmr4"
Jul 27 22:38:21.440046 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:21.439958    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1788568b-f83e-45b9-ba74-083ed033b3d7-proc\") pod \"perf-node-gather-daemonset-2mmr4\" (UID: \"1788568b-f83e-45b9-ba74-083ed033b3d7\") " pod="openshift-must-gather-t7jph/perf-node-gather-daemonset-2mmr4"
Jul 27 22:38:21.440046 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:21.439986    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1788568b-f83e-45b9-ba74-083ed033b3d7-lib-modules\") pod \"perf-node-gather-daemonset-2mmr4\" (UID: \"1788568b-f83e-45b9-ba74-083ed033b3d7\") " pod="openshift-must-gather-t7jph/perf-node-gather-daemonset-2mmr4"
Jul 27 22:38:21.440046 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:21.440007    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zsx7\" (UniqueName: \"kubernetes.io/projected/1788568b-f83e-45b9-ba74-083ed033b3d7-kube-api-access-9zsx7\") pod \"perf-node-gather-daemonset-2mmr4\" (UID: \"1788568b-f83e-45b9-ba74-083ed033b3d7\") " pod="openshift-must-gather-t7jph/perf-node-gather-daemonset-2mmr4"
Jul 27 22:38:21.440180 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:21.440042    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1788568b-f83e-45b9-ba74-083ed033b3d7-proc\") pod \"perf-node-gather-daemonset-2mmr4\" (UID: \"1788568b-f83e-45b9-ba74-083ed033b3d7\") " pod="openshift-must-gather-t7jph/perf-node-gather-daemonset-2mmr4"
Jul 27 22:38:21.440180 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:21.440046    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1788568b-f83e-45b9-ba74-083ed033b3d7-sys\") pod \"perf-node-gather-daemonset-2mmr4\" (UID: \"1788568b-f83e-45b9-ba74-083ed033b3d7\") " pod="openshift-must-gather-t7jph/perf-node-gather-daemonset-2mmr4"
Jul 27 22:38:21.440180 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:21.440084    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1788568b-f83e-45b9-ba74-083ed033b3d7-sys\") pod \"perf-node-gather-daemonset-2mmr4\" (UID: \"1788568b-f83e-45b9-ba74-083ed033b3d7\") " pod="openshift-must-gather-t7jph/perf-node-gather-daemonset-2mmr4"
Jul 27 22:38:21.440180 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:21.440085    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1788568b-f83e-45b9-ba74-083ed033b3d7-podres\") pod \"perf-node-gather-daemonset-2mmr4\" (UID: \"1788568b-f83e-45b9-ba74-083ed033b3d7\") " pod="openshift-must-gather-t7jph/perf-node-gather-daemonset-2mmr4"
Jul 27 22:38:21.440180 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:21.440147    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1788568b-f83e-45b9-ba74-083ed033b3d7-lib-modules\") pod \"perf-node-gather-daemonset-2mmr4\" (UID: \"1788568b-f83e-45b9-ba74-083ed033b3d7\") " pod="openshift-must-gather-t7jph/perf-node-gather-daemonset-2mmr4"
Jul 27 22:38:21.448391 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:21.448367    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zsx7\" (UniqueName: \"kubernetes.io/projected/1788568b-f83e-45b9-ba74-083ed033b3d7-kube-api-access-9zsx7\") pod \"perf-node-gather-daemonset-2mmr4\" (UID: \"1788568b-f83e-45b9-ba74-083ed033b3d7\") " pod="openshift-must-gather-t7jph/perf-node-gather-daemonset-2mmr4"
Jul 27 22:38:21.588897 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:21.588840    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t7jph/perf-node-gather-daemonset-2mmr4"
Jul 27 22:38:21.711376 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:21.711345    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t7jph/perf-node-gather-daemonset-2mmr4"]
Jul 27 22:38:21.714746 ip-10-0-142-219 kubenswrapper[2577]: W0727 22:38:21.714717    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1788568b_f83e_45b9_ba74_083ed033b3d7.slice/crio-611dcbaceda2ec819abaee1fd270fb0e6f286958e27f0c46bec32d053c6a485b WatchSource:0}: Error finding container 611dcbaceda2ec819abaee1fd270fb0e6f286958e27f0c46bec32d053c6a485b: Status 404 returned error can't find the container with id 611dcbaceda2ec819abaee1fd270fb0e6f286958e27f0c46bec32d053c6a485b
Jul 27 22:38:21.816780 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:21.816731    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t7jph/perf-node-gather-daemonset-2mmr4" event={"ID":"1788568b-f83e-45b9-ba74-083ed033b3d7","Type":"ContainerStarted","Data":"df20450643ac244a2edf1280ffb40e24a6af22723714608e0473a4716108dbd9"}
Jul 27 22:38:21.816886 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:21.816799    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t7jph/perf-node-gather-daemonset-2mmr4" event={"ID":"1788568b-f83e-45b9-ba74-083ed033b3d7","Type":"ContainerStarted","Data":"611dcbaceda2ec819abaee1fd270fb0e6f286958e27f0c46bec32d053c6a485b"}
Jul 27 22:38:21.816886 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:21.816868    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-t7jph/perf-node-gather-daemonset-2mmr4"
Jul 27 22:38:21.831625 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:21.831580    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-t7jph/perf-node-gather-daemonset-2mmr4" podStartSLOduration=0.831566539 podStartE2EDuration="831.566539ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 22:38:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:38:21.831167333 +0000 UTC m=+5625.198357201" watchObservedRunningTime="2026-07-27 22:38:21.831566539 +0000 UTC m=+5625.198756404"
Jul 27 22:38:22.619183 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:22.619158    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4dq86_c9aaedde-8f0d-4cf9-8210-f6d41d5a9138/dns/0.log"
Jul 27 22:38:22.641623 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:22.641599    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4dq86_c9aaedde-8f0d-4cf9-8210-f6d41d5a9138/kube-rbac-proxy/0.log"
Jul 27 22:38:22.767195 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:22.767174    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-frmv7_e610bffe-2c4d-4577-8a48-4849b0d3266d/dns-node-resolver/0.log"
Jul 27 22:38:23.310381 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:23.310356    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dtklb_0b84c165-5dfa-416c-bec6-28c9d55197ab/node-ca/0.log"
Jul 27 22:38:24.123933 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:24.123905    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-79799f5b69-7cqcc_13976f4a-be6e-4e64-bada-c4e12029fcf9/router/0.log"
Jul 27 22:38:24.510785 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:24.510737    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jk8jc_8931191d-6ec9-492d-a5e1-3293972c77bf/serve-healthcheck-canary/0.log"
Jul 27 22:38:24.885102 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:24.885031    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-2cnw9_59d66d60-9124-4133-b539-5d4649ba0eef/insights-operator/0.log"
Jul 27 22:38:24.885841 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:24.885821    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-2cnw9_59d66d60-9124-4133-b539-5d4649ba0eef/insights-operator/1.log"
Jul 27 22:38:24.906146 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:24.906125    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4kjk9_828f730d-81bd-4759-ab08-0959bdf45455/kube-rbac-proxy/0.log"
Jul 27 22:38:24.930506 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:24.930488    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4kjk9_828f730d-81bd-4759-ab08-0959bdf45455/exporter/0.log"
Jul 27 22:38:24.953197 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:24.953176    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4kjk9_828f730d-81bd-4759-ab08-0959bdf45455/extractor/0.log"
Jul 27 22:38:27.784155 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:27.784123    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-vmmff_cd489225-7e37-4dca-895f-3f44c79209c8/s3-init/0.log"
Jul 27 22:38:27.801148 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:27.801120    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-vmmff_cd489225-7e37-4dca-895f-3f44c79209c8/download-hf-model/0.log"
Jul 27 22:38:27.830109 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:27.830071    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-t7jph/perf-node-gather-daemonset-2mmr4"
Jul 27 22:38:27.831216 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:27.831176    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-tw4gf_672995b3-20db-4384-bc79-d21f79e6c10f/s3-tls-init-custom/0.log"
Jul 27 22:38:27.874855 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:27.874825    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-tw4gf_672995b3-20db-4384-bc79-d21f79e6c10f/download-hf-model/0.log"
Jul 27 22:38:27.899869 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:27.899842    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-vfwph_4958abd9-5a06-4b04-a8d6-aae8d80013ae/s3-tls-init-serving/0.log"
Jul 27 22:38:27.920000 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:27.919973    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-vfwph_4958abd9-5a06-4b04-a8d6-aae8d80013ae/download-hf-model/0.log"
Jul 27 22:38:33.984582 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:33.984552    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-667ww_fd05d22e-5585-4f39-8597-4fc6978df793/kube-multus-additional-cni-plugins/0.log"
Jul 27 22:38:34.012117 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:34.012048    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-667ww_fd05d22e-5585-4f39-8597-4fc6978df793/egress-router-binary-copy/0.log"
Jul 27 22:38:34.035202 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:34.035178    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-667ww_fd05d22e-5585-4f39-8597-4fc6978df793/cni-plugins/0.log"
Jul 27 22:38:34.094281 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:34.094242    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-667ww_fd05d22e-5585-4f39-8597-4fc6978df793/bond-cni-plugin/0.log"
Jul 27 22:38:34.112899 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:34.112841    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-667ww_fd05d22e-5585-4f39-8597-4fc6978df793/routeoverride-cni/0.log"
Jul 27 22:38:34.136723 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:34.136688    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-667ww_fd05d22e-5585-4f39-8597-4fc6978df793/whereabouts-cni-bincopy/0.log"
Jul 27 22:38:34.163212 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:34.163183    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-667ww_fd05d22e-5585-4f39-8597-4fc6978df793/whereabouts-cni/0.log"
Jul 27 22:38:34.447606 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:34.447565    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-xgbjk_7c0431ce-d23b-441e-86fc-aa2a5f1c4b0c/kube-multus/0.log"
Jul 27 22:38:34.576871 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:34.576835    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wb7n2_3af7400b-f39f-4f41-a492-586171130c30/network-metrics-daemon/0.log"
Jul 27 22:38:34.601009 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:34.600987    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wb7n2_3af7400b-f39f-4f41-a492-586171130c30/kube-rbac-proxy/0.log"
Jul 27 22:38:35.686347 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:35.686312    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pftzz_72112606-3839-4478-bac9-dfc58a7809f8/ovn-controller/0.log"
Jul 27 22:38:35.705950 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:35.705924    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pftzz_72112606-3839-4478-bac9-dfc58a7809f8/ovn-acl-logging/0.log"
Jul 27 22:38:35.734335 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:35.734306    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pftzz_72112606-3839-4478-bac9-dfc58a7809f8/kube-rbac-proxy-node/0.log"
Jul 27 22:38:35.762896 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:35.762868    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pftzz_72112606-3839-4478-bac9-dfc58a7809f8/kube-rbac-proxy-ovn-metrics/0.log"
Jul 27 22:38:35.787741 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:35.787715    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pftzz_72112606-3839-4478-bac9-dfc58a7809f8/northd/0.log"
Jul 27 22:38:35.815528 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:35.815499    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pftzz_72112606-3839-4478-bac9-dfc58a7809f8/nbdb/0.log"
Jul 27 22:38:35.838868 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:35.838845    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pftzz_72112606-3839-4478-bac9-dfc58a7809f8/sbdb/0.log"
Jul 27 22:38:35.891737 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:35.891699    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pftzz_72112606-3839-4478-bac9-dfc58a7809f8/ovnkube-controller/0.log"
Jul 27 22:38:37.078955 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:37.078929    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-7d88cb55d8-vlktg_835846a4-5fb1-4f89-b451-3aeee176a34d/check-endpoints/0.log"
Jul 27 22:38:37.129923 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:37.129892    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-nj9np_c5915867-c603-4d51-8417-42aada6603ef/network-check-target-container/0.log"
Jul 27 22:38:38.220905 ip-10-0-142-219 kubenswrapper[2577]: I0727 22:38:38.220869    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-fdrp9_c35bbe72-c030-4b9d-aece-459d907893dc/iptables-alerter/0.log"